Macular fovea centralis multipoint accurate positioning method and system based on multi-modal fusion

Abstract

The invention relates to a macular fovea centralis multipoint accurate positioning method and system based on multi-modal fusion, relates to the technical field of medical image processing, and solves the problem of how to overcome lesion interference and image quality difference in diabetic retinopathy patients to realize accurate positioning of macular fovea centralis. The method comprises the following steps: carrying out data preprocessing on multi-classification equal-circle pattern spots, and excluding the types of abnormal pixel area and distance interval in an eye fundus image and abnormal perfusion characteristics in a fluorescein eye fundus angiography image by utilizing a preset equal-circle and equal-distance multipoint constraint condition; and calculating the position, distance and angle of each equal circle according to an equal-circle and equal-distance multi-point constraint condition, and calculating the position of the center point of the macular according to a preset positioning method. The method has the following effect that the positioning accuracy of the macular fovea centralis is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a method and system for accurately locating multiple points of the fovea based on multimodal fusion. Background Art

[0002] The fovea is the area with the sharpest vision on the retina, and its accurate location plays a key role in the diagnosis and treatment of ophthalmic diseases. Especially for common ophthalmic diseases such as age-related macular degeneration and diabetic retinopathy, determining the location of the fovea is of great significance. Taking patients with diabetic retinopathy as an example, due to long-term high blood sugar levels, patients with diabetic retinopathy often experience various structural features of the center of the macula being destroyed, such as retinal edema and congestion, which makes it more difficult to locate the fovea.

[0003] In traditional methods, analysis is often performed based only on fundus images. By extracting features from fundus images, such as color, texture, and shape features, the area of ​​the fovea center feature and the location of the optic disc center feature are found.

[0004] However, for patients with common eye diseases such as age-related macular degeneration and diabetic retinopathy, on the one hand, the structural features of the center of the macula are destroyed, and traditional methods based on single fundus image analysis are difficult to accurately identify. Lesions such as retinal edema and congestion will change the color, texture, and shape characteristics of the fundus image, making the features originally used for positioning unreliable and increasing the difficulty of positioning. On the other hand, relying solely on fundus images is easily affected by image quality. Different shooting equipment, lighting conditions, physician operating experience, and camera parameter settings can lead to large differences in the imaging quality of fundus images. In this case, the existing algorithm has insufficient generalization ability and is difficult to adapt to images of different qualities, which affects the accurate determination of the location of the fovea and the center of the optic disc. Summary of the invention

[0005] In order to improve the accuracy of fovea positioning, the present application provides a fovea multi-point precise positioning method and system based on multimodal fusion.

[0006] In the first aspect, the present application provides a method for accurate multi-point positioning of the fovea based on multimodal fusion, which adopts the following technical solution: A multi-point accurate positioning method of the fovea based on multimodal fusion, comprising: Acquire fundus images and fluorescein fundus angiography images of the target person; The acquired fundus images and fluorescein fundus angiography images are recognized by using the trained multi-classification equal-circle spot analysis model to obtain multi-classification equal-circle spots, wherein the training process of the multi-classification equal-circle spot analysis model is as follows: fundus images and fluorescein fundus angiography images of different individuals are collected, and the positions of the fovea centralis and the optic disc center are marked, and the fovea centralis and the optic disc center are used as reference benchmarks to construct a multi-classification data set with equal circles and equal distances, and the constructed multi-classification data set with equal circles and equal distances is used for model training to form a multi-classification equal-circle spot analysis model capable of processing fundus images and fluorescein fundus angiography images simultaneously; Data preprocessing is performed on multi-classification equal-circle spots, and abnormal categories are excluded by using the preset equal-circle and equal-distance multi-point constraint conditions and the perfusion characteristic normative conditions of the fluorescein fundus angiography image. The abnormal categories include abnormal pixel area and distance interval in the fundus image and abnormal perfusion characteristics in the fluorescein fundus angiography image. According to the multi-point constraint conditions of equal circles and equal distances, the position, distance and angle of each equal circle are calculated, and the position of the macula center is determined according to the preset positioning method.

[0007] By adopting the above technical solutions, multimodal fusion is used to obtain richer image information and improve positioning accuracy. By constructing a multi-classification equal-circle spot analysis model and data preprocessing, abnormal categories are excluded and the reliability of the method is enhanced. The multi-point constraint conditions of equal circles and equal distances help to accurately calculate the position and achieve accurate positioning of the center of the macula. It provides a more accurate basis for ophthalmic diagnosis and treatment, which is of great significance for the early detection and intervention of macular-related diseases.

[0008] Optional annotation of the location of the fovea and optic disc center includes: Extracting image features of fundus images and fluorescein fundus angiography images, wherein the image features include color features, texture features, and shape features; The extracted color features, texture features and shape features are fused, and different types of features are combined using the feature concatenation method to form a concatenated feature vector; The concatenated feature vectors are input into the preset trained classifier. The classifier outputs the probability that different areas in the image belong to the fovea, optic disc or other areas. According to the category with the highest probability, the area that meets the characteristics of the fovea and the location of the optic disc center point characteristics are determined and marked. By adopting the above technical solution, the image features are fully extracted to improve the positioning accuracy, and the fusion of different features can make full use of various information. After the features are serially connected and input into the classifier, the position of the fovea and the optic disc can be automatically determined, reducing the error of manual labeling. Labeling is performed according to the category with the highest probability, which has high reliability and objectivity. At the same time, it provides an accurate reference benchmark for subsequent multi-classification equal circular spot analysis and macular center point positioning, which helps to improve the performance and accuracy of the entire macular fovea multi-point precise positioning method based on multimodal fusion.

[0009] Optionally, the method further includes steps after constructing a multi-classification dataset with equal circles and equal distances and before using the constructed multi-classification dataset with equal circles and equal distances for model training to form a multi-classification equal circle spot analysis model capable of simultaneously processing fundus images and fluorescein fundus angiography images, specifically as follows: Annotate the collected fundus images of different individuals and fluorescein fundus angiography images, and mark the location and characteristics of the blood vessels; Use the labeled dataset to train the vascular segmentation model and infer the vascular confidence map dataset; The vessel segmentation results and the vessel confidence map dataset are used together with the original images as the input of the multi-classification equal-circle pattern analysis model. The multi-classification equal-circle pattern analysis model extracts image features based on the input and classifies the equal-circle areas. The original images are the original fundus images of different individuals and the fluorescein fundus angiography images.

[0010] By adopting the above technical solutions, the location and characteristics of blood vessels are annotated to provide more accurate information for subsequent analysis. The blood vessel segmentation model is trained and the confidence map data set is obtained, which enriches the input of the multi-classification equal-circle spot analysis model. The combination of multiple models improves the analysis ability of fundus images, helps to classify equal-circle areas more accurately, and thus provides a more reliable basis for the precise positioning of multiple points of the fovea. The accuracy and robustness of the method are enhanced, making the entire positioning process more scientific and effective, and providing better technical support for the diagnosis and treatment of ophthalmic diseases.

[0011] Optionally, taking the fovea and the optic disc center as reference datums, constructing a multi-classification data set with equal circles and equal distances includes: calculating the distance between the fovea and the optic disc center according to the determined fovea and the optic disc center; Draw a circle with the fovea as the center and the preset ratio of the distance between the fovea and the center of the optic disc as the radius; Dividing a plurality of equidistant areas at preset intervals along the line connecting the fovea and the center of the optic disc; The image samples and their feature descriptions in each equidistant region are combined to form a multi-classification dataset.

[0012] By adopting the above technical solution, a multi-classification data set is constructed with the fovea and the center of the optic disc. The distance between the two is calculated and used to draw circles and divide equidistant areas, and the image samples and features are combined. Its beneficial effect is that it provides a structured data set construction method, uses key reference points to make the data more targeted, and equidistant division facilitates the analysis of different regional characteristics, providing a rich and accurate data basis for subsequent model training and fovea positioning, which helps to improve positioning accuracy and medical diagnosis efficiency, and has positive significance for the research and treatment of macular lesions.

[0013] Optionally, calculating the position of the macula center according to a preset positioning method includes: Using the trained classifier to detect whether there are special cases in the target image, wherein the target image includes a fundus image and a fluorescein fundus angiography image, and the types of special cases include deformation, blur, noise, and lesion area interference; If yes, then identify the type of special situation, and analyze and determine the positioning method according to the corresponding relationship between the type of special situation and the positioning method, and calculate the position of the center point of the macula using the positioning method determined by the analysis; If not, keep the original setting.

[0014] By adopting the above technical solution, the position of the macula center point is determined by detecting special conditions of the image through the classifier, which has a significant beneficial effect. It can accurately identify special conditions such as deformation and blur, and select positioning methods in a targeted manner, thereby improving positioning accuracy. When there are no special conditions, the original settings are maintained to ensure the stability of the method. It can adapt to different image states, provide a reliable basis for the diagnosis of macular lesions, help improve medical efficiency and accuracy, better serve patients, and is of great significance to ophthalmic clinical practice.

[0015] Optionally, if the type of the special case is fuzzy, the positioning method is analyzed and determined according to the corresponding relationship between the type of the special case and the positioning method, and the position of the center of the macula is calculated using the positioning method determined by the analysis, including: Obtain image clarity index data of fundus images and fluorescein fundus angiography images, where the image clarity indexes include resolution, contrast, and signal-to-noise ratio; According to the image clarity index data of the fundus image and the fluorescein fundus angiography image, a preset weighted average algorithm is applied to analyze the overall effective clarity of the fundus image and the fluorescein fundus angiography image; If the overall effective clarity of the fundus image and fluorescein fundus angiography image exceeds the preset clarity, the position of the macula center is calculated using the multi-point positioning method; If the overall effective clarity of the fundus image and fluorescein fundus angiography image is less than or equal to the preset clarity, the two-point positioning method is used to calculate the position of the macula center.

[0016] By adopting the above technical solution, the effect of determining the position of the center point of the macula for special blurred situations is significant. By obtaining the clarity index and weighted analysis of the overall effective clarity, the image status can be accurately judged. According to different clarity, multi-point positioning method or two-point positioning method is selected to improve positioning accuracy and adaptability. It can effectively deal with blurred images, provide a reliable method for the diagnosis of macular lesions, improve medical efficiency and accuracy, help to better understand the patient's condition, and provide a strong basis for the formulation of treatment plans, which has positive significance for ophthalmic clinical practice.

[0017] Optionally, if the type of the special case is deformation, the positioning method is analyzed and determined according to the corresponding relationship between the type of the special case and the positioning method, and the position of the center of the macula is calculated using the positioning method determined by the analysis, including: Extracting feature data related to fundus image and deformation, the features include edge features, corner features, and regional features; According to the extracted characteristic data, and by applying a preset deformation degree evaluation method, a deformation degree evaluation value is calculated; If the deformation degree evaluation value exceeds the preset deformation degree evaluation value, it is determined that the deformation degree is serious, and the deformed fundus image is matched with the standard fundus image using an image registration method to obtain a corrected image; Use the rectified image data with a preset ratio to conduct experiments, apply different positioning methods and record the accuracy of the results as the test results; According to the test results, the linear regression algorithm is used to automatically learn the weights of different positioning methods on different types of corrected images; according to the weighted average algorithm and the weights of different positioning methods on different types of corrected images, the positioning method is analyzed and determined, and the position of the center of the macula is calculated using the positioning method determined by the analysis; If the deformation degree evaluation value is less than or equal to the preset deformation degree evaluation value, the original setting is maintained.

[0018] By adopting the above technical solution, the location of the center of the macula is well located for special cases of deformation. By extracting multiple features and evaluating the degree of deformation, the deformation state can be accurately judged. The image is corrected by image registration, and the weight is determined by combining experiments and linear regression to improve the positioning accuracy and adaptability. It can effectively process deformed images, provide a reliable method for the diagnosis of macular lesions, improve medical efficiency and accuracy, help to better understand the patient's condition, and provide a strong basis for the formulation of treatment plans, which is of great significance to ophthalmic clinical practice.

[0019] In the second aspect, the present application provides a multi-point precise positioning system of the fovea based on multimodal fusion, which adopts the following technical solutions: A system for multi-point precise positioning of the fovea based on multi-modal fusion, comprising a memory, a processor and a program stored in the memory and executable on the processor, wherein the program can be loaded and executed by the processor to implement the multi-point precise positioning method of the fovea based on multi-modal fusion as described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a flow chart of a method for multi-point precise positioning of the fovea centralis based on multimodal fusion according to an embodiment of the present application.

[0021] Figure 2 It is a schematic diagram of a process for marking the positions of the fovea centralis and the optic disc center point according to another embodiment of the present application.

[0022] Figure 3 It is a flowchart diagram of the steps of another embodiment of the present application after constructing a multi-classification dataset of equal circles and equal intervals, and before using the constructed multi-classification dataset of equal circles and equal intervals for model training to form a multi-classification equal-circle spot analysis model capable of simultaneously processing fundus images and fluorescein fundus angiography images.

[0023] Figure 4 It is a flowchart of another embodiment of the present application for constructing a multi-classification data set with equal circles and equal distances using the fovea and the center of the optic disc as reference benchmarks. DETAILED DESCRIPTION

[0024] The present application is further described in detail below in conjunction with the accompanying drawings.

[0025] Reference Figure 1 , is a multi-point precise positioning method of the fovea based on multimodal fusion disclosed in this application, comprising: Step S100, acquiring a fundus image and a fluorescein fundus angiography image of a target person.

[0026] Among them, fundus images refer to images obtained by photographing the bottom of the human retina through specific equipment, which can show structures such as the retina, blood vessels, and macula. Fundus images can be taken by a fundus camera. The patient sits in front of the camera, and the doctor adjusts the camera parameters to photograph the patient's fundus to obtain a fundus image.

[0027] Fluorescein fundus angiography images require that fluorescein be injected into the patient's body first, and then captured using a special angiography camera. After injecting fluorescein, the doctor waits for a while to allow the fluorescein to circulate in the fundus blood vessels, and then uses the angiography camera to capture a series of images. Fluorescein fundus angiography images can more clearly show the morphology and blood flow of the fundus blood vessels.

[0028] Step S200 , using the trained multi-classification isocircular spot analysis model to identify the acquired fundus image and fluorescein fundus angiography image, to obtain a multi-classification isocircular spot.

[0029] Among them, the training process of the multi-classification equicircular spot analysis model is as follows: fundus images and fluorescein fundus angiography images of different individuals are collected, and the positions of the fovea and the center of the optic disc are marked. The fovea and the center of the optic disc are used as reference benchmarks to construct a multi-classification data set with equal circles and equal distances. The constructed multi-classification data set with equal circles and equal distances is used for model training to form a multi-classification equicircular spot analysis model that can simultaneously process fundus images and fluorescein fundus angiography images.

[0030] The multi-classification isocircular spot analysis model refers to a machine learning model that can analyze fundus images and fluorescein fundus angiography images and identify different types of isocircular spots. The model training process is as follows: Use a deep learning framework (such as TensorFlow or PyTorch) to build a multi-classification isocircular spot analysis model. You can choose a neural network architecture suitable for image classification tasks, such as a convolutional neural network (CNN). Input the labeled data set into the model for training. During the training process, the model continuously adjusts its own parameters to minimize the loss function and improve the classification accuracy of isocircular spots. Continuously adjust the model parameters to improve accuracy. You can use optimization algorithms (such as stochastic gradient descent) to update the model parameters, and you can use data enhancement, regularization and other techniques to prevent overfitting. For example, use a convolutional neural network to build a model, and input the labeled fundus images and fluorescein fundus angiography image datasets into the model for training. Through multiple iterations of training, adjust the network weights and biases so that the model can accurately identify different types of isocircular spots.

[0031] Equicircular spots refer to areas of equal circular spacing constructed with the fovea and the center of the optic disc as reference points. Each area is called an equicircular spot.

[0032] Specifically, fundus images and fluorescein fundus angiography images of different individuals can be collected from a large number of patients' image data in the hospital's ophthalmology database, or they can be obtained through cooperation with multiple medical institutions.

[0033] The location of the fovea and the center of the optic disc can be accurately marked on the image by a professional ophthalmologist using image annotation software. For example, using LabelMe software, the doctor clicks on the corresponding location on the image to mark it.

[0034] Constructing a multi-classification dataset with equal circles and equal distances can calculate the radius and spacing of equal circles based on the marked fovea and optic disc center points, divide the image into different equal circle areas, and classify and annotate each area, such as normal area, diseased area, etc.

[0035] Assume that fundus images and fluorescein fundus angiography images of 1,000 different patients are collected. The doctor first annotates the fovea and optic disc center in these images. Then, a multi-classification dataset with equal circles and equal distances is constructed based on the annotations.

[0036] For example, the first equal circle is constructed with the fovea as the center and a radius of 50 pixels, and the subsequent equal circles are constructed with a spacing of 20 pixels. After the image is divided into multiple equal circular areas, the doctor annotates each area according to the image characteristics. If the blood vessels in a certain area have normal morphology, uniform color, and no obvious pathological features, it is marked as a normal area; if there are pathological features such as vascular leakage and bleeding in the area, it is marked as a pathological area.

[0037] Next, use TensorFlow to build a convolutional neural network model. Input the labeled data set into the model for training, set the learning rate to 0.001, and use the stochastic gradient descent optimization algorithm. After multiple iterations of training, such as 100 epochs, the model gradually converges and can accurately identify different types of equal circular spots.

[0038] For example, for a new fundus image and fluorescein fundus angiography image, the model can quickly classify the equicircular spots in the image into normal areas or diseased areas, providing important basis for subsequent analysis and diagnosis.

[0039] Step S300, preprocessing the data of the multi-classified equal-circle spots, using the preset multi-point constraint conditions of equal circles and equal distances and the normative conditions of the perfusion characteristics of the fluorescein fundus angiography image to exclude abnormal categories, wherein the abnormal categories include abnormal pixel area and distance interval in the fundus image and abnormal perfusion characteristics in the fluorescein fundus angiography image.

[0040] Data preprocessing is to clean and filter the data to improve the data quality and model accuracy. Specifically, it is mainly reflected in the image quality assessment, as follows: First, the fundus images corresponding to the multi-classification equal circular spots and the fluorescein fundus angiography images are evaluated for quality. A variety of indicators can be used, such as peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), etc. For example, the PSNR value of each image is calculated. If the PSNR value is lower than a certain threshold, the image is considered to have poor quality and may affect subsequent analysis. It is marked as pending or directly excluded.

[0041] Multi-point constraint conditions: Constraints set based on the characteristics of equicircular spots and the positional relationship between the fovea and the center of the optic disc are used to exclude abnormal equicircular spots. Abnormal exclusion based on multi-point constraint conditions is as follows: According to the positional relationship between the equicircular spots and the fovea and the center of the optic disc, set a reasonable pixel area and distance interval range. For each equicircular spot, calculate its pixel area. If the pixel area exceeds the preset normal range, it is marked as an abnormal category. Similarly, measure the distance interval between the equicircular spot and the fovea and the center of the optic disc. If the distance interval does not meet the preset standard, it is also marked as abnormal. For example, under normal circumstances, the distance between a certain equicircular spot and the fovea should be within a certain range. If the actual distance exceeds this range, it is considered that the equicircular spot may be abnormal.

[0042] Perfusion feature normative conditions: normative conditions set according to the perfusion features of blood vessels in fluorescein fundus angiography images, used to exclude abnormal equicircular spots. Abnormal exclusion based on perfusion feature normative conditions is as follows: Analyze the perfusion features of blood vessels in fluorescein fundus angiography images, such as perfusion speed, intensity, uniformity, etc. Set the range or standard of normal perfusion features. For example, normal vascular perfusion should have a certain intensity and uniformity. If the perfusion intensity of a certain area is too high or too low, or the perfusion is uneven, it is considered that the equicircular spots in this area may have abnormal perfusion features. For each area where the equicircular spots are located, check whether its perfusion features meet the normative conditions. If not, mark it as an abnormal category.

[0043] For example, a batch of multi-classification circular spots were preprocessed and some images were found to be blurred, which were removed after PSNR evaluation. Then, based on the multi-point constraint conditions, some circular spots with abnormal pixel areas and abnormal distance intervals were excluded. At the same time, based on the perfusion feature normative conditions, some circular spots with abnormal vascular perfusion were excluded.

[0044] Step S400, according to the multi-point constraint condition of equal circles and equal distances, the position, distance and angle of each equal circle are calculated, and the position of the macula center is determined by analysis according to a preset positioning method.

[0045] Multi-point constraints: some constraints set according to the characteristics of the isocircular spots and the positional relationship between the fovea and the center of the optic disc are used to accurately calculate the position, distance and angle of the isocircles and ensure the accuracy of positioning. For example, if the pixel area or distance interval of a certain isocircular spot does not meet the preset range, it may be regarded as an abnormality, thus affecting the calculation of multi-point constraints.

[0046] Positioning method: A specific analysis method used to determine the location of the center of the macula based on the position, distance, and angle of equal circles. Different positioning methods may use different algorithms and strategies, such as weighted average method, geometric center method, etc.

[0047] Isocircle position: refers to the specific coordinate position of the isocircle in the image that is constructed with the fovea as the reference point and according to the preset radius and spacing. The process of determining the position of the isocircle is as follows: the positions of the fovea and the center of the optic disc are known. According to the preset radius and spacing of the isocircles, with the fovea as the center, circles are drawn outward in sequence to determine the position of each isocircle. For example, assuming the preset radius is r and the spacing is d, the radius of the first isocircle is r, the radius of the second isocircle is r+d, the radius of the third is r+2d, and so on.

[0048] Equicircle distance: refers to the distance between a point on the equicircle and the fovea or the center of the optic disc, which can be calculated using methods such as the Euclidean distance formula. The process of determining the equicircle distance is as follows: For each point on the equicircle, use the Euclidean distance formula to calculate its distance from the fovea and the center of the optic disc. The Euclidean distance formula is Where (x1, y1) and (x2, y2) are the coordinates of two points respectively.

[0049] Equicircle angle: After establishing a polar coordinate system with the fovea as the origin, the angle of the point on the equicircle relative to the fovea.

[0050] The process of determining the equal circle angle is as follows: establish a polar coordinate system with the fovea as the origin. For a point (x, y) on the equal circle, its polar angle θ can be calculated by the inverse tangent function, that is,

[0051] The process of determining the position of the macula center point is as follows: Analysis is performed according to the preset positioning method. For example, if the weighted average method is used, the points on each equal circle can be weighted according to the classification results of the equal circle spots and the distance from the fovea. The weight of the points in the normal area can be set higher, and the weight of the points in the diseased area can be set lower. The weight of the points closer to the fovea can also be increased accordingly.

[0052] Then, calculate the weighted average. For each point (x, y), its weight is w, then the formula for calculating the weighted average is Where i represents a different point.

[0053] The calculated (x 中心 ,y 中心 ) is the determined position of the macula center.

[0054] Reference Figure 2 , marking the location of the fovea and the center of the optic disc include: Step S210 , extracting image features of the fundus image and the fluorescein fundus angiography image, wherein the image features include color features, texture features, and shape features.

[0055] Color features refer to the color properties of different regions in an image, such as depth, hue, etc. Texture features refer to the texture patterns of different regions in an image, such as roughness, directionality, etc. Shape features refer to the shape characteristics of different regions in an image, such as circles, ellipses, etc.

[0056] For fundus images and fluorescein fundus angiography images, image processing algorithms can be used to extract image features. For example, for color features, statistics such as the color mean and variance of different regions in the image can be calculated. For texture features, texture analysis algorithms such as gray-level co-occurrence matrix can be used to extract texture roughness, contrast and other features. For shape features, shape detection algorithms such as edge detection and Hough transform can be used to extract the shape information of the region.

[0057] For example, in a fundus image, a specific area is selected and the color mean of the area is calculated to be a darker tone, a rougher texture, and a shape that is approximately circular. In this way, the color, texture, and shape features of different areas in the image can be extracted.

[0058] Step S220, the extracted color features, texture features and shape features are merged, and different types of features are combined together using a feature concatenation method to form a concatenated feature vector.

[0059] Feature fusion refers to combining different types of image features to obtain a more comprehensive image description.

[0060] Feature concatenation is a feature fusion method that connects different types of feature vectors in a certain order to form a longer feature vector.

[0061] After extracting the color features, texture features, and shape features, the feature concatenation method is used for fusion. Specifically, each feature vector is connected together in a predetermined order. For example, the color feature vector can be placed in the front, followed by the texture feature vector, and finally the shape feature vector. In this way, a concatenated feature vector is formed, which contains three different types of image features.

[0062] Assume that the color feature vector is [0.5, 0.3, 0.2] (indicating three eigenvalues ​​of color), the texture feature vector is [0.4, 0.6] (indicating two eigenvalues ​​of texture), and the shape feature vector is [1.0] (indicating one eigenvalue of shape). Through feature concatenation, the fused feature vector obtained is [0.5, 0.3, 0.2, 0.4, 0.6, 1.0].

[0063] Step S230, input the concatenated feature vectors into a preset trained classifier, and the classifier outputs the probability that different areas in the image belong to the fovea, optic disc or other areas. According to the category with the highest probability, the area that meets the characteristics of the fovea and the position of the optic disc center point characteristics are determined.

[0064] First, you need to prepare a trained classifier. This classifier can be a machine learning model trained with a large amount of labeled data, such as a support vector machine, a neural network, etc. The concatenated feature vector is input into the classifier, and the classifier will output the probability that different areas in the image belong to the fovea, the optic disc, or other areas. According to the category with the highest probability, the category to which the area belongs is determined, thereby determining the area that meets the characteristics of the fovea and the location of the optic disc center point characteristics.

[0065] For example, the fused feature vector [0.5, 0.3, 0.2, 0.4, 0.6, 1.0] is input into a trained neural network classifier. The classifier outputs that the probability of this area belonging to the fovea is 0.7, the probability of belonging to the optic disc is 0.2, and the probability of belonging to other areas is 0.1. Since the probability of belonging to the fovea is the highest, this area is determined to be an area that meets the characteristics of the fovea.

[0066] Reference Figure 3 The multi-point precise positioning of the fovea based on multimodal fusion also includes steps after constructing a multi-classification data set with equal circles and equal distances, and before using the constructed multi-classification data set with equal circles and equal distances for model training to form a multi-classification equal-circle spot analysis model capable of simultaneously processing fundus images and fluorescein fundus angiography images, specifically as follows: Step Sa00, annotating the collected fundus images and fluorescein fundus angiography images of different individuals, and marking the positions and characteristics of the blood vessels.

[0067] During the labeling process, the location of the blood vessels can be determined based on their color, shape, contrast and other characteristics. For example, in fundus images, blood vessels usually appear as dark red linear structures; in fluorescein fundus angiography images, blood vessels will appear more clearly after fluorescein is injected.

[0068] The feature annotation of blood vessels can include information such as the diameter, length, curvature, blood flow velocity, etc. These features can be determined by measuring tools or observations and recorded in the annotation software.

[0069] After the marking is completed, the marking results are reviewed and revised to ensure the accuracy and consistency of the marking.

[0070] Step Sb00: Use the labeled data set to train the blood vessel segmentation model and infer the blood vessel confidence map data set.

[0071] The labeled dataset refers to the labeled fundus images and fluorescein fundus angiography image datasets, which contain the location and feature annotation information of the blood vessels.

[0072] The vascular segmentation model refers to a machine learning model that can automatically identify and segment blood vessels in fundus images and fluorescein fundus angiography images.

[0073] The vascular confidence map dataset refers to a dataset representing the probability of the existence of blood vessels obtained by inferring the vascular segmentation model. The value of each pixel represents the probability that the point belongs to a blood vessel.

[0074] The specific process is as follows: Select appropriate machine learning algorithms and frameworks, such as convolutional neural networks (CNNs) in deep learning, to build a vascular segmentation model. Divide the labeled data set into a training set, a validation set, and a test set. Use the training set to train the vascular segmentation model, and continuously adjust the parameters of the model so that the model can accurately identify and segment the blood vessels. During the training process, a loss function can be used to measure the difference between the model's prediction results and the true annotations, and the loss function can be minimized by an optimization algorithm. Common loss functions include cross entropy loss function, mean square error loss function, etc. After the training is completed, the model is verified using a validation set to evaluate the performance and generalization ability of the model. If the model performance is not ideal, the model structure, parameters, or training data can be adjusted. Finally, the trained vascular segmentation model is used to infer the test set to obtain a vascular confidence map data set. The value of each pixel in the vascular confidence map represents the probability that the point belongs to a blood vessel, and the location and range of the blood vessel can be determined based on the size of the probability value.

[0075] Step Sc00, the vascular segmentation result and the vascular confidence map dataset are used together with the original image as the input of the multi-classification equal-circle pattern analysis model. The multi-classification equal-circle pattern analysis model extracts image features based on the input and classifies the equal-circle areas. The original images are the original fundus images of different individuals and the fluorescein fundus angiography images.

[0076] The vascular segmentation result refers to the vascular position and range information obtained through the vascular segmentation model.

[0077] The specific process is as follows: The vascular segmentation results and the vascular confidence map dataset are fused with the original image by using methods such as image stitching and channel superposition. For example, the vascular segmentation results are spliced ​​with the original image in the form of a binary image, or the vascular confidence map dataset is superimposed as an additional channel with the color channel of the original image.

[0078] The fused image is used as the input of the multi-classification circular pattern analysis model. The model can use convolutional neural networks or other machine learning algorithms in deep learning to identify different types of circular patterns by extracting and analyzing the features of the input image.

[0079] Feature extraction can be achieved through network structures such as convolutional layers and pooling layers to extract image features such as color, texture, and shape. Then, based on the extracted features, a classifier is used to classify the equal circle area. The classifier can be a fully connected layer, a support vector machine, etc.

[0080] During the classification process, the loss function can be used to measure the difference between the model's prediction results and the true annotations, and the optimization algorithm can be used to adjust the model's parameters to improve the accuracy of classification.

[0081] Reference Figure 4 , taking the fovea and the center of the optic disc as reference points, constructing a multi-classification dataset with equal circles and equal distances, including: Step S2A0, calculating the distance between the fovea center and the optic disc center according to the determined fovea center and optic disc center.

[0082] Specifically, after determining the positions of the fovea and the center of the optic disc, mathematical methods can be used to calculate the distance between them. For example, if the coordinates of the fovea are (x1, y1) and the coordinates of the center of the optic disc are (x2, y2), the distance between them can be calculated according to the Euclidean distance formula: d = sqrt((x2-x1)^2+(y2-y1)^2).

[0083] For example: Assuming the coordinates of the fovea are (30,40) and the coordinates of the center of the optic disc are (60,50), the distance between them calculated according to the Euclidean distance formula is sqrt((60-30)^2+(50-40)^2)=sqrt(900+100)=sqrt(1000)≈31.62.

[0084] Step S2B0, drawing a circle with the fovea as the center and a preset ratio of the distance between the fovea and the center of the optic disc as the radius.

[0085] Specifically, after the fovea is determined as the center of the circle, the radius length is determined according to the distance between the fovea and the center of the optic disc calculated in step S1a0, according to a preset ratio. For example, the radius can be set to half of this distance. Then, a drawing software or algorithm is used to draw a circle with the fovea as the center and the determined radius.

[0086] For example, if the distance between the fovea and the center of the optic disc is 31.62 and the preset ratio is 0.5, then the radius is 31.62*0.5=15.81. Draw a circle with the fovea as the center and 15.81 as the radius.

[0087] Step S2C0, dividing the line between the fovea and the center of the optic disc into a plurality of equidistant areas at preset intervals.

[0088] Specifically, after determining the direction of the line connecting the fovea and the center of the optic disc, the area is divided in this direction according to a preset interval. For example, the interval can be set to a certain pixel length. Starting from the fovea, multiple equidistant areas are divided in sequence along the direction of the line.

[0089] Assuming that the preset interval is 5 pixels in length, starting from the fovea, a region is divided every 5 pixels in length along the connecting line direction, so that multiple equidistant regions can be obtained.

[0090] Step S2D0, combining the image samples and their feature descriptions in each equidistant area to form a multi-classification data set.

[0091] For each equidistant region, extract the image samples in the region and calculate its feature description. Combine the image samples and feature descriptions of each equidistant region to form a multi-classification dataset. Each sample in this dataset contains the information of the region and the corresponding feature description.

[0092] For example, for each equidistant region, extract the fundus image portion within the region as an image sample. Then calculate the color features, texture features, and shape features of the image sample as feature descriptions. Combine the image samples and their feature descriptions of all equidistant regions to form a multi-classification dataset, such as {(region 1, image sample 1, feature description 1), (region 2, image sample 2, feature description 2)}.

[0093] The calculation of the macular center position according to the preset positioning method includes: Step S410, using the trained classifier to detect whether there is a special case in the target image. If yes, execute step S420; if no, execute step S430.

[0094] The target images include fundus images and fluorescein fundus angiography images, and the types of special cases include deformation, blur, noise, and interference of lesion areas.

[0095] Specifically, a trained classifier is prepared, which can be a model based on deep learning, and is trained by a large amount of image data with special cases annotated. The target image (fundus image and fluorescein fundus angiography image) is input into the classifier, and the classifier performs forward propagation calculation and outputs the judgment result of whether there is a special case in the image. If the classifier determines that there is a special case in the image, step S420 is executed; if it is determined that there is no special case in the image, step S430 is executed.

[0096] For example, a convolutional neural network is used as a classifier to train a large number of fundus images and fluorescein fundus angiography images containing different types of special cases and normal cases. When a new target image is input, the classifier outputs a judgment result that there is a special case of image blur.

[0097] Step S420, identifying the type of special case, and analyzing and determining the positioning method according to the corresponding relationship between the type of special case and the positioning method, and calculating the position of the macula center point using the positioning method determined by analysis.

[0098] Specifically, when a special situation is detected in an image, the specific type of the special situation is further identified. For example, another classifier or feature extraction algorithm can be used to determine whether the special situation is deformation, blur, noise, or lesion area interference. Then, based on the pre-established correspondence between the type of special situation and the positioning method, the positioning method suitable for the current special situation is determined. Different special situations may require different positioning methods to improve the accuracy of positioning. For example, for blurred images, a method of positioning after image enhancement can be used; for images with lesion area interference, a method of positioning after excluding the lesion area can be used.

[0099] For example, after identification, it is determined that the special situation in the current image is noise interference. According to the pre-established correspondence, for the image with noise interference, the method of filtering to remove the noise and then positioning can be used. Use a specific filtering algorithm to process the image, remove the noise, and then use the original positioning method to calculate the position of the macula center.

[0100] Step S430, maintaining the original settings.

[0101] If the type of the special case is fuzzy, the positioning method is analyzed and determined according to the corresponding relationship between the type of the special case and the positioning method, and the position of the center of the macula is calculated using the positioning method determined by the analysis, including: Step S42a, acquiring image clarity index data of the fundus image and the fluorescein fundus angiography image, where the image clarity index includes resolution, contrast, and signal-to-noise ratio.

[0102] Among them, resolution refers to the degree of the smallest detail that can be distinguished in the image, usually expressed in the number of pixels. Contrast refers to the degree of brightness difference between different areas in the image. The signal-to-noise ratio refers to the ratio of signal to noise, reflecting the degree of interference between the intensity of the signal and the noise in the image. Specifically, for fundus images and fluorescein fundus angiography images, image processing algorithms can be used to obtain image clarity index data. For example, for resolution, the pixel size of the image can be directly obtained. For contrast, the brightness difference between different areas in the image can be calculated, such as by calculating the standard deviation of the grayscale histogram of the image to measure the contrast. For the signal-to-noise ratio, it can be obtained by calculating the ratio of the signal power to the noise power in the image.

[0103] For example, for a fundus image, the resolution is 1024×1024 pixels. By calculating the standard deviation of the grayscale histogram, the contrast is 0.5. By performing frequency domain analysis on the image and calculating the ratio of signal power to noise power, the signal-to-noise ratio is 20 dB.

[0104] Step S42b, based on the image clarity index data of the fundus image and the fluorescein fundus angiography image, a preset weighted average algorithm is applied to analyze the overall effective clarity of the fundus image and the fluorescein fundus angiography image.

[0105] The weighted average algorithm refers to an algorithm that assigns different weights according to the importance of different indicators, and then performs weighted summation to obtain the average value.

[0106] Overall effective clarity refers to a value that measures the overall clarity of the image after comprehensively considering the clarity indicators of the fundus image and the fluorescein fundus angiography image.

[0107] Specifically, first determine the weights of resolution, contrast, and signal-to-noise ratio in measuring image clarity. For example, based on experience or experiments, the weight of resolution can be determined to be 0.4, the weight of contrast can be determined to be 0.3, and the weight of signal-to-noise ratio can be determined to be 0.3. Then, the clarity index data of the fundus image and the fluorescein fundus angiography image are weighted averaged according to these weights. For example, let the resolution of the fundus image be R1, the contrast be C1, and the signal-to-noise ratio be S1; let the resolution of the fluorescein fundus angiography image be R2, the contrast be C2, and the signal-to-noise ratio be S2. Then the overall effective clarity can be calculated as 0.4×(R1+R2) / 2+0.3×(C1+C2) / 2+0.3×(S1+S2) / 2.

[0108] For example, assume that the resolution of the fundus image is 1024×1024 pixels, the contrast is 0.5, and the signal-to-noise ratio is 20dB; the resolution of the fluorescein fundus angiography image is 800×800 pixels, the contrast is 0.4, and the signal-to-noise ratio is 18dB. The weight of resolution is 0.4, the weight of contrast is 0.3, and the weight of signal-to-noise ratio is 0.3. The overall effective clarity is 0.4×((1024×1024+800×800) / 2)^(1 / 2)+0.3×(0.5+0.4) / 2+0.3×(20+18) / 2≈418.8.

[0109] Step S42c: if the overall effective clarity of the fundus image and the fluorescein fundus angiography image exceeds the preset clarity, a multi-point positioning method is used to calculate the position of the macula center.

[0110] Multi-point positioning refers to a method that uses multiple reference points and complex calculations to determine the location of the center of the macula. It is usually calculated by combining multiple feature points and constraints in fundus images and fluorescein fundus angiography images to improve the accuracy of positioning.

[0111] The specific process is as follows: Determine reference points: When the overall effective clarity of the fundus image and the fluorescein fundus angiography image exceeds the preset clarity, multiple reference points are first determined. These reference points may include the fovea and the center of the optic disc, as well as other significant feature points determined by image analysis, such as vascular branch points, the center of a specific lesion area, etc. For example, an image processing algorithm may be used to automatically detect vascular branch points and use them as one of the reference points. For the fovea and the center of the optic disc, their positions may be determined by previous annotations or automatic detection algorithms.

[0112] Establish constraints: Establish a series of constraints based on the positional relationship between reference points and image features. These constraints can include distance constraints, angle constraints, brightness constraints, etc. Distance constraints can specify the minimum and maximum distance ranges between reference points. For example, the distance between the fovea and the center of the optic disc is usually within a certain range. Angle constraints can specify the angle range between the line between reference points and a specific direction. Brightness constraints can limit the position of the macula center based on the brightness difference between different areas in the image. For example, it is empirically determined that the distance between the fovea and the center of the optic disc should be within the range of [50,200] pixels, and the angle between the line and the horizontal direction should be within the range of [-30°,30°].

[0113] Calculate the position of the center of the macula: Based on the determined reference points and the established constraints, a complex calculation method is used to determine the position of the center of the macula. Optimization algorithms, such as genetic algorithms, simulated annealing algorithms, etc., can be used to minimize the objective function to solve the optimal position of the center of the macula. The objective function can be defined based on the constraints such as the distance, angle, brightness between the reference points and the characteristics of the image. For example, the objective function can be to minimize the sum of the distances between the reference points and the center of the macula while satisfying the brightness constraint. For example, a genetic algorithm is used for optimization. First, a set of initial solutions are randomly generated, each solution represents a possible position of the center of the macula. Then the fitness value of each solution is calculated according to the objective function. Through genetic operations such as selection, crossover and mutation, the set of solutions is continuously updated until the optimal solution is found or the stop condition is met.

[0114] Step S42d: If the overall effective clarity of the fundus image and the fluorescein fundus angiography image is less than or equal to the preset clarity, the two-point positioning method is used to calculate the position of the macula center.

[0115] The two-point positioning method is a relatively simple positioning method, which usually determines the position of the macula center based on the positional relationship between two known points and is suitable for situations with low image clarity.

[0116] Specifically, if the overall effective clarity of the fundus image and the fluorescein fundus angiography image is less than or equal to the preset clarity, it means that the image is relatively blurred, and the two-point positioning method is used at this time. This method can determine the position of the macula center based on the known position of the optic disc center and some simple image features. For example, it can be determined empirically that the macula center is usually within a specific direction and distance range of the optic disc center. Then, within this range, based on some simple features of the image, such as color changes, texture direction, etc., the position of the macula center is determined.

[0117] For example, if the overall effective clarity of the fundus image and the fluorescein fundus angiography image is less than or equal to the preset clarity, it means that the image is relatively blurred, and the two-point positioning method is used at this time. This method can determine the position of the macula center based on the known position of the optic disc center and some simple image features. For example, it can be determined empirically that the macula center is usually within a specific direction and distance range of the optic disc center. Then, within this range, based on some simple features of the image, such as color changes, texture direction, etc., the position of the macula center is determined.

[0118] If the type of special case is deformation, the positioning method is analyzed and determined according to the corresponding relationship between the type of special case and the positioning method, and the position of the center of the macula is calculated using the positioning method determined by the analysis, including: Step S42A, extracting feature data related to the fundus image and deformation, the features including edge features, corner features, and region features.

[0119] Among them, edge features are the boundary information between different areas in the image, corner features are points with high curvature in the image, usually at the edge of the image, and regional features refer to the properties of a specific area in the image, such as area, shape, etc.

[0120] For fundus images, image processing algorithms can be used to extract deformation-related feature data. For example, edge detection algorithms (such as Canny edge detection) are used to extract edge features. Edge detection algorithms determine the location of edges by calculating the gradient of an image. Corner detection algorithms (such as Harris corner detection) are used to extract corner features. Corner detection algorithms determine the location of corners by calculating the autocorrelation matrix of a local area of ​​an image. Regional segmentation algorithms (such as the watershed algorithm) are used to extract regional features. Regional segmentation algorithms divide an image into different regions and calculate the attributes of each region.

[0121] For example, for a fundus image, the Canny edge detection algorithm is used to extract edge features and obtain edge lines in the image. The Harris corner detection algorithm is used to extract corner features and obtain the corner positions in the image. The watershed algorithm is used to extract regional features and obtain different regions in the image and their attributes.

[0122] Step S42B: Calculate a deformation degree evaluation value based on the extracted feature data and by applying a preset deformation degree evaluation method.

[0123] Feature data refers to deformation-related information extracted from fundus images, including edge features, corner features, and region features.

[0124] Deformation degree evaluation method: an algorithm or formula used to calculate the deformation degree of the fundus image based on feature data. The calculation process of the deformation degree evaluation value is as follows: 1. Calculate the curvature of the edge: For the extracted edge features, the edge line obtained by the edge detection algorithm can determine the curvature by calculating the angle change between adjacent points on the edge. First, traverse each point on the edge line and find its two adjacent points. Then, calculate the angle between the two line segments formed by these three points. The dot product and modulus of the vector can be used to calculate the cosine value of the angle, and then the size of the angle is obtained. Perform statistical analysis on the angles of all edge points, such as calculating the average angle, the standard deviation of the angle, etc., as an indicator of the degree of edge curvature. 2. Calculate the displacement of the corner point: For the corner point feature, the displacement is calculated by comparing the positions of the corner points before and after deformation. Determine the position coordinates of the corner point in the fundus image before deformation, and then find the corresponding corner point position in the deformed image. Calculate the Euclidean distance between the two positions, which is the displacement of the corner point. Perform statistical analysis on the displacement of all corner points, such as calculating the average displacement, the maximum and minimum displacement, etc., as an indicator of the displacement of the corner point. 3. Calculate the shape change of the region: For regional features, use the regional segmentation algorithm to obtain different regions and their attributes, such as area, perimeter, etc. Calculate the area and perimeter changes of the region before and after deformation. For example, the area change rate is calculated as (area after deformation - area before deformation) / area before deformation, and the perimeter change rate is (perimeter after deformation - perimeter before deformation) / perimeter before deformation. The shape change indicators of multiple regions can be combined, such as calculating the average and standard deviation of the area change rate of all regions, the average and standard deviation of the perimeter change rate, etc. 4. Combine indicators to obtain the deformation degree assessment value: Determine the weights of edge curvature, corner displacement, and regional shape change. These weights can be determined based on experience or experiment, for example, the weight of edge curvature is 0.4, the weight of corner displacement is 0.3, and the weight of regional shape change is 0.3. Perform weighted average of each index according to the weight to obtain a comprehensive deformation degree evaluation value. For example, if the edge curvature index is E, the corner displacement index is C, and the regional shape change index is R, then the deformation degree evaluation value is 0.4E+0.3C+0.3R.

[0125] Step S42C: if the deformation degree evaluation value exceeds the preset deformation degree evaluation value, it is determined that the deformation degree is serious, and an image registration method is used to match the deformed fundus image with a standard fundus image to obtain a corrected image.

[0126] Image registration is a technique for matching a deformed image with a standard image to achieve image correction. The corrected image refers to the fundus image obtained after image registration with the deformation corrected.

[0127] Specifically, if the deformation degree evaluation value exceeds the preset deformation degree evaluation value, it is determined that the deformation degree is severe. At this time, the image registration method can be used to correct the fundus image. The image registration method generally includes the steps of feature extraction, feature matching and transformation estimation. First, features are extracted from the deformed fundus image and the standard fundus image, such as the edge features and corner features extracted in step S42A. Then, feature matching is performed to find the corresponding feature points in the deformed image and the standard image. Finally, the transformation matrix is ​​estimated based on the matched feature points, and the deformed fundus image is transformed to match it with the standard image to obtain the corrected image.

[0128] Assuming that the preset deformation evaluation value is 0.5, the calculated deformation evaluation value is 0.6, which exceeds the preset value. Using the image registration method, edge features and corner features are extracted from the deformed fundus image and the standard fundus image, and the transformation matrix is ​​estimated after feature matching. The deformed fundus image is transformed to obtain the corrected image.

[0129] Step S42D, using the corrected image data with a preset ratio to conduct experiments, applying different positioning methods and recording the accuracy of the results as the test results.

[0130] The accuracy of the results refers to the correctness of the results obtained on the test data using different positioning methods.

[0131] For example, 50% of the data from the rectified image is selected for testing. The data is calculated using three different positioning methods to obtain the position of the center of the macula. Then, the calculated position is compared with the true position, and the accuracy of each positioning method is calculated to be 80%, 70%, and 60% respectively.

[0132] Step S42E: Based on the test results, a linear regression algorithm is used to automatically learn the weights of different positioning methods on different types of corrected images.

[0133] Specifically, a linear regression algorithm is used to automatically learn the weights of different positioning methods on different types of corrected images using the test results as input. The linear regression algorithm determines the relationship between variables by minimizing the error between the predicted value and the true value. In this problem, the accuracy of different positioning methods is used as a variable, and the linear regression algorithm is used to learn the weight of each positioning method so that the weighted accuracy is the highest.

[0134] Taking the test result in step S42D as an example, the weights of the three positioning methods are learned using a linear regression algorithm, assuming that the learned weights are 0.5, 0.3, and 0.2, respectively.

[0135] Step S42F, analyzing and determining the positioning method based on the weighted average algorithm and the weights of different positioning methods on different types of corrected images, and using the positioning method determined by the analysis to calculate the position of the center of the macula.

[0136] Taking the weights obtained in step S42E as an example, assuming that the accuracy rates of the three positioning methods are 80%, 70% and 60% respectively, and the weights are 0.5, 0.3 and 0.2 respectively. The comprehensive accuracy rate is 0.5×80%+0.3×70%+0.2×60%=73%. The positioning method with the highest comprehensive accuracy rate is selected and used to calculate the position of the macula center point.

[0137] Step S42G: if the deformation degree evaluation value is less than or equal to the preset deformation degree evaluation value, the original setting is maintained.

[0138] Furthermore, considering that the number of excluded abnormal categories is related to data quality, the change in the number of abnormal categories will affect the positioning method. The calculation of the macular center position according to the preset positioning method includes: Step S42H, analyzing and obtaining the number of excluded abnormal categories.

[0139] Specifically, the abnormal category refers to the situation in which the preset multi-point constraint conditions of equal circles and equal distances in the fundus image and the fluorescein fundus angiography image and the normative conditions of the perfusion characteristics of the fluorescein fundus angiography image are not met, such as abnormalities in the pixel area and distance interval in the fundus image and abnormal perfusion characteristics in the fluorescein fundus angiography image.

[0140] The number of excluded anomaly categories refers to the number of images that were identified as anomaly categories after data preprocessing and excluded from subsequent analysis.

[0141] The specific process is as follows: 1. Using the preset constraints and normative conditions, check the fundus images and fluorescein fundus angiography images one by one. 2. For each image, determine whether it has abnormal pixel area, abnormal distance interval, or abnormal perfusion characteristics. If so, mark it as an abnormal category. 3. Count the number of images marked as abnormal categories, which is the number of excluded abnormal categories.

[0142] Step S42I, analyzing and determining the positioning method according to the correspondence between the interval range into which the acquired number ratio of abnormal categories falls and the positioning method.

[0143] Quantity ratio: the ratio of the number of excluded abnormal categories to the total number of images.

[0144] Range: Divide the quantity proportion into different ranges, such as small abnormality (0%-10%), medium abnormality (11%-30%), and large abnormality (31% and above).

[0145] Specifically, for a small number of abnormal categories, more conventional positioning methods, such as the two-point positioning method, can continue to be used. Because the overall distribution and characteristics of the data are relatively stable, the simplicity of the two-point positioning method can meet the needs and the calculation speed is fast.

[0146] For moderate abnormalities, consider using triangulation. At this time, the data has some abnormalities, but there are still relatively clear reference points. Triangulation uses three known points to determine the position of the center of the macula, which can improve the accuracy of positioning.

[0147] A large number of anomalies, using multi-lateral positioning. Due to poor data quality, more known points need to be combined to improve positioning accuracy. The anomaly detection algorithm in deep learning is introduced to pre-process the data and remove more abnormal data.

[0148] Step S42J, calculating the position of the macula center according to the determined positioning method.

[0149] In addition, when blur, deformation, and abnormal number of categories occur simultaneously, priority consideration needs to be given. At this time, the position of the center of the macula is calculated according to the preset positioning method as follows: Step S42K, obtaining the deformation degree evaluation value, image clarity index data, and the number of excluded abnormal categories.

[0150] Step S42L, analyzing whether the deformation degree evaluation value exceeds a preset evaluation value.

[0151] The preset evaluation value of the deformation degree is determined based on experience or experiments, for example, set to 0.5 (this value is only an example and can be determined according to specific needs in actual situations).

[0152] Assume that the deformation degree evaluation value calculated in the previous step is 0.6, and compare it with the preset evaluation value 0.5. Since 0.6>0.5, it is determined that the deformation degree is serious.

[0153] Step S42M, if yes, it is determined that the degree of deformation is serious, and the positioning method determined based on the severity of the deformation is used as the positioning method used this time to calculate the position of the macula center point.

[0154] This step may refer to the example of step S42C to step S42F.

[0155] Step S42N, if no, then analyze and determine the positioning method based on the correspondence between the interval range that the image clarity falls into, the interval range of the number of excluded abnormal categories and the positioning method, and use it as the positioning method used this time to calculate the position of the macula center point.

[0156] The image clarity interval range is determined as follows: Based on the acquired image clarity index data, the overall effective clarity of the fundus image and the fluorescein fundus angiography image is calculated. If the overall effective clarity is less than or equal to the preset clarity, the image is determined to be in a blurred state and further divided into blur degree intervals. For example, the overall effective clarity of 0 to 200 is severe blur, 201 to 400 is moderate blur, and 401 to 500 is mild blur.

[0157] Calculate the ratio of the number of excluded abnormal categories to the total number of images. Divide the interval range according to the ratio, such as 0% to 10% for a small amount of abnormalities, 11% to 30% for a medium amount of abnormalities, and 31% and above for a large amount of abnormalities.

[0158] The positioning method is determined based on the correspondence between the range of image clarity and the number of abnormal categories and the positioning method. For example, when the image is slightly blurred and the number of abnormal categories is small, a slightly adjusted two-point positioning method is used to increase the weight of some known features; when the image is moderately blurred and the number of abnormal categories is below medium, a two-point positioning method combined with simple image enhancement and feature extraction is used. If the number of abnormal categories is large, clinical information can be combined to assist positioning.

[0159] Based on the same inventive concept, an embodiment of the present invention provides a multi-point accurate positioning system for the fovea centralis based on multimodal fusion, including a memory and a processor, wherein the memory stores a program that can be executed on the processor to implement the following Figures 1 to 4 Procedure for either method.

[0160] The embodiments of this specific implementation method are all preferred embodiments of the present application, and are not intended to limit the protection scope of the present application. Therefore, all equivalent changes made based on the structure, shape, and principle of the present application should be included in the protection scope of the present application.

Claims

1. A multi-point accurate positioning method of the fovea based on multimodal fusion, characterized in that: include: Acquire fundus images and fluorescein fundus angiography images of the target person; The acquired fundus images and fluorescein fundus angiography images are recognized by using the trained multi-classification equal-circle spot analysis model to obtain multi-classification equal-circle spots, wherein the training process of the multi-classification equal-circle spot analysis model is as follows: fundus images and fluorescein fundus angiography images of different individuals are collected, and the positions of the fovea macula and the center of the optic disc are marked, and the fovea macula and the center of the optic disc are used as reference benchmarks to construct a multi-classification data set with equal circles and equal distances, and the constructed multi-classification data set with equal circles and equal distances is used for model training to form a multi-classification equal-circle spot analysis model capable of processing fundus images and fluorescein fundus angiography images simultaneously; Data preprocessing is performed on multi-classification equal-circle spots, and abnormal categories are excluded by using the preset equal-circle and equal-distance multi-point constraint conditions and the perfusion characteristic normative conditions of the fluorescein fundus angiography image. The abnormal categories include abnormal pixel area and distance interval in the fundus image and abnormal perfusion characteristics in the fluorescein fundus angiography image. According to the multi-point constraint conditions of equal circles and equal distances, the position, distance and angle of each equal circle are calculated, and the position of the macula center is determined according to the preset positioning method.

2. The method for multi-point precise positioning of the fovea based on multimodal fusion according to claim 1, characterized in that: The locations of the fovea and optic disc center are marked as follows: Extracting image features of fundus images and fluorescein fundus angiography images, wherein the image features include color features, texture features, and shape features; The extracted color features, texture features and shape features are fused, and different types of features are combined using the feature concatenation method to form a concatenated feature vector; The concatenated feature vectors are input into the preset trained classifier. The classifier outputs the probability that different areas in the image belong to the fovea, optic disc or other areas. According to the category with the highest probability, the area that meets the characteristics of the fovea and the location of the optic disc center point characteristics are determined and marked.

3. The method for multi-point precise positioning of the fovea based on multimodal fusion according to claim 2, characterized in that: The method further includes steps after constructing a multi-classification dataset of equal circles and equal distances and before using the constructed multi-classification dataset of equal circles and equal distances for model training to form a multi-classification equal circle spot analysis model capable of simultaneously processing fundus images and fluorescein fundus angiography images, which are specifically as follows: Annotate the collected fundus images of different individuals and fluorescein fundus angiography images, and mark the location and characteristics of the blood vessels; Use the labeled dataset to train the vascular segmentation model and infer the vascular confidence map dataset; The vessel segmentation results and the vessel confidence map dataset are used together with the original images as the input of the multi-classification equal-circle pattern analysis model. The multi-classification equal-circle pattern analysis model extracts image features based on the input and classifies the equal-circle areas. The original images are the original fundus images of different individuals and the fluorescein fundus angiography images.

4. The method for multi-point precise positioning of the fovea based on multimodal fusion according to claim 1, characterized in that: Taking the fovea and the center of the optic disc as reference points, a multi-classification dataset with equal circles and equal distances is constructed, including: According to the determined fovea and optic disc center points, the distance between the fovea and the optic disc center point is calculated; Draw a circle with the fovea as the center and the preset ratio of the distance between the fovea and the center of the optic disc as the radius; Dividing a plurality of equidistant areas at preset intervals along the line connecting the fovea and the center of the optic disc; The image samples and their feature descriptions in each equidistant region are combined to form a multi-classification dataset.

5. The method for multi-point precise positioning of the fovea based on multimodal fusion according to claim 1, characterized in that: The calculation of the macular center position according to the preset positioning method includes: Using the trained classifier to detect whether there are special cases in the target image, wherein the target image includes a fundus image and a fluorescein fundus angiography image, and the types of special cases include deformation, blur, noise, and lesion area interference; If yes, then identify the type of special situation, and analyze and determine the positioning method according to the corresponding relationship between the type of special situation and the positioning method, and calculate the position of the center point of the macula using the positioning method determined by the analysis; If not, keep the original setting.

6. The method for multi-point precise positioning of the fovea based on multimodal fusion according to claim 5, characterized in that: If the type of the special case is fuzzy, the positioning method is analyzed and determined according to the corresponding relationship between the type of the special case and the positioning method, and the position of the center of the macula is calculated using the positioning method determined by the analysis, including: Obtain image clarity index data of fundus images and fluorescein fundus angiography images, where the image clarity indexes include resolution, contrast, and signal-to-noise ratio; According to the image clarity index data of the fundus image and the fluorescein fundus angiography image, a preset weighted average algorithm is applied to analyze the overall effective clarity of the fundus image and the fluorescein fundus angiography image; If the overall effective clarity of the fundus image and fluorescein fundus angiography image exceeds the preset clarity, the position of the macula center is calculated using the multi-point positioning method; If the overall effective clarity of the fundus image and fluorescein fundus angiography image is less than or equal to the preset clarity, the two-point positioning method is used to calculate the position of the macula center.

7. The method for multi-point precise positioning of the fovea based on multimodal fusion according to claim 5, characterized in that: If the type of special case is deformation, the positioning method is analyzed and determined according to the corresponding relationship between the type of special case and the positioning method, and the position of the center of the macula is calculated using the positioning method determined by the analysis, including: Extracting feature data related to fundus image and deformation, the features include edge features, corner features, and regional features; According to the extracted characteristic data, and by applying a preset deformation degree evaluation method, a deformation degree evaluation value is calculated; If the deformation degree evaluation value exceeds the preset deformation degree evaluation value, it is determined that the deformation degree is serious, and the deformed fundus image is matched with the standard fundus image using an image registration method to obtain a corrected image; Use the rectified image data with a preset ratio to conduct experiments, apply different positioning methods and record the accuracy of the results as the test results; According to the experimental results, a linear regression algorithm is used to automatically learn the weights of different positioning methods on different types of rectified images; According to the weighted average algorithm and the weights of different positioning methods on different types of corrected images, the positioning method is analyzed and determined, and the position of the center of the macula is calculated using the positioning method determined by the analysis; If the deformation degree evaluation value is less than or equal to the preset deformation degree evaluation value, the original setting is maintained.

8. A multi-point precise positioning system of the fovea based on multimodal fusion, characterized in that: It includes a memory, a processor, and a program stored in the memory and executable on the processor, wherein the program can be loaded and executed by the processor to implement a multi-point precise positioning method of the fovea centralis based on multimodal fusion as described in any one of claims 1 to 7.

Images