An automatic positioning method for plasma leakage points in an eye fundus

Abstract

The application discloses an automatic positioning method for plasma leakage points in an eye fundus, and particularly relates to the technical field of medical image processing, and comprises five steps of coordinate extraction, coordinate conversion, data enhancement, initial positioning and accurate positioning. The YOLO model can be used to position key points in the leakage points with a time consumption of milliseconds, the coordinates of the key points are sent into the SAM model as prompt words, and the leakage point region is accurately segmented, the advantages of the YOLO model positioning quickly and the SAM model segmenting accurately are combined, the rapid positioning capacity of the YOLO model and the strong generalization capacity of the SAM model are utilized, and the rapid and accurate positioning of the plasma leakage points in the FFA image is realized in a segmentation mode.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a method for automatically locating plasma leakage points in the fundus. Background Art

[0002] Currently, the main treatments for mesothelioma are medication and laser therapy. Laser therapy is an effective treatment for recurrent, chronic, and long-term mesothelioma, and is divided into traditional laser photocoagulation surgery and micropulse laser photocoagulation surgery. Among them, traditional laser photocoagulation surgery directly photocoagulates the serous leakage site in the mesothelioma edema area, that is, the leakage point. It is a local surgery in the mesothelioma edema area and is suitable for mesothelioma patients with obvious leakage points visible in fundus fluorescein angiography (FFA) images, and the leakage points are not below or near the fovea. Because the leakage points are located near the macula, when the laser is applied to the leakage points, it needs to act precisely on the leakage points and avoid normal fundus tissue, otherwise it will cause irreversible damage to the patient's vision. Therefore, the precise positioning of the mesothelioma leakage points is of great significance in guiding laser photocoagulation surgery.

[0003] Regarding traditional laser photocoagulation surgery, discussions with ophthalmologists and clinical feedback revealed that, prior to surgery, the physician must first confirm the approximate location of the mid-scleral leakage point in the FFA image, carefully examine the edge of the leakage point, and then manually segment the leakage point from the background. Clinically, locating mid-scleral leakage points has the following drawbacks: manual location of the leakage point requires considerable effort, the automation level of the location process is low, and the repeatability of the location results is poor. Different physicians may not be able to achieve completely consistent segmentation of the leakage point on the same fundus image.

[0004] In order to solve the shortcomings of clinical mid-sera leakage point positioning, the existing technology uses a leakage point joint positioning model based on the combination of region growing method and initial positioning model to locate the leakage point, which has achieved good results. However, this method has many modules, is prone to introduce errors in the process of multiple steps, and the operation is relatively cumbersome. Summary of the Invention

[0005] In order to address the shortcomings of clinical manual operation methods and existing research methods in the task of locating mid-sclerosing leakage points, improve the positioning speed and accuracy of mid-sclerosing leakage points, and improve the automation level of mid-sclerosing leakage point positioning, the present invention proposes an automatic positioning method for mid-sclerosing leakage points in the fundus to solve the problems raised in the background technology.

[0006] The technical solution provided is as follows: a method for automatically locating the serous leakage point in the fundus, comprising the following steps:

[0007] 1) Coordinate extraction: LabelMe software is used to manually annotate the image of the slurry leakage point, and the image includes a labeling box containing the leakage point area and a key point containing the central area of ​​the leakage point; a json format file is obtained from the annotated image, and the json format file stores the coordinates of the labeling box and the coordinates of the key points annotated by the LabelMe software; the coordinates of the labeling box and the coordinates of the key points are extracted by the program to generate the original data, and the original format coordinates are shown as (1):

[0008] (x1 y1 x2 y2 x kpt y kpt )(1)

[0009] Among them, x1 and y1 are the x and y coordinates of the upper left corner of the annotation box; x2 and y2 are the x and y coordinates of the lower right corner of the annotation box; kpt and y kpt are the x and y coordinates of the key points; all coordinates are separated by spaces;

[0010] 2) Coordinate conversion: convert the original format coordinates (1) into the center format coordinates. The center format coordinates are the intermediate format from the original format coordinates to the label format coordinates, which are used to communicate the two formats for easy inspection and reading.

[0011] The conversion from the original format coordinates to the center format coordinates is performed using formula (2):

[0012]

[0013] The data required to obtain the center format coordinates are as follows:

[0014] (x c y c whx kpt y kpt )(3)

[0015] Among them, x c and y c They are the x and y coordinates of the center point of the annotation box, w and h are the width and height of the annotation box, x kpt and y kpt are the x and y coordinates of the key points, each separated by a space;

[0016] After the center format is normalized, 0 is added before the coordinates to obtain the label format coordinates; the normalization formula is as follows:

[0017]

[0018] where n_x c , n_y c, n_w, n_h, n_x kpt , n_y kpt are the normalized x and y coordinates of the center point, the normalized width and height of the annotation box, and the normalized x and y coordinates of the key points; w pic and h pic are the width and height of the image;

[0019] The coordinates of the annotation box, the coordinates of the key points and the image are sent to the YOLO model for training. The label format required by the YOLO model is shown in formula (5):

[0020] (0n_x c n_y c n_w n_h n_x kpt n_y kpt )(5)

[0021] Among them, 0 is the class label required by the YOLO format label. The method only has one class of leakage points, so the label format starts with 0; n_x c ,n_y c ,n h ,n w ,n_x kpt ,n_y kpt The prefix n refers to normalize; n_x c ,n_y c Refers to the coordinates of the center point of the normalized annotation box; n_w, n_h refer to the width and height of the normalized annotation box; n_x kpt ,n_y kpt Refers to the normalized coordinates of the key points; after normalization, all coordinate values ​​are between [0,1];

[0022] 3) Data enhancement: Data enhancement includes data enhancement of images and labels; the following formulas (6) and (7) are used to enhance data and images simultaneously;

[0023]

[0024] Among them, T represents the translation operation, (x, y) represents the coordinate vector of the image translation; R represents the rotation operation, θ represents the rotation angle, and the rotation center is x c ,y c ,The angle θ of the rotation and translation operation and the translation vector (x, y) are shown in Equation (7);

[0025]

[0026] The first row of formula (7) represents rotating the image clockwise around the image center by 2, 4, 6, 8, 10, 12, 14, 16, 18, and 20 degrees, and counterclockwise around the image center by 2, 4, 6, 8, 10, 12, 14, 16, 18, and 20 degrees. The second row of formula represents translating the image 10 pixels to the right, 10 pixels to the left, 20 pixels up, 20 pixels down, 20 pixels to the right, and 20 pixels up at the same time.

[0027] After data augmentation, the training dataset was expanded from 127 cases to 3175 cases, the validation dataset was expanded from 39 cases to 975 cases, and the test dataset was not augmented, with 46 cases. One case refers to a leakage point map and a corresponding label. After data augmentation, the augmented label format required by YOLO was produced.

[0028] 4) Initial positioning: The amplified image and amplified labels are all fed into the YOLO model. After learning from a large amount of labeled data and corresponding images, the YOLO model predicts the coordinates of key points on the unlabeled image. A key point refers to any pixel within the leakage point area. The x and y coordinates of this pixel serve as the prompt word for the SAM model to guide segmentation. The output at this time is the label format described in Equation (5), including the coordinates of the annotation box and the coordinates of the key points.

[0029] 5) Accurate positioning: After obtaining the predicted label format of YOLO in step 4), the normalized (x, y) coordinates of the key points are extracted as the prompt words of the SAM model, and then converted into absolute coordinates through inverse calculation. The inverse calculation formula is shown in formula (8):

[0030]

[0031] Among them, x kpt ,y kpt is the coordinate of the key point, n_x kpt ,n_y kpt is the normalized coordinate of the key point, w pic ,h pic is the width and height of the image; after inverse calculation, the (x, y) coordinates of the key point in the image are sent to the SAM model. This coordinate serves as a prompt word to instruct the SAM model to segment the leakage point part of the image where the key point is located from other parts, that is, to complete the positioning of the leakage point and then obtain the prompt segmentation map.

[0032] The present invention aims to solve the problem of accurate and rapid positioning of slurry leakage points, and achieves the following beneficial effects in positioning the slurry leakage points:

[0033] 1. Quickly locate key points: The YOLO model can locate key points in a leak in milliseconds. A key point is any point in the leak area. The coordinates of the key point are fed into the SAM model as prompts to accurately segment the leak area.

[0034] 2. Accurately segment leakage points: The SAM model is a model that segments everything. It is pre-trained on a large number of images and can accurately segment various natural and medical images, as well as leakage point areas.

[0035] 3. Combining the advantages of the YOLO model and the SAM model: This method combines the advantages of the YOLO model's fast positioning with the SAM model's accurate segmentation. Leveraging the YOLO model's fast positioning capabilities and the SAM model's powerful generalization capabilities, this method uses segmentation to quickly and accurately locate slurry leakage points in FFA images. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A diagram of pulp leakage points on the FFA image provided by the present invention;

[0037] Figure 2 A roadmap for the method provided by the present invention;

[0038] Figure 3 The middle slurry leakage point map marked by Label Me software provided by the present invention;

[0039] Figure 4 A schematic diagram illustrating the original format coordinates provided by the present invention;

[0040] Figure 5 A schematic diagram of the conversion relationship between the original format, central format and label format provided by the present invention;

[0041] Figure 6 A schematic diagram of the central format provided by the present invention;

[0042] Figure 7 A roadmap for image and data enhancement provided by this invention;

[0043] Figure 8 Schematic diagram of the YOLO model provided by the present invention;

[0044] Figure 9 Schematic diagram of the SAM model provided by the present invention;

[0045] Figure 10 The final segmentation effect diagram provided by the present invention;

[0046] Figure 11 A step diagram of the implementation method provided by the present invention;

[0047] Figure 12 A manually annotated schematic diagram of the implementation method provided by the present invention;

[0048] Figure 13 An image and its annotation diagram of the implementation method provided by the present invention rotated 10 degrees counterclockwise;

[0049] Figure 14 The original diagram and its annotated diagram of the implementation method provided by the present invention;

[0050] Figure 15 The image and its annotation diagram of the implementation method provided by the present invention rotated 10 degrees clockwise;

[0051] Figure 16 A prompt segmentation diagram for the implementation method provided by the present invention;

[0052] Figure 17 This is a comparison chart of the segmentation effects of the implementation method provided by the present invention. DETAILED DESCRIPTION

[0053] The following describes the implementation of the present invention using specific embodiments. Those skilled in the art will readily understand the other advantages and benefits of the present invention from the disclosure herein. Obviously, the embodiments described are only a portion of the present invention, not all of it. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are intended to fall within the scope of protection of the present invention.

[0054] Fundus fluorescein angiography images of mid-scleral leakage points Figure 1 As shown in the figure, the small white dots in the red circle are the leakage points of the slurry.

[0055] In order to address the shortcomings of clinical manual operation methods and existing research methods in the task of locating mid-scleral leakage points, improve the positioning speed and accuracy of mid-scleral leakage points, and improve the automation level of mid-scleral leakage point positioning, the present invention proposes an automatic positioning method for mid-scleral leakage points in the fundus based on the YOLO model and the SAM (segment anything model) model. This method combines the advantages of YOLO's rapid target recognition and the advantages of the SAM model's accurate image segmentation, which is conducive to improving the recognition speed and positioning accuracy of mid-scleral leakage points. This method has the following advantages:

[0056] 1. Rapid identification: The YOLO model is an open-source algorithm, YOLOv8, which has the ability to quickly detect targets. Applying it to the detection of slurry leakage points can achieve rapid identification of leakage points, laying the foundation for subsequent precise positioning of leakage points.

[0057] 2. Accurate Segmentation: SAM is an open-source SAM model, renowned for its efficiency and accuracy, and widely used in medical imaging lesion segmentation. Applying it to the segmentation of leak points can facilitate efficient and accurate segmentation, thereby enabling precise location of leak points.

[0058] 3. Simple and efficient: Both the YOLO model and the SAM model use pre-trained weights. The organic combination of the two models enables the entire algorithm to automatically process input data and directly generate output results during runtime. Users do not need to perform complex configuration or multi-step operations. The algorithm is efficient, simple, and easy to use.

[0059] The details are as follows:

[0060] like Figure 2 As shown, the present invention provides a method for automatically locating a serous leakage point in the fundus, comprising the following steps:

[0061] 1) Coordinate extraction: In order to train the YOLO model, a small number of images of pulp leakage points need to be manually annotated. LabelMe software is used to manually annotate a small number of images of pulp leakage points. The images include a label box containing the leakage point area and key points containing the central area of ​​the leakage point; Figure 3 The image shows a mid-slurry leak after being annotated using Label Me. The green box is the annotation frame, enclosing the leak area; the red dot is the key point, located in the center of the leak area. The purpose of annotation is to train the YOLO model for better leak point identification. A JSON file is generated from the annotated image, which contains the coordinates of the annotation frame and key points annotated by Label Me.

[0062] Some of the text in the Json file is as follows:

[0063]

[0064] The array named rectangle stores the x, y coordinates (x1, y1) of the upper left corner of the annotation box, and the x, y coordinates (x2, y2) of the lower right corner; the array named l eak_point stores the coordinates of the key points (x kpt ,y kpt ).

[0065] The coordinates of the annotation box and the key points are extracted by the program and converted into original data. The original format coordinates are shown in (1):

[0066] (x1 y1 x2 y2 x kpt y kpt )(1);

[0067] Among them, x1 and y1 are the x and y coordinates of the upper left corner of the annotation box; x2 and y2 are the x and y coordinates of the lower right corner of the annotation box; kpt and y kpt are the x and y coordinates of the key points; all coordinates are separated by spaces; the coordinate positions are as follows Figure 4 As shown in the figure, the green is the annotation box, the upper left corner and the lower right corner of the annotation box are represented by green dots, and the red dots are key points, marking the white leakage points in the image.

[0068] 2) Coordinate conversion: Convert the original format coordinates (1) into the center format coordinates. The center format coordinates are the intermediate format from the original format coordinates to the label format coordinates. They are used to communicate the two formats for easy inspection and reading. The relationship between the three formats is as follows: Figure 5 shown.

[0069] The conversion from the original format coordinates to the center format coordinates is performed using formula (2):

[0070]

[0071] The data required to obtain the center format coordinates are as follows:

[0072] (x c y c whx kpt y kpt )(3);

[0073] Among them, x c and y c They are the x and y coordinates of the center point of the annotation box, w and h are the width and height of the annotation box, x kpt and y kpt are the x and y coordinates of the key points, with spaces between each coordinate; the center format diagram is as follows Figure 6 shown.

[0074] After the center format is normalized, 0 is added before the coordinates to obtain the label format coordinates; the normalization formula is as follows:

[0075]

[0076] where n_x c , n_y c , n_w, n_h, n_x kpt , n_y kpt are the normalized x and y coordinates of the center point, the normalized width and height of the annotation box, and the normalized x and y coordinates of the key points; w pic and h pic are the width and height of the image;

[0077] The coordinates of the annotation box, the coordinates of the key points and the image are sent to the YOLO model for training. The label format required by the YOLO model is shown in formula (5):

[0078] (0n_x c n_y c n_w n_h n_x kpt n_y kpt )(5);

[0079] Among them, 0 is the class label required by the YOLO format label. The method only has one class of leakage points, so the label format starts with 0; n_x c ,n_y c ,n h ,n w ,n_x kpt ,n_y kpt The prefix n refers to normalize; n_x c ,n_y c Refers to the coordinates of the center point of the normalized annotation box; n_w, n_h refer to the width and height of the normalized annotation box; n_x kpt ,n_y kpt Refers to the normalized coordinates of the key points; after normalization, all coordinate values ​​are between [0,1];

[0080] 3) Data enhancement: Data enhancement refers to performing the same translation and rotation operations on the labeled image and the labeled data to obtain more data sets, so that the YOLO model can obtain more learning samples and improve the YOLO model's recognition of leakage points. Data enhancement includes data enhancement of images and labels; the following formulas (6) and (7) are used to enhance both data and images simultaneously;

[0081]

[0082] Among them, T represents the translation operation, (x, y) represents the coordinate vector of the image translation; R represents the rotation operation, θ represents the rotation angle, and the rotation center is x c ,y c ,The angle θ of the rotation and translation operation and the translation vector (x, y) are shown in Equation (7);

[0083]

[0084] The first row of formula (7) represents rotating the image clockwise around the image center by 2, 4, 6, 8, 10, 12, 14, 16, 18, and 20 degrees, and counterclockwise around the image center by 2, 4, 6, 8, 10, 12, 14, 16, 18, and 20 degrees. The second row of formula represents translating the image 10 pixels to the right, 10 pixels to the left, 20 pixels up, 20 pixels down, 20 pixels to the right, and 20 pixels up at the same time.

[0085] After data augmentation, the training dataset was expanded from 127 cases to 3175 cases, the validation dataset was expanded from 39 cases to 975 cases, and the test dataset was not augmented, with 46 cases. One case refers to a leakage point map and a corresponding label. After data augmentation, the augmented label format required by YOLO was produced. The image and data augmentation route is as follows: Figure 7 shown.

[0086] 4) Initial positioning: After the above steps, the data of YOLO positioning leakage point is completed. The amplified image and amplified label are all sent to the YOLO model. After learning a large amount of labeled data and corresponding images, the YOLO model predicts the coordinates of key points on the unlabeled image. The key point refers to any pixel in the leakage point area. The x and y coordinates of this pixel are used as the prompt words of the SAM model to guide segmentation. The output at this time is the label format described in formula (5), including the coordinates of the annotation box and the coordinates of the key points. The role of the YOLO model is as follows: Figure 8 shown.

[0087] 5) Accurate Positioning: How the SAM model works: This model requires a prompt word. The coordinates of a keypoint can be used as a prompt word and input into the SAM model along with the corresponding image to guide the model to identify and segment the specific object part indicated by the keypoint in the image.

[0088] After obtaining the predicted label format of YOLO in step 4), the normalized (x, y) coordinates of the key points are extracted as the prompt words of the SAM model, and then converted into absolute coordinates through inverse calculation. The inverse calculation formula is shown in formula (8):

[0089]

[0090] Among them, x kpt ,y kpt is the coordinate of the key point, n_x kpt ,n_y kpt is the normalized coordinate of the key point, w pic ,h picis the width and height of the image; after the inverse solution, the (x, y) coordinates of the key point in the image are fed into the SAM model. This coordinate serves as a prompt word to instruct the SAM model to segment the leakage point part of the image where the key point is located from the other parts, that is, to complete the location of the leakage point and then obtain the prompt segmentation map. The role of the SAM model is as follows Figure 9 shown.

[0091] The final segmentation effect is as follows Figure 10 shown. Figure 10 Three examples of leak point segmentation are shown, one per row. Each row of three images shows the leak point in the original image (the white flocculent part), the manually annotated leak point (the white part), and the final segmentation result (the red dot represents the key point fed into the SAM model, and the green part represents the leak point segmented by the SAM model). To facilitate visualization of the segmentation results, each image has been cropped to magnify the leak point.

[0092] In order to further illustrate the present invention, the specific implementation method has the following steps:

[0093] The entire implementation method steps are as follows Figure 11 shown.

[0094] 1. Manual labeling: Use LabelMe software to label the leakage points of multiple images in the form of label boxes and key points. Set "leak_point" as the keyword to store the coordinates of the key points, and set "rectangle" as the keyword to store the coordinates of the upper left and lower right corners of the label box. Use the mouse to drag to create a label box, and use the label box to include the leakage point. Use the mouse to click any point in the leakage point area to generate a key point, and use the key point to indicate the location of the leakage point. Figure 12 As shown, the green box is the annotation box. The upper left and lower right corners of the green box indicate the position and size of the annotation box. The red dot is the key point for mouse click, which indicates the location of the white leakage point. The white dot in the figure is the medium pulp leakage point. Because the leakage point is very small in the entire image, this figure only shows the image near the leakage point to facilitate the identification of the leakage point.

[0095] 2. Extract data: Extract the coordinates of the leakage points from the annotation data of Label Me stored in the json file and store them in the original format using the formula. Use the program to extract the data corresponding to the keywords label_point int and rectangl e in the json file and store them in the form of formula (1).

[0096] 3. Augmentation of the dataset: Randomly divide the images and their original format coordinates into training, validation, and test sets. According to formulas (6) and (7), the images and original format coordinates of the training and validation datasets are subjected to the same translation and rotation operations for data augmentation. No data augmentation is performed on the images and labels of the test set. Figure 13-15 The following figure shows the rotated image, the original image, and its annotations, using a 10-degree counterclockwise rotation and a 10-degree clockwise rotation as examples. The green annotation box and red key points in the figure also undergo the same rotation operation. Therefore, the red key points can accurately locate the leakage point after the image is rotated; the green annotation box can also accurately contain the leakage point after the image is rotated.

[0097] 4. Coordinate transformation: According to equations (2) and (4), the original format coordinates of the training, validation, and test sets after data augmentation in step 3 are transformed into labeled format coordinates. Table 1 shows an example of coordinate transformation. The first row in the table is the original format coordinates, the second row is the center format coordinates, and the third row is the labeled format coordinates. In this example, the image width and height are both 768 pixels. The three coordinates can be calculated according to equations (2) and (4) and compared with the data in this table for verification.

[0098] Table 1 Examples of three coordinate formats

[0099] 401.213115396.704918452.032787452.442623428.672131425.803279 426.622951424.57377050.81967255.737705428.672131425.803279 00.5554990.5528300.0661710.0725750.5581670.554431

[0100] 5. Training the YOLO model: The amplified label format coordinates and the amplified leakage point image are input into the YOLO model for training. The output of the YOLO model is the coordinates of the key points of the leakage point in the test set image, that is, the (x, y) coordinates of the key points output by YOLO after learning. If this key point is within the leakage point in the image, the leakage point is successfully identified; otherwise, it means that the YOLO model has not identified the leakage point.

[0101] 6. Hint segmentation: The (x, y) coordinates of the key points output by the YOLO model and the corresponding image are input into the SAM model. The key point coordinates are used as hint words to prompt SAM to segment the leakage points. Figure 16 It is the prompt segmentation map; the red dot is the position of the prompt coordinates in the original image, indicating the location of the leakage point, and the green part is the SAM model segmenting the leakage point from the background according to the segmentation mask of the prompt word. Figure 17 This is a comparison chart of the segmentation effects. The three rows of images are the prompt segmentation map, the manually labeled map, and the original image. For easy observation, the prompt segmentation map near the leakage point, the manual labeling results, and the original image are magnified and displayed.

[0102] The present invention aims to make up for the shortcomings of existing mid-scleral leakage point positioning schemes and clinical manual positioning of leakage points, provide technical support for assisting ophthalmologists in accurately and quickly positioning mid-scleral leakage points during subsequent operations, improve the accuracy and automation of traditional laser photocoagulation surgery, and enhance the treatment level of mid-scleral disease.

[0103] Although the present invention has been described in detail above using general descriptions and specific embodiments, it will be apparent to those skilled in the art that modifications and improvements may be made thereto. Therefore, such modifications and improvements, without departing from the spirit of the present invention, are intended to be within the scope of protection claimed herein.

Claims

1. A method for automatically locating a slurry leakage point in the fundus, characterized by: By manually annotating the image of the pulp leakage point, a labeling box containing the leakage point area and a labeling file containing the key points of the central area of ​​the leakage point are obtained. The labeling file is converted into the original format coordinates to complete the coordinate extraction. The original format coordinates are then converted into the center format and normalized to obtain the label format coordinates required by the YOLO model. After data enhancement of the image and label, the amplified label format required by YOLO is produced; the amplified image and amplified label are all sent to the YOLO model. After learning the annotated data and the corresponding image, the YOLO model predicts the coordinates of the key points on the unlabeled image for initial positioning; after obtaining the predicted label format of YOLO, the coordinates of the normalized key points are extracted, and after inverse calculation, they are used as the prompt words of the SAM model and input into the SAM model together with the corresponding image. The prompt words indicate the position of the leakage point in the SAM model image, and the leakage point is separated from other parts, thus completing the precise positioning of the leakage point; Among them, the label format required by the YOLO model is shown in formula (1): (0n_x c n_y c n_w n_h n_x kpt n_y kpt )(1); Among them, 0 is the class label required by the YOLO format label. The method only has one class of leakage points, so the label format starts with 0; n_x c ,n_y c ,n h ,n w ,n_x kpt ,n_y kpt The prefix n refers to normalize; n_x c ,n_y c Refers to the coordinates of the center point of the normalized annotation box; n_w, n_h refer to the width and height of the normalized annotation box; n_x kpt ,n_y kpt Refers to the normalized coordinates of the key points; after normalization, all coordinate values ​​are between [0,1]; The initial positioning includes the following steps: The amplified image and amplified labels are all fed into the YOLO model. After learning from the labeled data and the corresponding image, the YOLO model predicts the coordinates of the key points on the unlabeled image. The key point refers to any pixel in the leakage point area. The x and y coordinates of this pixel are used as the prompt words of the SAM model to guide the segmentation. The output at this time is the label format described in formula (1), including the coordinates of the labeled box and the coordinates of the key points. Among them, precise positioning includes the following steps: After obtaining the predicted label format of YOLO, the normalized (x, y) coordinates of the key points are extracted as the prompt words of the SAM model, and then converted into absolute coordinates through inverse calculation. The inverse calculation formula is shown in formula (2): Among them, x kpt ,y kpt is the coordinate of the key point, n_x kpt ,n_y kpt is the normalized coordinate of the key point, w pic ,h pic is the width and height of the image; after inverse calculation, the (x, y) coordinates of the key point in the image are sent to the SAM model. This coordinate serves as a prompt word to instruct the SAM model to segment the leakage point part of the image where the key point is located from other parts, that is, to complete the positioning of the leakage point and then obtain the prompt segmentation map.

2. The method for automatically locating a slurry leakage point in the fundus according to claim 1, characterized in that: Coordinate extraction includes the following steps: Manually mark the image of the slurry leakage point, the image includes a marking frame containing the leakage point area and a key point containing the central area of ​​the leakage point; obtain a json format file from the marked image, the json format file stores the coordinates of the marked marking frame and the coordinates of the key points; the coordinates of the marked frame and the coordinates of the key points are extracted by the program to generate raw data, and the raw format coordinates are shown in formula (3): (x1 y1 x2 y2 x kpt y kpt )(3); Among them, x1 and y1 are the x and y coordinates of the upper left corner of the annotation box; x2 and y2 are the x and y coordinates of the lower right corner of the annotation box; kpt and y kpt are the x, y coordinates of the key points; all coordinates are separated by spaces.

3. The method for automatically locating a slurry leakage point in the fundus according to claim 2, characterized in that: Coordinate transformation and normalization include the following steps: Convert the original format coordinates (3) to the center format coordinates. The center format coordinates are an intermediate format from the original format coordinates to the label format coordinates, and are used to communicate the two formats. The conversion from the original format coordinates to the center format coordinates is performed using formula (4): The data required to obtain the center format coordinates are as follows: (x c y c w h x kpt y kpt )(5); Among them, x c and y c They are the x and y coordinates of the center point of the annotation box, w and h are the width and height of the annotation box, x kpt and y kpt are the x and y coordinates of the key points, each separated by a space; After the center format is normalized, 0 is added before the coordinates to obtain the label format coordinates; the normalization formula is as follows: where n_x c , n_y c , n_w, n_h, n_x kpt , n_y kpt are the normalized x and y coordinates of the center point, the normalized width and height of the annotation box, and the normalized x and y coordinates of the key points; w pic and h pic are the width and height of the image; The coordinates of the annotation box, the coordinates of the key points and the image are sent to the YOLO model for training.

4. The method for automatically locating a slurry leakage point in the fundus according to claim 1, characterized in that: Data augmentation involves the following steps: Data enhancement includes data enhancement of images and labels; the following formulas (7) and (8) are used to enhance data and images simultaneously; Among them, T represents the translation operation, (x, y) represents the coordinate vector of the image translation; R represents the rotation operation, θ represents the rotation angle, and the rotation center is x c ,y c ,The angle θ of the rotation and translation operation and the translation vector (x, y) are shown in Equation (8); The first row of formula (8) represents rotating the image clockwise around the image center by 2, 4, 6, 8, 10, 12, 14, 16, 18, and 20 degrees, and counterclockwise around the image center by 2, 4, 6, 8, 10, 12, 14, 16, 18, and 20 degrees. The second row of formula represents translating the image 10 pixels to the right, 10 pixels to the left, 20 pixels up, 20 pixels down, 20 pixels to the right, and 20 pixels up at the same time. After data enhancement, the amplified label format required by YOLO is produced.

Images