Automatic positioning method for eyeground middle plasma leakage point

Abstract

The invention discloses an automatic positioning method for a slurry leakage point in a fundus, and particularly relates to the technical field of medical image processing, which comprises five steps of coordinate extraction, coordinate conversion, data enhancement, initial positioning and accurate positioning. A key point in a leakage point can be positioned with millisecond-level time consumption by using a YOLO model, coordinates of the key point are used as cue words to be sent into an SAM model to accurately segment a leakage point area, the advantages of rapid positioning of the YOLO model and accurate segmentation of the SAM model are combined, the rapid positioning capability of the YOLO model and the strong generalization capability of the SAM model are utilized, and the leakage point area can be accurately segmented. And the medium slurry leakage point in the FFA image is quickly and accurately positioned in a segmentation manner.

Description

Technical Field

[0001] The invention relates to the technical field of medical image processing, and in particular to a method for automatically locating plasma leakage points in fundus. Background Art

[0002] At present, the main treatments for mesothelioma are drug therapy and laser therapy. Laser therapy is an effective treatment for recurrent, chronic, and long-term mesothelioma, and is divided into traditional laser photocoagulation surgery and micro-pulse laser photocoagulation surgery. Among them, traditional laser photocoagulation surgery directly photocoagulates the serous leakage site in the mesothelioma edema area, that is, the leakage point, which is a local surgery in the mesothelioma edema area. It is suitable for mesothelioma patients with obvious leakage points in the fundus fluorescein angiography (FFA) image, and the leakage points are not under or near the fovea. Because the leakage point is near the macula, the laser needs to act precisely on the leakage point when acting on the leakage point, avoiding normal fundus tissue, otherwise it will cause irreversible damage to the patient's vision. It can be seen that the precise positioning of the mesothelioma leakage point is of great significance for guiding laser photocoagulation surgery.

[0003] For traditional laser photocoagulation surgery, through communication with ophthalmologists and combined with clinical feedback, it is found that in the current clinical laser surgery for mid-scleral leakage points, doctors are required to first confirm the approximate location of the mid-scleral leakage point in the FFA image before surgery, carefully check the edge of the leakage point, and then manually outline the leakage point from the background. There are the following disadvantages in the clinical positioning of mid-scleral leakage points: the manual positioning of the leakage point requires doctors to spend more energy, the positioning process has a low level of automation, and the repeatability of the positioning results is poor. Different doctors cannot achieve completely consistent segmentation of leakage points on the same fundus image.

[0004] In order to solve the shortcomings of clinical mid-sera leakage point positioning, the existing technology uses a leakage point joint positioning model based on the regional growing method and the initial positioning model to locate the leakage point, and has achieved good results. However, this method has many modules, is prone to introduce errors in the process of multiple steps, and the operation is relatively cumbersome. Summary of the invention

[0005] In order to address the shortcomings of clinical manual operation methods and existing research methods in the task of locating mid-sclerosing leakage points, improve the locating speed and accuracy of mid-sclerosing leakage points, and improve the automation level of locating mid-sclerosing leakage points, the present invention proposes an automatic locating method for mid-sclerosing leakage points in the fundus to solve the problems raised in the background technology.

[0006] The technical solution provided is as follows: a method for automatically locating the serous leakage point in the fundus, comprising the following steps:

[0007] 1) Coordinate extraction: LabelMe software is used to manually annotate the image of the leakage point in the slurry, wherein the image includes a labeling frame containing the leakage point area and a key point containing the central area of ​​the leakage point; a json format file is obtained from the annotated image, wherein the coordinates of the labeling frame and the key point annotated by the LabelMe software are stored in the json format file; the coordinates of the labeling frame and the key point are extracted by the program to generate the original data, and the original format coordinates are shown in (1):

[0008] (x1 y1 x2 y2 x kpt y kpt )(1)

[0009] Where x1 and y1 are the x and y coordinates of the upper left corner of the annotation box; x2 and y2 are the x and y coordinates of the lower right corner of the annotation box; kpt and kpt are the x and y coordinates of the key points; all coordinates are separated by spaces;

[0010] 2) Coordinate conversion: convert the original format coordinates (1) into the center format coordinates. The center format coordinates are the intermediate format from the original format coordinates to the label format coordinates. They are used to communicate the two formats for easy inspection and reading.

[0011] The conversion from the original format coordinates to the central format coordinates is performed using formula (2):

[0012]

[0013] The data required for the center format coordinates are obtained. The center format coordinates are as follows:

[0014] (x c y c wxya kpt y kpt )(3)

[0015] Among them, x c and c are the x and y coordinates of the center point of the annotation box, w and h are the width and height of the annotation box, and x kpt and kpt are the x and y coordinates of the key points, each separated by a space;

[0016] After the center format is normalized, add 0 before the coordinate to get the label format coordinate; the normalization formula is as follows:

[0017]

[0018] where n_x c , n_y c, n_w, n_h, n_x kpt , n_y kpt are the normalized x and y coordinates of the center point, the normalized width and height of the annotation box, and the normalized x and y coordinates of the key points; w pic and h pic are the width and height of the image;

[0019] The coordinates of the annotation box, the coordinates of the key points and the image are sent to the YOLO model for training. The label format required by the YOLO model is shown in formula (5):

[0020] (0n_x c n_y c n_w n_h n_x kpt n_y kpt )(5)

[0021] Among them, 0 is the class label required by the YOLO format label. The method only has one class of leakage points, so the label format starts with 0; n_x c ,n_y c ,n h ,n w ,n_x kpt ,n_y kpt The prefix n refers to normalize; n_x c ,n_y c refers to the normalized center coordinates of the annotation box; n_w, n_h refer to the normalized width and height of the annotation box; n_x kpt ,n_y kpt Refers to the normalized key point coordinates; after normalization, all coordinate values ​​are between [0,1];

[0022] 3) Data enhancement: Data enhancement includes data enhancement of images and labels. The following formulas (6) and (7) are used to enhance data and images simultaneously.

[0023]

[0024] Among them, T represents the translation operation, (x, y) represents the coordinate vector of the image translation; R represents the rotation operation, θ represents the rotation angle, and the rotation center is x c ,y c ,The angle θ of the rotation and translation operation and the translation vector (x, y) are shown in formula (7);

[0025]

[0026] The first row of formula (7) indicates that the image is rotated clockwise around the center of the image by 2, 4, 6, 8, 10, 12, 14, 16, 18, 20 degrees and counterclockwise around the center of the image by 2, 4, 6, 8, 10, 12, 14, 16, 18, 20 degrees; the second row of formula indicates that the image is translated 10 pixels to the right, 10 pixels to the left, 20 pixels upward, 20 pixels downward, 20 pixels to the right and 20 pixels upward at the same time;

[0027] After data enhancement, the training data set was expanded from the original 127 cases to 3175 cases, the validation data set was expanded from the original 39 cases to 975 cases, and the test data set was not enhanced, with 46 cases, one case refers to a leakage point map and a corresponding label; after data enhancement, the amplified label format required by YOLO was produced;

[0028] 4) Initial positioning: The amplified image and amplified labels are all sent to the YOLO model. After learning from a large amount of labeled data and corresponding images, the YOLO model predicts the coordinates of key points on the unlabeled image. The key point refers to any pixel in the leakage point area. The x and y coordinates of this pixel are used as the prompt words of the SAM model to guide segmentation. The output at this time is the label format described in formula (5), including the coordinates of the annotation box and the coordinates of the key points.

[0029] 5) Accurate positioning: After obtaining the predicted label format of YOLO in step 4), the normalized (x, y) coordinates of the key points are extracted as the prompt words of the SAM model, and then converted into absolute coordinates through inverse calculation. The inverse calculation formula is shown in formula (8):

[0030]

[0031] Among them, x kpt ,y kpt are the coordinates of the key points, n_x kpt ,n_y kpt is the normalized coordinate of the key point, w pic ,h pic is the width and height of the image; after inverse solution, the (x, y) coordinates of the key point in the image are sent to the SAM model. This coordinate is used as a prompt word to instruct the SAM model to segment the leakage point part of the image where the key point is located from other parts, that is, to complete the positioning of the leakage point, and then obtain the prompt segmentation map.

[0032] The present invention aims to solve the problem of accurate and rapid positioning of the slurry leakage point, and achieves the following beneficial effects in positioning the slurry leakage point:

[0033] 1. Quickly locate key points: The YOLO model can locate key points in the leakage point in milliseconds. The key point is any point in the leakage area. The coordinates of the key point are sent to the SAM model as prompts to accurately segment the leakage area.

[0034] 2. Accurately segment leakage points: The SAM model is a model that segments everything. It is pre-trained on a large number of images and can accurately segment various natural images and medical images, as well as leakage point areas.

[0035] 3. Combining the advantages of the YOLO model and the SAM model: Combining the advantages of the YOLO model's fast positioning with the SAM model's accurate segmentation, using the YOLO model's fast positioning capability and the SAM model's powerful generalization capability, the segmentation method is used to achieve fast and accurate positioning of the slurry leakage point in the FFA image. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A middle pulp leakage point map on the FFA image provided by the present invention;

[0037] Figure 2 A method roadmap provided for the present invention;

[0038] Figure 3 The middle slurry leakage point map annotated by Label Me software provided by the present invention;

[0039] Figure 4 A schematic diagram illustrating the original format coordinates provided by the present invention;

[0040] Figure 5 A schematic diagram of the conversion relationship between the original format, the central format and the label format provided by the present invention;

[0041] Figure 6 A schematic diagram of the central format provided by the present invention;

[0042] Figure 7 A roadmap for image and data enhancement provided by the present invention;

[0043] Figure 8 A schematic diagram of the YOLO model provided by the present invention;

[0044] Fig. 9 A schematic diagram of the SAM model provided by the present invention;

[0045] Fig.10 The final segmentation effect diagram provided by the present invention;

[0046] Fig.11 A step diagram of the implementation method provided by the present invention;

[0047] Fig.12 A manually annotated schematic diagram of the implementation method provided by the present invention;

[0048] Fig.13 The image and its annotation diagram of the implementation method provided by the present invention rotated 10 degrees counterclockwise;

[0049] Fig.14 The original image and its annotated image of the implementation method provided by the present invention;

[0050] Fig.15 The image and its annotation diagram of the implementation method provided by the present invention rotated 10 degrees clockwise;

[0051] Fig.16 A prompt segmentation diagram for the implementation method provided by the present invention;

[0052] Fig.17 This is a comparison chart of the segmentation effects of the implementation method provided by the present invention. DETAILED DESCRIPTION

[0053] The following is a description of the implementation of the present invention by specific embodiments. People familiar with the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] Fundus fluorescein angiography images of mid-scleral leakage points Figure 1 As shown, the small white dots in the red circles are the leakage points of the middle slurry.

[0055] In order to cope with the shortcomings of clinical manual operation methods and existing research methods in the task of locating mid-scleral leakage points, improve the positioning speed and accuracy of mid-scleral leakage points, and improve the automation level of mid-scleral leakage point positioning, the present invention proposes an automatic positioning method for mid-scleral leakage points in the fundus based on the YOLO model and the SAM (segment anythin model 1) model. This method combines the advantages of YOLO's rapid target recognition and the advantages of the SAM model's accurate image segmentation, which is conducive to improving the recognition speed and positioning accuracy of mid-scleral leakage points. This method has the following advantages:

[0056] 1. Rapid identification: The YOLO model is an open source algorithm YOLOv8, which has the ability to quickly detect targets. Applying it to the detection of medium slurry leakage points can achieve rapid identification of leakage points and lay the foundation for the precise positioning of subsequent leakage points.

[0057] 2. Accurate segmentation: SAM is an open source SAM model, which is well-known for its efficiency and accuracy and is widely used in the field of lesion segmentation in medical images. Applying it to the segmentation of leakage points can promote efficient and accurate segmentation of leakage points, thereby achieving precise positioning of leakage points.

[0058] 3. Simple and efficient: Both the YOLO model and the SAM model use pre-trained weights. The organic combination of the two models enables the entire algorithm to automatically process input data during operation and directly generate output results. Users do not need to perform complex configurations or multiple steps of operation. The algorithm is efficient, simple and easy to use.

[0059] The details are as follows:

[0060] like Figure 2 As shown, the present invention provides a method for automatically locating a serous leakage point in the fundus, comprising the following steps:

[0061] 1) Coordinate extraction: In order to train the YOLO model, a small number of images of medium-plasma leakage points need to be manually annotated. LabelMe software is used to manually annotate a small number of images of medium-plasma leakage points. The images include a labeling box containing the leakage point area and key points containing the central area of ​​the leakage point; Figure 3 The figure shows the leakage point in the pulp after being annotated by Label Me software. The green box is the annotation box, which includes the leakage point area; the red point is the key point, which is located in the center of the leakage point. The purpose of annotation is to train the YOLO model to produce a better leakage point identification effect. The json format file is obtained from the annotated image. The json format file stores the coordinates of the annotation box and the coordinates of the key points annotated by Label Me software;

[0062] Some of the text in the Json file is as follows:

[0063]

[0064] The array named rectangle stores the x, y coordinates (x1, y1) of the upper left corner of the annotation box, and the x, y coordinates (x2, y2) of the lower right corner; the array named lead_point stores the coordinates of the key points (x kpt ,y kpt ).

[0065] The coordinates of the annotation box and the key points are extracted by the program and converted into raw data. The raw format coordinates are shown in (1):

[0066] (x1 y1 x2 y2 x kpt y kpt )(1);

[0067] Where x1 and y1 are the x and y coordinates of the upper left corner of the annotation box; x2 and y2 are the x and y coordinates of the lower right corner of the annotation box; kpt and kpt are the x and y coordinates of the key points; all coordinates are separated by spaces; the positions of the coordinates are as follows Figure 4 As shown in the figure, the green is the annotation box, the upper left corner and the lower right corner of the annotation box are represented by green dots, and the red dots are key points, marking the white leakage points in the image.

[0068] 2) Coordinate conversion: Convert the original format coordinates (1) to the center format coordinates. The center format coordinates are the intermediate format from the original format coordinates to the label format coordinates. They are used to communicate the two formats for easy inspection and reading. The relationship between the three formats is as follows: Figure 5 shown.

[0069] The conversion from the original format coordinates to the central format coordinates is performed using formula (2):

[0070]

[0071] The data required for the center format coordinates are obtained. The center format coordinates are as follows:

[0072] (x c y c wxya kpt y kpt )(3);

[0073] Among them, x c and c are the x and y coordinates of the center point of the annotation box, w and h are the width and height of the annotation box, and x kpt and kpt are the x and y coordinates of the key points, each separated by a space; the central format diagram is as follows Figure 6 shown.

[0074] After the center format is normalized, add 0 before the coordinate to get the label format coordinate; the normalization formula is as follows:

[0075]

[0076] where n_x c , n_y c , n_w, n_h, n_x kpt , n_y kpt are the normalized x and y coordinates of the center point, the normalized width and height of the annotation box, and the normalized x and y coordinates of the key points; w pic and h pic are the width and height of the image;

[0077] The coordinates of the annotation box, the coordinates of the key points and the image are sent to the YOLO model for training. The label format required by the YOLO model is shown in formula (5):

[0078] (0n_x c n_y c n_w n_h n_x kpt n_y kpt )(5);

[0079] Among them, 0 is the class label required by the YOLO format label. The method only has one class of leakage points, so the label format starts with 0; n_x c ,n_y c ,n h ,n w ,n_x kpt ,n_y kpt The prefix n refers to normalize; n_x c ,n_y c refers to the normalized center coordinates of the annotation box; n_w, n_h refer to the normalized width and height of the annotation box; n_x kpt ,n_y kpt Refers to the normalized key point coordinates; after normalization, all coordinate values ​​are between [0,1];

[0080] 3) Data enhancement: Data enhancement refers to obtaining more data sets by performing the same translation and rotation operations on the labeled image and the labeled data, so that the YOLO model can obtain more learning samples and improve the YOLO model's recognition of leakage points. Data enhancement includes data enhancement of images and labels; the following formulas (6) and (7) are used to enhance data and images simultaneously;

[0081]

[0082] Among them, T represents the translation operation, (x, y) represents the coordinate vector of the image translation; R represents the rotation operation, θ represents the rotation angle, and the rotation center is x c ,y c ,The angle θ of the rotation and translation operation and the translation vector (x, y) are shown in formula (7);

[0083]

[0084] The first row of formula (7) indicates that the image is rotated clockwise around the center of the image by 2, 4, 6, 8, 10, 12, 14, 16, 18, 20 degrees and counterclockwise around the center of the image by 2, 4, 6, 8, 10, 12, 14, 16, 18, 20 degrees; the second row of formula indicates that the image is translated 10 pixels to the right, 10 pixels to the left, 20 pixels upward, 20 pixels downward, 20 pixels to the right and 20 pixels upward at the same time;

[0085] After data enhancement, the training data set was expanded from the original 127 cases to 3175 cases, the validation data set was expanded from the original 39 cases to 975 cases, and the test data set was not enhanced, with 46 cases, one case refers to a leakage point map and a corresponding label; after data enhancement, the amplified label format required by YOLO was produced; the route of image and data enhancement is as follows Figure 7 shown.

[0086] 4) Initial positioning: After the above steps, the data of YOLO positioning leakage points is completed. The amplified image and amplified labels are all sent to the YOLO model. After learning a large amount of labeled data and corresponding images, the YOLO model predicts the coordinates of key points on the unlabeled image. The key point refers to any pixel in the leakage point area. The x and y coordinates of this pixel are used as the prompt words of the SAM model to guide segmentation. The output at this time is the label format described in formula (5), including the coordinates of the annotation box and the coordinates of the key points. The role of the YOLO model is as follows: Figure 8 shown.

[0087] 5) Accurate positioning: The usage mechanism of the SAM model: This model requires a prompt word. The coordinates of a key point can be used as a prompt word and input into the SAM model together with the corresponding image to guide the model to identify and segment the specific object part indicated by the key point in the image;

[0088] After obtaining the predicted label format of YOLO in step 4), the normalized (x, y) coordinates of the key points are extracted as the prompt words of the SAM model, and then converted into absolute coordinates through inverse calculation. The inverse calculation formula is shown in formula (8):

[0089]

[0090] Among them, x kpt ,y kpt are the coordinates of the key points, n_x kpt ,n_y kpt is the normalized coordinate of the key point, w pic ,h picis the width and height of the image; after inverse solution, the (x, y) coordinates of the key point in the image are sent to the SAM model. This coordinate is used as a prompt word to instruct the SAM model to segment the leakage point part of the image where the key point is located from other parts, that is, to complete the location of the leakage point, and then obtain the prompt segmentation map. The role of the SAM model is as follows Fig. 9 shown.

[0091] The final segmentation effect is as follows Fig.10 shown. Fig.10 Three examples of leakage point segmentation effects are listed, one in each row. The three images in each row are the leakage points in the original image (white flocculent part), the manually annotated labels of the leakage points (white part), and the final segmentation effect (the red points are the key points sent to the SAM model, and the green parts are the leakage points segmented by the SAM model). In order to facilitate the observation of the segmentation effect, each image is cropped in the same way to enlarge the leakage point part.

[0092] In order to further illustrate the present invention, the specific implementation method has the following steps:

[0093] The entire implementation method steps are as follows Fig.11 shown.

[0094] 1. Manual labeling: Use LabelMe software to label the leakage points of multiple images in the form of labeling boxes and key points. Set the keyword "leak_point" to store the key point coordinates, and set the keyword "rectangle" to store the coordinates of the upper left corner and lower right corner of the labeling box. Use the mouse to drag to generate a labeling box, and use the labeling box to include the leakage point. Use the mouse to click any point in the leakage point area to generate a key point, and use the key point to indicate the location of the leakage point. Fig.12 As shown in the figure, the green box is the annotation box. The upper left corner and lower right corner of the green box are used to indicate the position and size of the annotation box. The red point is the key point for mouse click, which is used to indicate the position of the white leakage point. The white dot part in the figure is the leakage point of the medium pulp. Since the leakage point is very small in the whole image, this figure only shows the image near the leakage point to facilitate the viewing of the leakage point.

[0095] 2. Extract data: Extract the coordinates of the leakage point in the annotation box of the Label Me annotation data stored in the json file, and store them as original format coordinates in the format of formula. Use the program to extract the data corresponding to the keywords Leak_point and Rectangle in the json file and store them in the form of formula (1).

[0096] 3. Augmentation of the data set: randomly divide the images and the corresponding original format coordinates into training, validation, and test sets. According to formulas (6) and (7), the leakage point images and original format coordinates of the training and validation data sets are subjected to the same translation and rotation operations for data augmentation. No data augmentation operation is performed on the images and labels of the test set. Figure 13-Figure 15 As shown in the figure, the image after rotation and the original image and their annotations are shown, taking the counterclockwise rotation of 10 degrees and the clockwise rotation of 10 degrees as examples. The green annotation box and the red key point in the figure have also undergone the same rotation operation, so the red key point can accurately locate the leakage point after the image is rotated; the green annotation box can also accurately contain the leakage point after the image is rotated.

[0097] 4. Coordinate conversion: According to formula (2) and (4), the original format coordinates of the training, validation, and test sets after data enhancement in step 3 are converted into label format coordinates. Table 1 shows an example of coordinate conversion. The first row in the table is the original format coordinates, the second row is the center format coordinates, and the third row is the label format coordinates. In this example, the image width and height are both 768 pixels. The three coordinates can be calculated according to formula (2) and (4) and compared with the data in this table for verification.

[0098] Table 1 Examples of three coordinate formats

[0099] 401.213115396.704918452.032787452.442623428.672131425.803279 426.622951424.57377050.81967255.737705428.672131425.803279 00.5554990.5528300.0661710.0725750.5581670.554431

[0100] 5. Training the YOLO model: The amplified label format coordinates and the amplified leakage point image are input into the YOLO model for training. The output result of the YOLO model is the coordinates of the key points of the leakage point in the image of the test set, that is, the (x, y) coordinates of the key points output by YOLO after learning. If this key point is within the leakage point in the image, the leakage point is successfully identified; otherwise, it means that the YOLO model has not identified the leakage point.

[0101] 6. Hint segmentation: The (x, y) coordinates of the key points output by the YOLO model and the corresponding image are input into the SAM model, and the key point coordinates are used as hint words to prompt SAM to segment the leakage points. Fig.16 It is the prompt segmentation map; the red dot is the position of the prompt coordinates in the original image, indicating the position of the leakage point, and the green part is the SAM model that segments the leakage point from the background according to the segmentation mask of the prompt word. Fig.17 It is a comparison chart of the segmentation effects. The three rows of images are the prompt segmentation map, the manual annotation map and the original image respectively. For easy observation, the prompt segmentation map and the manual annotation result and the original image near the leakage point are magnified and displayed.

[0102] The present invention aims to make up for the shortcomings of existing mid-scleral leakage point positioning schemes and clinical manual leakage point positioning, provide technical support for assisting ophthalmologists in accurately and quickly positioning mid-scleral leakage points during subsequent operations, improve the accuracy and automation of traditional laser photocoagulation surgery, and enhance the treatment level of mid-scleral disease.

[0103] Although the present invention has been described in detail above by general description and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements can be made to the present invention. Therefore, these modifications or improvements made without departing from the spirit of the present invention all belong to the scope of protection claimed by the present invention.

Claims

1. A method for automatically locating the leakage point of the ocular fundus, characterized in that: By manually annotating the image of the pulp leakage point, the annotation box containing the leakage point area and the annotation file containing the key points of the central area of ​​the leakage point are obtained. The annotation file is converted into the original format coordinates to complete the coordinate extraction, and then the original format coordinates are converted into the central format and normalized to obtain the label format coordinates required by the YOLO model. After data enhancement of the image and label, the amplified label format required by YOLO is produced; the amplified image and the amplified label are all sent to the YOLO model. After learning a large amount of annotated data and corresponding images, the YOLO model predicts the coordinates of the key points on the unlabeled image for initial positioning; after obtaining the predicted label format of YOLO, the coordinates of the normalized key points are extracted, and after inverse solution, they are used as the prompt words of the SAM model and input into the SAM model together with the corresponding image. The prompt words indicate the position of the leakage point in the SAM model image, and the leakage point is separated from other parts, that is, the precise positioning of the leakage point is completed.

2. The method for automatically locating the serous leakage point in the fundus according to claim 1, characterized in that: Coordinate extraction includes the following steps: Manually annotate the image of the slurry leakage point, wherein the image includes an annotation frame including the leakage point area and a key point including the central area of ​​the leakage point; obtain a json format file from the annotated image, wherein the coordinates of the annotated annotation frame and the coordinates of the key points are stored in the json format file; the coordinates of the annotation frame and the coordinates of the key points are extracted by a program to form original data, and the original format coordinates are shown in formula (1): (x1 y1 x2 y2 x kpt y kpt )(1); Where x1 and y1 are the x and y coordinates of the upper left corner of the annotation box; x2 and y2 are the x and y coordinates of the lower right corner of the annotation box; kpt and kpt are the x, y coordinates of the key points; all coordinates are separated by spaces.

3. The method for automatically locating the leakage point of the fundus according to claim 2, characterized in that: Coordinate transformation and normalization include the following steps: Convert the obtained original format coordinates (1) into central format coordinates, where the central format coordinates are an intermediate format from the original format coordinates to the label format coordinates, and are used to communicate the two formats; The conversion from the original format coordinates to the central format coordinates is performed using formula (2): The data required for the center format coordinates are obtained. The center format coordinates are as follows: (x c y c w h x kpt y kpt )(3); Among them, x c and c are the x and y coordinates of the center point of the annotation box, w and h are the width and height of the annotation box, and x kpt and kpt are the x and y coordinates of the key points, each separated by a space; After the center format is normalized, 0 is added before the coordinates to obtain the label format coordinates; the normalization formula is as follows: where n_x c , n_y c , n_w, n_h, n_x kpt , n_y kpt are the normalized x and y coordinates of the center point, the normalized width and height of the annotation box, and the normalized x and y coordinates of the key points; w pic and h pic are the width and height of the image; The coordinates of the annotation box, the coordinates of the key points and the image are sent to the YOLO model for training. The label format required by the YOLO model is shown in formula (5): (0n_x c n_y c n_w n_h n_x kpt n_y kpt )(5); Among them, 0 is the class label required by the YOLO format label. The method only has one class of leakage points, so the label format starts with 0; n_x c ,n_y c ,n h ,n w ,n_x kpt ,n_y kpt The prefix n refers to normalize; n_x c ,n_y c refers to the normalized center coordinates of the annotation box; n_w, n_h refer to the normalized width and height of the annotation box; n_x kpt ,n_y kpt Refers to the normalized coordinates of the key points; after normalization, all coordinate values ​​are between [0,1].

4. The method for automatically locating the leakage point of the fundus according to claim 1, characterized in that: Data augmentation involves the following steps: Data enhancement includes data enhancement of images and labels; the following formulas (6) and (7) are used to enhance data and images simultaneously; Among them, T represents the translation operation, (x, y) represents the coordinate vector of the image translation; R represents the rotation operation, θ represents the rotation angle, and the rotation center is x c ,y c ,The angle θ of the rotation and translation operation and the translation vector (x, y) are shown in formula (7); The first row of formula (7) indicates that the image is rotated clockwise around the center of the image by 2, 4, 6, 8, 10, 12, 14, 16, 18, 20 degrees and counterclockwise around the center of the image by 2, 4, 6, 8, 10, 12, 14, 16, 18, 20 degrees; the second row of formula indicates that the image is translated 10 pixels to the right, 10 pixels to the left, 20 pixels upward, 20 pixels downward, 20 pixels to the right and 20 pixels upward at the same time; After data enhancement, the amplified label format required by YOLO is produced.

5. The method for automatically locating the serous leakage point in the fundus according to claim 3, characterized in that: Initial positioning includes the following steps: The amplified image and amplified labels are all sent to the YOLO model. After learning from a large amount of labeled data and corresponding images, the YOLO model predicts the coordinates of key points on the unlabeled image. The key point refers to any pixel in the leakage point area. The x and y coordinates of this pixel are used as the prompt words of the SAM model to guide segmentation. The output at this time is the label format described in formula (5), including the coordinates of the annotation box and the coordinates of the key points.

6. The method for automatically locating the serous leakage point in the fundus according to claim 5, characterized in that: Precise positioning includes the following steps: After obtaining the predicted label format of YOLO, the normalized (x, y) coordinates of the key points are extracted as the prompt words of the SAM model, and then converted into absolute coordinates through inverse calculation. The inverse calculation formula is shown in formula (8): Among them, x kpt ,y kpt are the coordinates of the key points, n_x kpt ,n_y kpt is the normalized coordinate of the key point, w pic ,h pic is the width and height of the image; after inverse solution, the (x, y) coordinates of the key point in the image are sent to the SAM model. This coordinate is used as a prompt word to instruct the SAM model to segment the leakage point part of the image where the key point is located from other parts, that is, to complete the positioning of the leakage point, and then obtain the prompt segmentation map.

Images