Fundus Image-based Lesion Analysis Method and Device

Abstract

The present invention relates to the technical field of image processing, and specifically discloses a lesion analysis method and device based on fundus images, including: obtaining a first fundus image set and a second fundus image set; performing image stitching on the first fundus image set and the second fundus image set to obtain a first stitched fundus image and a second stitched fundus image; performing intersection analysis on the second stitched fundus image based on the first stitched fundus image, and processing the first stitched fundus image and the second stitched fundus image based on the intersection analysis result to obtain a first intersection image and a second intersection image; cropping the first intersection image and the second intersection image to obtain a first cropped image set and a second cropped image set; performing lesion analysis on the first cropped image set and the second cropped image set to generate first analysis data and second analysis data; generating a lesion analysis result based on the first analysis data and the second analysis data. The present invention improves the accuracy and reliability of lesion recognition.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and particularly to a lesion analysis method and device based on fundus images. Background Art

[0002] As one of the serious complications of diabetes, diabetic retinopathy (DR) often leads to irreversible blindness if not detected and treated in time, with extremely serious consequences. If it can be identified in the early and middle stages of the disease and intervened and treated in a timely manner according to different stages of the disease, the vision of patients can often be saved and the quality of life can be improved.

[0003] The Early Treatment Diabetic Retinopathy Study (ETDRS) standard seven-field (7SF) color fundus photography has always been the preferred method for fundus imaging of diabetic patients. Diabetic retinopathy is one of the most common complications in diabetic patients, which can lead to vision loss and blindness. Currently, the ETDRS 7-field imaging technology is usually used to evaluate the severity of diabetic retinopathy. This method is simple and more sensitive, and can accurately evaluate the degree of diabetic retinopathy by combining an automated algorithm, with higher accuracy and more accurate assessment of the condition. However, currently, this method relies on the human eye to roughly estimate the number of lesions and then give a rating, which leads to the consumption of professional human resources. When the image is at the grade boundary, the grading subjectivity is relatively large, and different people may have different grading results, which leads to a decrease in the credibility of the results.

[0004] In order to solve the above technical problems, technical personnel considered using an intelligent model to analyze and identify lesions in fundus photos. However, in the actual application process, on the one hand, due to the large difference between the training images and the actual input images, the recognition effect of the model is poor; on the other hand, the model analyzes the lesions based on fundus images taken at different times, but the fundus images taken at different times have deviations due to reasons such as jitter, resulting in huge differences in the lesion recognition results, which has a great impact on the recognition results and cannot meet the actual needs. Summary of the Invention

[0005] In order to overcome the above technical problems existing in the prior art, the embodiments of the present invention provide a lesion analysis method and device based on fundus images. By analyzing the actual problems existing in the lesion analysis of fundus images by an intelligent model, consistency processing is carried out in each link such as image acquisition, analysis area determination, and image cropping, ensuring the accuracy of lesion analysis.

[0006] To achieve the above object, an embodiment of the present invention provides a lesion analysis method based on fundus images, and the method includes: obtaining a first fundus image set and a second fundus image set, where the first fundus image set is taken at the initial stage of fundus detection, and the second fundus image set is taken at a subsequent stage of fundus detection; determining the stitching priority of each image in the first fundus image set and the second fundus image set; performing image stitching on the first fundus image set and the second fundus image set respectively based on the stitching priority to obtain corresponding first stitched fundus images and second stitched fundus images; performing intersection analysis on the second stitched fundus image based on the first stitched fundus image, processing the first stitched fundus image based on the intersection analysis result to obtain a first intersection image, and processing the second stitched fundus image based on the intersection analysis result to obtain a second intersection image; respectively cropping the first intersection image and the second intersection image to obtain corresponding first cropped image sets and second cropped image sets, respectively performing lesion analysis on the first cropped image sets and the second cropped image sets to generate corresponding first analysis data and second analysis data; generating a lesion analysis result based on the first analysis data and the second analysis data.

[0007] Preferably, determining the stitching priority of each image in the first fundus image set and the second fundus image set includes: determining the initial stitching order of each image, and determining the stitching weight of each image based on the initial stitching order; evaluating the lesion quality of each image to determine the corresponding quality evaluation result; determining the stitching priority of each image in the first fundus image set and the second fundus image set based on the stitching weight and the quality evaluation result.

[0008] Preferably, performing intersection analysis on the second stitched fundus image based on the first stitched fundus image includes: obtaining the first optic disc feature and the first macula region of the first stitched fundus image, and obtaining the second optic disc feature and the second macula region of the second stitched fundus image; performing image registration processing on the first stitched fundus image and the second stitched fundus image based on the first optic disc feature, the second optic disc feature, the first macula region and the second macula region; performing intersection analysis on the registered first stitched fundus image and second stitched fundus image to generate an intersection analysis result.

[0009] Preferably, the method further includes: after obtaining the intersection analysis result, obtaining the offset degree of the first stitched fundus image and the second stitched fundus image based on the intersection analysis result; determining whether the offset degree is greater than a preset offset threshold; if so, obtaining the offset region and the offset orientation; generating and feedbacking a corresponding reshooting instruction based on the offset region and the offset orientation.

[0010] Preferably, the method further includes: before performing the intersection analysis, determining whether the shooting wide angles of the first spliced fundus image and the second spliced fundus image meet a preset consistency requirement; if the preset consistency requirement is not met, performing image normalization processing to obtain a processed first image and a processed second image; and performing intersection analysis on the processed first image and the processed second image.

[0011] Preferably, determining whether the shooting wide angles of the first spliced fundus image and the second spliced fundus image meet a preset consistency requirement includes: performing image wide angle analysis to determine a first shooting wide angle of the first spliced fundus image and a second shooting wide angle of the second spliced fundus image; determining whether the first shooting wide angle and the second shooting wide angle are the same; if the first shooting wide angle and the second shooting wide angle are equal, determining that the preset consistency requirement is met; if the first shooting wide angle and the second shooting wide angle are not equal, determining that the preset consistency requirement is not met; or: determining a preset reference wide angle; determining whether the first shooting wide angle and the second shooting wide angle are equal to the preset reference wide angle; if the first shooting wide angle and the second shooting wide angle are equal to the preset reference wide angle, determining that the preset consistency requirement is met; if the first shooting wide angle and / or the second shooting wide angle is not equal to the preset reference wide angle, determining that the preset consistency requirement is not met.

[0012] Preferably, performing the image wide angle analysis includes: identifying the optic disc of the target spliced fundus image to obtain optic disc information; determining the optic disc area based on the optic disc information; obtaining the outer arc of the target spliced fundus image, and determining the center and radius based on the outer arc; determining the image area of the target spliced fundus image based on the center and the radius; and determining the shooting wide angle of the target spliced fundus image based on the optic disc area and the image area.

[0013] Preferably, performing the image normalization processing includes: determining a reference image and an image to be processed from the first spliced fundus image and the second spliced fundus image; determining the optic disc center of the reference image; determining a cropping ratio for the image to be processed based on the first shooting wide angle and the second shooting wide angle; performing a cropping operation on the corresponding fundus image in the image set to be processed based on the optic disc center and the cropping ratio to obtain a normalized image; and determining the corresponding processed first image and processed second image based on the reference image and the normalized image.

[0014] Preferably, the cropping of the first intersection image to obtain a first set of cropped images includes: obtaining the optic disc and macula of the first intersection image; determining an upper dividing line based on the optic disc vertex of the optic disc and the macula vertex of the macula, determining a lower dividing line based on the optic disc bottom point of the optic disc and the macula bottom point of the macula, and determining a vertical dividing line based on the center point of the optic disc of the optic disc; determining seven divided regions based on the upper dividing line, the lower dividing line, and the vertical dividing line; and cropping the first intersection image based on the seven divided regions to obtain a first set of cropped images.

[0015] Correspondingly, the present invention further provides a lesion analysis device based on fundus images, where the device includes: an image set acquisition unit for acquiring a first fundus image set and a second fundus image set, where the first fundus image set is taken at the initial stage of fundus detection, and the second fundus image set is taken at a subsequent stage of fundus detection; a priority determination unit for determining the stitching priority of each image in the first fundus image set and the second fundus image set; a stitching unit for performing image stitching on the first fundus image set and the second fundus image set respectively based on the stitching priority to obtain corresponding first stitched fundus images and second stitched fundus images; an intersection processing unit for performing intersection analysis on the second stitched fundus image based on the first stitched fundus image, processing the first stitched fundus image based on the intersection analysis result to obtain a first intersection image, and processing the second stitched fundus image based on the intersection analysis result to obtain a second intersection image; a cropping and analysis unit for cropping the first intersection image and the second intersection image respectively to obtain corresponding first sets of cropped images and second sets of cropped images, performing lesion analysis on the first sets of cropped images and the second sets of cropped images respectively, and generating corresponding first analysis data and second analysis data; and a result generation unit for generating a lesion analysis result based on the first analysis data and the second analysis data.

[0016] Through the technical solution provided by the present invention, the present invention has at least the following technical effects:

[0017] By combining the actual problems encountered in the process of using an intelligent model to analyze the condition of a patient's fundus lesions, data consistency processing is performed in each link such as obtaining the patient's fundus images, determining the analysis area, and cropping the model input images, so that the patient's lesions to be analyzed are highly consistent at any time, thereby effectively improving the accuracy and reliability of the condition analysis based on the lesions and meeting the actual needs.

[0018] Other features and advantages of the embodiments of the present invention will be described in detail in the subsequent specific implementation part. Description of the Drawings

[0019] The accompanying drawings are used to provide a further understanding of the embodiments of the present invention, and constitute a part of the specification. Together with the following specific implementation manners, they are used to explain the embodiments of the present invention, but do not constitute a limitation to the embodiments of the present invention. In the accompanying drawings:

[0020] Figure 1 is a specific implementation flowchart of the lesion analysis method based on fundus images provided by the embodiments of the present invention;

[0021] Figure 2 is a schematic diagram of performing image wide-angle analysis provided by the embodiments of the present invention;

[0022] Figure 3 is a schematic diagram of cropping the first spliced fundus image provided by the embodiments of the present invention;

[0023] Figure 4 is a schematic structural diagram of the lesion analysis device based on fundus images provided by the embodiments of the present invention. Specific Implementation Manner

[0024] The following will detail the specific implementation manners of the embodiments of the present invention with reference to the accompanying drawings. It should be understood that the specific implementation manners described herein are only used to illustrate and explain the embodiments of the present invention, and are not used to limit the embodiments of the present invention.

[0025] The terms "system" and "network" in the embodiments of the present invention can be used interchangeably. "Multiple" means two or more. In view of this, in the embodiments of the present invention, "multiple" can also be understood as "at least two". "And / or" describes the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " generally represents an "or" relationship between the front and rear associated objects unless otherwise specified. In addition, it should be understood that in the description of the embodiments of the present invention, words such as "first" and "second" are only used for the purpose of distinguishing descriptions, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying order.

[0026] First, the background technology of the present invention will be introduced below.

[0027] During the existing fundus image shooting process, the operator takes pictures of the patient according to the predetermined operation specifications or relevant standards. However, in the actual application process, due to the large subjectivity of manual operation and the existence of factors such as jitter, there are slight deviations in the fundus images taken by the patient at different times. If the images taken by the patient at different times are directly input into the intelligent model, it may lead to huge deviations in the lesion detection results. For example, during the first shooting, the shooting angle deviates to the left, resulting in excessive inclusion of lesions on the left side. During the second shooting, the shooting angle deviates to the right, resulting in a large reduction in the area of the left lesion, and a wrong diagnosis conclusion that the area of the lesion has been greatly reduced and the condition has been significantly alleviated is drawn.

[0028] To solve the above technical problems, please refer to Figure 1 , an embodiment of the present invention provides a lesion analysis method based on fundus images, and the method includes:

[0029] S10: Obtain a first fundus image set and a second fundus image set, where the first fundus image set is taken during the initial period of fundus detection, and the second fundus image set is taken during the subsequent period of fundus detection;

[0030] S20: Determine the splicing priority of each image in the first fundus image set and the second fundus image set;

[0031] S30: Perform image splicing on the first fundus image set and the second fundus image set respectively based on the splicing priority to obtain a corresponding first spliced fundus image and a second spliced fundus image;

[0032] S40: Perform intersection analysis on the second spliced fundus image based on the first spliced fundus image, process the first spliced fundus image based on the intersection analysis result to obtain a first intersection image, and process the second fundus image set based on the intersection analysis result to obtain a second intersection image;

[0033] S50: Crop the first intersection image and the second intersection image respectively to obtain corresponding first cropped image sets and second cropped image sets, perform lesion analysis on the first cropped image sets and the second cropped image sets respectively, and generate corresponding first analysis data and second analysis data;

[0034] S60: Generate a lesion analysis result based on the first analysis data and the second analysis data.

[0035] In a possible implementation, first, obtain a first fundus image set captured by a patient in a first period and a second fundus image set captured in a second period. It should be noted that, based on actual treatment needs, the patient may need to capture fundus image sets in multiple periods to continuously analyze the efficacy of medication or determine whether the condition has deteriorated, etc. Those skilled in the art can, based on the above implementation provided by the embodiments of the present invention, think of applying it to the lesion analysis scenario of fundus image sets captured in multiple periods, which should also fall within the protection scope of the embodiments of the present invention and will not be elaborated here. Since doctors actually pay more attention to the changes in the lesions in the first fundus image captured by the patient and the lesions in the last fundus image captured, preferably, the first fundus image set is captured in the initial period of fundus detection, and the second fundus image set is captured in the final period of fundus detection. In the embodiments of the present invention, both the above-mentioned first fundus image set and second fundus image set include 7 views captured according to the set fundus image capture specifications.

[0036] After obtaining the fundus image sets captured in the above two periods, immediately perform a 7-view splicing operation on them for subsequent analysis. In the embodiments of the present invention, to ensure the best splicing effect, an existing standard splicing method can be used to splice them. For example, determine the splicing order of each fundus image in turn from the middle to the periphery and splice them in this order. However, in the specific implementation process, there may be cases where the shooting quality of some fundus images is poor. Using the traditional splicing method may result in covering the images with better shooting quality with the images with poor shooting quality and reducing the lesion analysis effect.

[0037] To solve the above technical problems, on the one hand, perform a preliminary sorting on them according to the splicing priority order of each fundus image, and then determine the best splicing priority in combination with the clarity and splicing weight of each image. In the embodiments of the present invention, determining the splicing priority of each image in the first fundus image set and the second fundus image set includes: determining the initial splicing order of each image, determining the splicing weight of each image based on the initial splicing order; evaluating the lesion quality of each image to determine the corresponding quality evaluation result; determining the splicing priority of each image in the first fundus image set and the second fundus image set based on the splicing weight and the quality evaluation result.

[0038] Specifically, first, determine the initial stitching order of each image. It should be noted that the stitching priority analysis can be performed separately on multiple images in each fundus image set. According to the initial stitching order, the stitching weight of each image can be determined. For example, in the embodiment of the present invention, the middle image is determined as order 1, and then the macula map and the optic disc map are respectively determined as order 2 and 3. Then, according to the pre-determined rule (such as the clockwise direction), the order of the other surrounding images is determined as 4-7 in sequence, and the stitching weight of each image is determined according to the above order (that is, the importance of each image). For example, the weight of the central picture is determined as 25%. Since doctors are more concerned about the lesion situation in the fundus map and do not pay attention to the quality of other images, in the embodiment of the present invention, the lesion quality of each picture is further evaluated to generate a quality evaluation result. Finally, the stitching priority of each picture is determined according to the stitching weight and the quality evaluation result of each image.

[0039] In the embodiment of the present invention, by comprehensively analyzing the stitching importance and shooting quality of each picture, the best stitching priority is determined by taking into account the stitching priority degree and clarity of the image. Stitching the images according to this priority can effectively ensure the stitching effect of the images, improve the clarity of the display of the lesions, and improve the accuracy of lesion analysis.

[0040] After determining the stitching priority, perform image stitching on the fundus image set, and obtain the corresponding first stitched fundus map and the second stitched fundus map. Although doctors are required to take pictures according to the specifications during shooting, due to factors such as hand shaking and shooting deviation, there may be a slight deviation between the fundus maps taken before and after, and this deviation may cause the sizes of the same lesion displayed twice to be inconsistent, resulting in a huge change in the lesion analysis data. In order to avoid the situation where the lesion areas analyzed twice are inconsistent, perform intersection analysis to ensure that the lesion analysis of the fundus maps taken at different times is for the same lesion area, ensuring consistency before and after, and ensuring the accuracy of lesion analysis.

[0041] In the embodiment of the present invention, the intersection analysis of the second stitched fundus map based on the first stitched fundus map includes: obtaining the first optic disc feature and the first macula area of the first stitched fundus map, and obtaining the second optic disc feature and the second macula area of the second stitched fundus map; performing image registration processing on the first stitched fundus map and the second stitched fundus map based on the first optic disc feature, the second optic disc feature, the first macula area, and the second macula area; performing intersection analysis on the registered first stitched fundus map and the second stitched fundus map to generate an intersection analysis result.

[0042] It is very easy for those skilled in the art to know that for two images, even if they are taken by the same person, with the same device, and even in the same period, due to various reasons, it is very difficult to register the two images at the pixel level. There are more or less certain perspective changes in the two taken images, so there will inevitably be a certain degree of "ghosting" in the stitched image.

[0043] In order to improve the stitching efficiency and reduce the stitching difficulty, in a possible implementation manner, after obtaining the first stitched fundus image and the second stitched fundus image, first extract the first optic disc feature and the first macula area of the first stitched fundus image, and extract the second optic disc feature and the second macula area of the second stitched fundus image. Then, use a combination of feature points and regions to register the first stitched fundus image and the second stitched fundus image, so as to quickly register the two images within an acceptable error range, and then perform intersection analysis on the two registered images to quickly generate an intersection analysis result.

[0044] However, during the shooting process, if the deviation between the two stitched fundus images is too large, it will lead to a small intersection area at the edge of the image, a reduction in the analyzable lesion area lost, and a decrease in the reliability and accuracy of lesion analysis. Therefore, in the embodiments of the present invention, the method further includes: after obtaining the intersection analysis result, obtaining the deviation degree of the first stitched fundus image and the second stitched fundus image based on the intersection analysis result; determining whether the deviation degree is greater than a preset deviation threshold; if so, obtaining the deviation area and the deviation direction; generating and feeding back a corresponding reshooting instruction based on the deviation area and the deviation direction.

[0045] Specifically, after obtaining the intersection analysis result, further analyze the deviation degree of the first stitched fundus image and the second stitched fundus image. If the deviation degree is too large, for example, the deviation degree of the overall image is greater than the preset deviation threshold, the deviation area and the deviation direction can be immediately obtained, and a corresponding reshooting instruction can be generated. Further, if the technician finds that the deviation degree of the middle image is greater than the acceptable range, the set of fundus images taken this time can be directly discarded, and an immediate reshoot is required to ensure that the deviation between the sets of fundus images taken in the previous and subsequent periods is within the acceptable range.

[0046] In the embodiments of the present invention, by analyzing the deviation degree of the sets of fundus images taken in the previous and subsequent periods, it is effectively ensured that the deviation degree of the fundus images taken in the previous and subsequent periods is within the acceptable range, avoiding excessive loss of lesions during the intersection-taking process, and ensuring the reliability and accuracy of lesion analysis.

[0047] At this time, the first spliced fundus image is processed according to the intersection analysis result to obtain a first intersection image. Specifically, the first spliced fundus image is cropped according to the edge features of the intersection area of the first spliced fundus image to obtain the corresponding first intersection image. Based on the same principle, after processing the second spliced fundus image, a second intersection image is obtained.

[0048] In the embodiment of the present invention, by performing intersection processing on the images taken at different times, it is possible to effectively avoid large deviations in the lesion detection results due to slight deviations in the fundus images taken at different times, ensuring that the lesion areas analyzed during the two detections are the same lesion area and ensuring the reliability of the analysis results.

[0049] In the actual application process, although in general implementation scenarios, doctors or hospitals will require that the fundus images taken of patients at different times be taken using the same specification of equipment, due to the existence of imaging devices with different shooting angles, and if the fundus images taken of a patient at two times use imaging devices with different shooting angles (for example, different-angle devices are used in the same hospital, or the imaging devices in different hospitals have different angles), it will lead to mismatches during the splicing, cropping, and recognition of the fundus images, and further lead to abnormal lesion recognition analysis results.

[0050] To solve the above technical problems, in the embodiment of the present invention, the method further includes: before performing the intersection analysis, determining whether the shooting angles of the first spliced fundus image and the second spliced fundus image meet the preset consistency requirements; if the preset consistency requirements are not met, performing image normalization processing to obtain a processed first image and a processed second image; performing intersection analysis on the processed first image and the processed second image.

[0051] In a possible implementation, after obtaining the first post - spliced fundus image and the second post - spliced fundus image, it is further determined whether they meet the preset consistency requirements. In the embodiments of the present invention, the determination of whether the shooting wide - angles of the first post - spliced fundus image and the second post - spliced fundus image meet the preset consistency requirements includes: performing image wide - angle analysis to determine the first shooting wide - angle of the first post - spliced fundus image and the second shooting wide - angle of the second post - spliced fundus image; determining whether the first shooting wide - angle and the second shooting wide - angle are consistent; if the first shooting wide - angle and the second shooting wide - angle are equal, it is determined that the preset consistency requirements are met; if the first shooting wide - angle and the second shooting wide - angle are not equal, it is determined that the preset consistency requirements are not met; or: determining a preset reference wide - angle; determining whether the first shooting wide - angle and the second shooting wide - angle are equal to the preset reference wide - angle; if the first shooting wide - angle and the second shooting wide - angle are equal to the preset reference wide - angle, it is determined that the preset consistency requirements are met; if the first shooting wide - angle and / or the second shooting wide - angle are not equal to the preset reference wide - angle, it is determined that the preset consistency requirements are not met.

[0052] Generally, the wide - angle parameters of the shooting devices for the patient's front and back shots should be consistent. At this time, the fundus image sets taken in the two periods can be directly analyzed without considering what the actual wide - angle parameters of the shooting devices are, that is, as long as the shooting wide - angles of the fundus image sets taken twice are consistent, subsequent analysis can be directly carried out. Therefore, in the first embodiment, image wide - angle analysis is first performed on the two fundus images.

[0053] In the embodiments of the present invention, the performing of image wide - angle analysis includes: identifying the optic disc of the target post - spliced fundus image to obtain optic disc information; determining the optic disc area based on the optic disc information; obtaining the outer arc of the target post - spliced fundus image, and determining the center and radius based on the outer arc; determining the image area of the target post - spliced fundus image based on the center and the radius; determining the shooting wide - angle of the target post - spliced fundus image based on the optic disc area and the image area.

[0054] Specifically, please refer to Figure 2 , in a specific implementation, first, the optic disc of the target post - spliced fundus image is identified to obtain optic disc information, and the optic disc information includes, but is not limited to, information such as the optic disc position and the optic disc area. In the embodiments of the present invention, the target post - spliced fundus image is characterized as either the first post - spliced fundus image or the second post - spliced fundus image, and no more details will be elaborated here.

[0055] After obtaining the above disc information, determine the disc area. At this time, further obtain the outer arc of the target mosaicked fundus image. For example, extract the edge of the target mosaicked fundus image through an edge extraction algorithm to form the outer arc. At this time, determine the corresponding center and radius according to this outer arc. For example, randomly select 3 points on this outer arc, calculate the corresponding center and radius according to the coordinate information of these 3 points, and the image area of the captured fundus image can be approximately calculated according to this center and radius. At this time, the disc ratio can be obtained according to the above disc area and image area. According to the ratio of the disc area in the fundus image captured by the imaging device with different shooting wide angles in the whole image, the shooting wide angle of the current image can be determined. For example, in the fundus image captured by the imaging device with a shooting wide angle of 30°, the disc ratio is 6%-8%; in the fundus image captured by the imaging device with a shooting wide angle of 45°, the disc ratio is 4%-6%; in the fundus image captured by the imaging device with a shooting wide angle of 55°, the disc ratio is 2%-4%; in the fundus image captured by the imaging device with a shooting wide angle of 90°, the disc ratio is less than 1%.

[0056] After performing the image wide angle analysis, the first shooting wide angle of the first mosaicked fundus image and the second shooting wide angle of the second mosaicked fundus image can be determined. If the first shooting wide angle is equal to the second shooting wide angle, it can be determined that the sizes of the fundus images captured by the patient at two times are the same, and the lesion analysis can be directly performed. Otherwise, it can be determined that the wide angles of the fundus images captured by the patient at two times are inconsistent, and image normalization processing needs to be performed before the lesion analysis can be performed, otherwise the analysis result will be abnormal.

[0057] For doctors, the size of the fundus image shown with a shooting wide angle of 30° is relatively reasonable and the image details are relatively clear. Therefore, an imaging device with a shooting wide angle of 30° is often used to capture the fundus image. However, with the needs of different doctors and different actual application scenarios, doctors may expect to analyze the patient's fundus image under a certain standard shooting wide angle to obtain the best fundus image capture effect and the most comprehensive fundus information. Therefore, in the second embodiment, after performing the image wide angle analysis, further determine a preset reference wide angle. For example, this preset reference wide angle is the standard wide angle specified in advance by the doctor. After obtaining the first shooting wide angle and the second shooting wide angle, immediately determine whether they are equal to this preset reference wide angle. If the shooting wide angle of any one fundus image is not equal to this preset reference wide angle, it can be determined that it does not meet the preset consistency requirement. At this time, image normalization processing needs to be performed on the fundus image that does not meet the requirement.

[0058] In an embodiment of the present invention, performing image normalization processing includes: determining a reference image and an image to be processed from the first spliced fundus image and the second spliced fundus image; determining the optic disc center of the reference image; determining a cropping ratio for the image to be processed based on the first shooting wide angle and the second shooting wide angle; performing a cropping operation on the image to be processed based on the optic disc center and the cropping ratio to obtain a set of normalized images; and determining corresponding processed first image and processed second image based on the reference image set and the set of normalized images.

[0059] In a possible implementation manner, if it is detected that the first shooting wide angle and the second shooting wide angle are not equal, further determine the reference image and the image to be processed from the first spliced fundus image and the second spliced fundus image. For example, take the first spliced fundus image taken for the first time as the reference image, and take the fundus images taken subsequently as images to be processed. Then determine the optic disc center of the reference image, and in subsequent image scaling and cropping processes, perform equal-proportion scaling and cropping of the image with this optic disc center as the center, thereby ensuring the cropping accuracy of fundus images taken at different times.

[0060] At this time, further determine the cropping ratio for the image to be processed according to the first shooting wide angle and the second shooting wide angle. For example, in a specific embodiment, register the fundus image with a shooting wide angle of 45° to the fundus image with a shooting wide angle of 30°. First, determine the optic disc center of the fundus image with a shooting wide angle of 30°, then take the fundus image with a wide angle of 45° as the center, gradually expand the circular coverage range, and at the same time, calculate the pixel mean square error between the circular coverage range and the fundus image with a shooting wide angle of 30° in real time, and find the minimum value of the mean square error. At this time, the magnification corresponding to this value is the cropping ratio for registering the fundus image with a wide angle of 45° to the fundus image with a shooting wide angle of 30°.

[0061] Finally, perform a cropping operation on the image to be processed according to the optic disc center of the reference image and the determined cropping ratio to obtain a normalized image, thereby obtaining the processed first image and the processed second image.

[0062] In the second embodiment, the shooting wide angles adopted in the two sets of fundus images taken by the patient do not conform to the preset reference wide angle. Therefore, in the subsequent normalization processing, perform image normalization processing on both the first spliced fundus image and the second spliced fundus image according to the preset reference wide angle to obtain the processed first image and the processed second image to meet the actual requirements.

[0063] After configuring all sets of fundus images to have imaging regions corresponding to a unified and standard shooting wide angle, perform intersection analysis on them to obtain an accurate intersection analysis result.

[0064] In the embodiments of the present invention, by corresponding processing according to the actual shooting situation of fundus images, it is ensured that the fundus images taken in different periods are all based on the same shooting standard, effectively guaranteeing the accuracy and reliability in the subsequent image stitching and cropping processes, and ensuring the accuracy of lesion diagnosis.

[0065] After obtaining the first intersection image and the second intersection image, they are respectively cropped. In some conventional cropping methods, a sliding window method can be used to crop the fundus image. However, in the actual application process, technicians found that there are at least the following technical problems: on the one hand, when the intelligent recognition model of lesions is trained, a large number of complete retinal pictures are pre-selected by technicians and input for training according to the input format of the intelligent recognition model. After cropping the fundus image using a sliding window, for the edge of the fundus image, there are a large number of blank areas or abnormal areas in the cropped image. When inputting it into the intelligent model for lesion recognition, since there is a large difference from the training image, the lesion recognition effect is greatly reduced; on the other hand, after the image cropped by the above sliding window is input into the intelligent model, some images are identified as lesions due to the lack of anterior and posterior blood vessel information, thereby reducing the recognition accuracy.

[0066] To solve the above technical problems, please refer to Figure 3 , in the embodiments of the present invention, the cropping of the first intersection image to obtain the first cropped image set includes: obtaining the optic disc and macula of the first intersection image; determining the upper dividing line based on the optic disc vertex of the optic disc and the macula vertex of the macula, determining the lower dividing line based on the optic disc bottom point of the optic disc and the macula bottom point of the macula, and determining the vertical dividing line based on the optic disc center point of the optic disc; determining seven divided areas based on the upper dividing line, the lower dividing line, and the vertical dividing line; and cropping the first intersection image based on the seven divided areas to obtain the first cropped image set.

[0067] In a possible implementation manner, after obtaining the intersection image of each shooting, seven divided areas are constructed and used to crop it. Specifically, the optic disc and macula of the first intersection image are obtained. For example, a deep neural detection model is used to identify the optic disc area and macula area from the first intersection image, and then the upper dividing line, the lower dividing line, and the vertical dividing line are determined according to the relevant features of the optic disc and macula, such as vertices, bottom points, center points, etc. The vertical dividing line passes through the optic disc center point and is perpendicular to the upper dividing line and the lower dividing line. The first intersection image is divided into 4 areas by the above three lines.

[0068] Then, determine the radius of the cropping circle. Specifically, the radius of the fundus image in the reference image can be used as the radius of the cropping circle, or the radius of the fundus image with a shooting wide angle of 30° can be used as the radius of the cropping circle. At this time, further determine the seven divided areas to be cropped according to the cropping rules of the fundus image and the radius of the above-mentioned cropping circle. For example, in the embodiment of the present invention, with the radius of the cropping circle as the radius, the area covered with the optic disc center as the center is determined as the first area; the area covered with the fovea center as the center is determined as the second area; the area covered with the point 1-1.5 optic disc diameters on the temporal side of the fovea as the center is determined as the third area; the area covered above the second area and tangent to the vertical dividing line and the upper dividing line is determined as the fourth area; based on the same principle as the fourth area, the fifth area, the sixth area and the seventh area can be obtained respectively, thus realizing the standardization process of each image cropping. On the basis of accurate and reliable fundus image stitching, it is ensured that the seven-area images cropped from the fundus images taken in each period are highly consistent.

[0069] Finally, perform lesion recognition on the cropped image set composed of seven-area images. For example, input the cropped image set composed of seven-area images into an intelligent model. For example, the intelligent model is an intelligent recognition model pre-trained based on models such as a conventional deep neural network model. The conventional deep neural network model includes, but is not limited to, a convolutional neural network model, a recurrent neural network model, a U2-Net model, etc., which will not be elaborated here. Through this intelligent model, the lesion information of each fundus image can be effectively recognized. For example, in the embodiment of the present invention, after performing lesion analysis on the first cropped image set through a preset deep learning model, first analysis data including data such as lesion density, total lesion area, and number of lesions can be generated; based on the same principle, second analysis data of the lesions in the fundus image of the patient in the second shooting period can be analyzed. The first analysis data strictly corresponds to the second analysis data. Therefore, according to the above first analysis data and second analysis data, accurate and reliable data support can be provided for the change of the patient's condition at different times. On this basis, lesion analysis can be performed to obtain accurate and reliable lesion analysis results of the patient, such as quantitatively grading the patient's lesions according to the ETDRS grading standard, so as to obtain accurate grading results.

[0070] In the embodiment of the present invention, by introducing the seven-division segmentation method in the cropping process of the fundus image, the blood vessel information in the fundus image can be maximally retained, and the recognition error of the intelligent model is greatly reduced; at the same time, through the above seven-division segmentation method, the image content can be maximally retained in the cropped image, and the form difference between the image input into the intelligent model for analysis and the training image is greatly reduced, so that the recognition accuracy of the intelligent model for the cropped image is effectively improved, meeting the actual needs of doctors and improving the accuracy of lesion recognition.

[0071] Please refer toFigure 4 , based on the same inventive concept, an embodiment of the present invention provides a lesion analysis device based on fundus images. The device includes: an image set acquisition unit for acquiring a first fundus image set and a second fundus image set, where the first fundus image set is taken at the initial stage of fundus detection, and the second fundus image set is taken at a subsequent stage of fundus detection; a priority determination unit for determining the splicing priority of each image in the first fundus image set and the second fundus image set; a splicing unit for performing image splicing on the first fundus image set and the second fundus image set respectively based on the splicing priority to obtain corresponding first spliced fundus images and second spliced fundus images; an intersection processing unit for performing intersection analysis on the second spliced fundus image based on the first spliced fundus image, processing the first spliced fundus image based on the intersection analysis result to obtain a first intersection image, and processing the second spliced fundus image based on the intersection analysis result to obtain a second intersection image; a cropping and analysis unit for cropping the first intersection image and the second intersection image respectively to obtain corresponding first cropped image sets and second cropped image sets, performing lesion analysis on the first cropped image sets and the second cropped image sets respectively to generate corresponding first analysis data and second analysis data; and a result generation unit for generating a lesion analysis result based on the first analysis data and the second analysis data.

[0072] The optional implementation manners of the embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the embodiments of the present invention are not limited to the specific details in the above implementation manners. Within the technical concept scope of the embodiments of the present invention, various simple modifications can be made to the technical solutions of the embodiments of the present invention, and these simple modifications all belong to the protection scope of the embodiments of the present invention.

[0073] In addition, it should be noted that, in the case of no contradiction, the various specific technical features described in the above specific implementation manners can be combined in any suitable manner. To avoid unnecessary repetition, the embodiments of the present invention will not separately describe various possible combination manners.

[0074] Those skilled in the art can understand that all or part of the steps for implementing the methods in the above embodiments can be completed by instructing relevant hardware through a program. The program is stored in a storage medium, including several instructions for causing a single-chip microcomputer, a chip, or a processor to execute all or part of the steps of the methods described in various embodiments of the present application. The foregoing storage medium includes: various media such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disc that can store program codes.

[0075] In addition, any combination can be made among various different embodiments of the embodiments of the present invention, as long as it does not violate the idea of the embodiments of the present invention, and it should also be regarded as the content disclosed by the embodiments of the present invention.

Claims

1. A lesion analysis method based on fundus images, characterized in that, The method includes: Obtaining a first fundus image set and a second fundus image set, where the first fundus image set is taken at the initial stage of fundus detection, and the second fundus image set is taken at a subsequent stage of fundus detection; Determining the stitching priority of each image in the first fundus image set and the second fundus image set; Performing image stitching on the first fundus image set and the second fundus image set respectively based on the stitching priority to obtain a corresponding first stitched fundus image and a second stitched fundus image; Performing intersection analysis on the second stitched fundus image based on the first stitched fundus image, processing the first stitched fundus image based on the intersection analysis result to obtain a first intersection image, and processing the second stitched fundus image based on the intersection analysis result to obtain a second intersection image; Cropping the first intersection image and the second intersection image respectively to obtain corresponding first cropped image sets and second cropped image sets, performing lesion analysis on the first cropped image sets and the second cropped image sets respectively to generate corresponding first analysis data and second analysis data; Generating a lesion analysis result based on the first analysis data and the second analysis data; The cropping the first intersection image to obtain a first cropped image set includes: Obtaining the optic disc and macula of the first intersection image; Determining an upper segmentation line based on the optic disc vertex of the optic disc and the macula vertex of the macula, determining a lower segmentation line based on the optic disc bottom point of the optic disc and the macula bottom point of the macula, and determining a vertical segmentation line based on the optic disc center point of the optic disc; Determining a seven-segmentation area based on the upper segmentation line, the lower segmentation line, and the vertical segmentation line; Cropping the first intersection image based on the seven-segmentation area to obtain a first cropped image set.

2. The method according to claim 1, characterized in that The determining the stitching priority of each image in the first fundus image set and the second fundus image set includes: Determining the initial stitching order of each image, and determining the stitching weight of each image based on the initial stitching order; Evaluating the lesion quality of each image to determine a corresponding quality evaluation result; Determining the stitching priority of each image in the first fundus image set and the second fundus image set based on the stitching weight and the quality evaluation result.

3. The method according to claim 1, wherein The performing intersection analysis on the second stitched fundus image based on the first stitched fundus image includes: Obtaining a first optic disc feature and a first macula area of the first stitched fundus image, and obtaining a second optic disc feature and a second macula area of the second stitched fundus image; Performing image registration processing on the first stitched fundus image and the second stitched fundus image based on the first optic disc feature, the second optic disc feature, the first macula area, and the second macula area; Performing intersection analysis on the registered first stitched fundus image and second stitched fundus image to generate an intersection analysis result.

4. The method according to claim 3, characterized in that, The method further includes: After obtaining the intersection analysis result, obtaining the offset degree of the first stitched fundus image and the second stitched fundus image based on the intersection analysis result; Judging whether the offset degree is greater than a preset offset threshold; If so, obtain the offset area and the offset orientation; Generate and feedback a corresponding reshooting instruction based on the offset area and the offset orientation.

5. The method according to claim 1, characterized in that, The method further includes: Before performing the intersection analysis, determine whether the shooting wide angles of the first spliced fundus image and the second spliced fundus image meet a preset consistency requirement; If the preset consistency requirement is not met, perform image normalization processing to obtain a processed first image and a processed second image; Perform intersection analysis on the processed first image and the processed second image.

6. The method according to claim 5, wherein The determination of whether the shooting wide angles of the first spliced fundus image and the second spliced fundus image meet the preset consistency requirement includes: Perform image wide angle analysis to determine the first shooting wide angle of the first spliced fundus image and the second shooting wide angle of the second spliced fundus image; determine whether the first shooting wide angle and the second shooting wide angle are consistent; if the first shooting wide angle and the second shooting wide angle are equal, determine that the preset consistency requirement is met; if the first shooting wide angle and the second shooting wide angle are not equal, determine that the preset consistency requirement is not met; or: Determine a preset reference wide angle; determine whether the first shooting wide angle and the second shooting wide angle are equal to the preset reference wide angle; if the first shooting wide angle and the second shooting wide angle are equal to the preset reference wide angle, determine that the preset consistency requirement is met; if the first shooting wide angle and / or the second shooting wide angle is not equal to the preset reference wide angle, determine that the preset consistency requirement is not met.

7. The method according to claim 6, wherein The performing of the image wide angle analysis includes: Perform optic disc recognition on the target spliced fundus image to obtain optic disc information; Determine the optic disc area based on the optic disc information; Obtain the outer arc of the target spliced fundus image, and determine the center and radius based on the outer arc; Determine the image area of the target spliced fundus image based on the center and the radius; Determine the shooting wide angle of the target spliced fundus image based on the optic disc area and the image area.

8. The method according to claim 7, wherein The performing of the image normalization processing includes: Determine a reference image and an image to be processed from the first spliced fundus image and the second spliced fundus image; Determine the optic disc center of the reference image; Determine the cropping ratio for each fundus image in the image set to be processed based on the first shooting wide angle and the second shooting wide angle; Perform a cropping operation on the image to be processed based on the optic disc center and the cropping ratio to obtain a normalized image; Determine the corresponding processed first image and processed second image based on the reference image and the normalized image.

9. A lesion analysis device based on fundus images, characterized in that, The device is applied to the method according to any one of claims 1-8, and the device includes: An image set acquisition unit, configured to acquire a first fundus image set and a second fundus image set, where the first fundus image set is taken at the initial stage of fundus detection, and the second fundus image set is taken at a subsequent stage of fundus detection; A priority determination unit, configured to determine the splicing priority of each image in the first fundus image set and the second fundus image set; A stitching unit, configured to perform image stitching on the first fundus image set and the second fundus image set respectively based on the stitching priority, to obtain a corresponding first stitched fundus image and a second stitched fundus image; An intersection processing unit, configured to perform intersection analysis on the second stitched fundus image based on the first stitched fundus image, process the first stitched fundus image based on the intersection analysis result to obtain a first intersection image, and process the second stitched fundus image based on the intersection analysis result to obtain a second intersection image; A cropping analysis unit, configured to crop the first intersection image and the second intersection image respectively to obtain corresponding first cropped image sets and second cropped image sets, and perform lesion analysis on the first cropped image sets and the second cropped image sets respectively to generate corresponding first analysis data and second analysis data; A result generation unit, configured to generate a lesion analysis result based on the first analysis data and the second analysis data; The cropping of the first intersection image to obtain a first cropped image set includes: Obtaining the optic disc and macula of the first intersection image; Determining an upper dividing line based on the optic disc vertex of the optic disc and the macula vertex of the macula, determining a lower dividing line based on the optic disc bottom point of the optic disc and the macula bottom point of the macula, and determining a vertical dividing line based on the optic disc center point of the optic disc; Determining a seven-region division based on the upper dividing line, the lower dividing line, and the vertical dividing line; Cropping the first intersection image based on the seven-region division to obtain a first cropped image set.

Images