Focus analysis method and device based on fundus image

Abstract

The invention relates to the technical field of image processing, and particularly discloses a focus analysis method and device based on fundus images, and the method comprises the steps: obtaining a first fundus image set and a second fundus image set; performing image stitching on the first fundus image set and the second fundus image set to obtain a first stitched fundus image and a second stitched fundus image; performing intersection analysis on the second spliced eye fundus image based on the first spliced eye fundus image, and processing the first spliced eye fundus image and the second spliced eye fundus image based on an intersection analysis result to obtain a first intersection image and a second intersection image; cutting the first intersection image and the second intersection image to obtain a first cut image set and a second cut image set; performing focus analysis on the first cutting image set and the second cutting image set to generate first analysis data and second analysis data; and generating a lesion analysis result based on the first analysis data and the second analysis data. The accuracy and reliability of focus recognition are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method and device for analyzing lesions based on fundus images. Background Art

[0002] Diabetic retinopathy (DR) is one of the serious complications of diabetes. If it is not discovered and treated in time, it often leads to irreversible blindness, with extremely serious consequences. If it can be identified in the early and middle stages of the disease, timely intervention and treatment according to the different stages of the disease can often save the patient's vision and improve the quality of life.

[0003] The Early Treatment of Diabetic Retinopathy Study (ETDRS) standard seven-field (7SF) color fundus photography has always been the preferred method for fundus imaging in diabetic patients. Diabetic retinopathy is one of the most common complications in diabetic patients and can lead to vision loss and blindness. At present, the severity of diabetic retinopathy is usually assessed using ETDRS 7-field imaging technology. This method is simple and more sensitive. It can be combined with automated algorithms to accurately assess the severity of diabetic retinopathy with higher accuracy and more accurate assessment of the disease. However, this method currently relies on the human eye to roughly estimate the number of lesions and then give a rating, which leads to the consumption of professional human resources. When the image is at the boundary of the grade, the grading is relatively subjective, and different people may have different grading results, which leads to a decrease in the credibility of the results.

[0004] In order to solve the above technical problems, the technicians considered using intelligent models to analyze and identify lesions in fundus photographs. However, in actual application, on the one hand, due to the large difference between the training images and the actual input images, the model recognition effect was poor. On the other hand, the model analyzed the lesions based on the fundus images taken at different times. However, the fundus images taken at different times had deviations due to jitter and other reasons, resulting in huge differences in lesion recognition results, which had a great impact on the recognition results and could not meet actual needs. Summary of the invention

[0005] In order to overcome the above-mentioned technical problems existing in the prior art, the embodiments of the present invention provide a lesion analysis method and device based on fundus images. By analyzing the practical problems existing in the lesion analysis of fundus images by intelligent models, consistency processing is performed in each link from image acquisition, analysis area determination, image cropping, etc., to ensure the accuracy of lesion analysis.

[0006] In order to achieve the above-mentioned purpose, an embodiment of the present invention provides a method for analyzing lesions based on fundus images, the method comprising: acquiring a first fundus image set and a second fundus image set, the first fundus image set being taken during an initial period of fundus detection, and the second fundus image set being taken during a subsequent period of fundus detection; determining a stitching priority of each image in the first fundus image set and the second fundus image set; performing image stitching on the first fundus image set and the second fundus image set based on the stitching priority to obtain a corresponding first stitched fundus image and a second stitched fundus image; and performing stitching on the first fundus image based on the first stitched fundus image. The second stitched fundus image is subjected to intersection analysis, the first stitched fundus image is processed based on the intersection analysis result to obtain a first intersection image, and the second stitched fundus image is processed based on the intersection analysis result to obtain a second intersection image; the first intersection image and the second intersection image are cropped respectively to obtain a corresponding first cropped image set and a second cropped image set, lesion analysis is performed on the first cropped image set and the second cropped image set respectively to generate corresponding first analysis data and second analysis data; lesion analysis results are generated based on the first analysis data and the second analysis data.

[0007] Preferably, determining the stitching priority of each image in the first fundus image set and the second fundus image set includes: determining an initial stitching order for each image, and determining a stitching weight for each image based on the initial stitching order; evaluating the lesion quality of each image to determine a corresponding quality evaluation result; and determining the stitching priority of each image in the first fundus image set and the second fundus image set based on the stitching weight and the quality evaluation result.

[0008] Preferably, the intersection analysis of the second stitched fundus image based on the first stitched fundus image includes: obtaining a first optic disc feature and a first macular area of ​​the first stitched fundus image, and obtaining a second optic disc feature and a second macular area of ​​the second stitched fundus image; performing image registration processing on the first stitched fundus image and the second stitched fundus image based on the first optic disc feature, the second optic disc feature, the first macular area and the second macular area; performing intersection analysis on the registered first stitched fundus image and the second stitched fundus image to generate an intersection analysis result.

[0009] Preferably, the method also includes: after obtaining the intersection analysis result, obtaining the degree of offset between the first stitched fundus image and the second stitched fundus image based on the intersection analysis result; judging whether the degree of offset is greater than a preset offset threshold; if so, obtaining the offset area and the offset orientation; generating and feeding back a corresponding reshoot instruction based on the offset area and the offset orientation.

[0010] Preferably, the method also includes: before performing the intersection analysis, determining whether the shooting wide angles of the first stitched fundus image and the second stitched fundus image meet a preset consistency requirement; if the preset consistency requirement is not met, performing image normalization processing to obtain a processed first image and a processed second image; and performing intersection analysis on the processed first image and the processed second image.

[0011] Preferably, the judging whether the shooting wide angles of the first stitched fundus image and the second stitched fundus image meet the preset consistency requirement includes: performing image wide-angle analysis to determine the first shooting wide angle of the first stitched fundus image and the second shooting wide angle of the second stitched fundus image; judging whether the first shooting wide angle and the second shooting wide angle are consistent; if the first shooting wide angle and the second shooting wide angle are equal, determining that the preset consistency requirement is met; if the first shooting wide angle and the second shooting wide angle are not equal, determining that the preset consistency requirement is not met; or: determining a preset reference wide angle; judging whether the first shooting wide angle and the second shooting wide angle are equal to the preset reference wide angle; if the first shooting wide angle and the second shooting wide angle are equal to the preset reference wide angle, determining that the preset consistency requirement is met; if the first shooting wide angle and / or the second shooting wide angle are not equal to the preset reference wide angle, determining that the preset consistency requirement is not met.

[0012] Preferably, the performing of wide-angle image analysis includes: performing optic disc identification on the target stitched fundus image to obtain optic disc information; determining the optic disc area based on the optic disc information; obtaining the outer arc of the target stitched fundus image, and determining the center and radius based on the outer arc; determining the image area of ​​the target stitched fundus image based on the center and the radius; and determining the shooting wide angle of the target stitched fundus image based on the optic disc area and the image area.

[0013] Preferably, the performing of image normalization processing includes: determining a reference image and an image to be processed from the first stitched fundus image and the second stitched fundus image; determining the optic disc center of the reference image; determining a cropping ratio for the image to be processed based on the first shooting wide angle and the second shooting wide angle; performing a cropping operation on the corresponding fundus image in the image set to be processed based on the optic disc center and the cropping ratio to obtain a normalized image; and determining a corresponding processed first image and a processed second image based on the reference image and the normalized image.

[0014] Preferably, the cropping of the first intersection image to obtain a first cropped image set includes: acquiring the optic disc and macula of the first intersection image; determining an upper dividing line based on the optic disc vertex of the optic disc and the macula vertex of the macula, determining a lower dividing line based on the optic disc bottom point of the optic disc and the macula bottom point of the macula, and determining a vertical dividing line based on the optic disc center point of the optic disc; determining seven segmentation areas based on the upper segmentation line, the lower segmentation line, and the vertical segmentation line; and cropping the first intersection image based on the seven segmentation areas to obtain a first cropped image set.

[0015] Accordingly, the present invention further provides a device for analyzing lesions based on fundus images, the device comprising: an image set acquisition unit, used to acquire a first fundus image set and a second fundus image set, the first fundus image set being taken during an initial period of fundus detection, and the second fundus image set being taken during a subsequent period of fundus detection; a priority determination unit, used to determine a stitching priority of each image in the first fundus image set and the second fundus image set; a stitching unit, used to perform image stitching on the first fundus image set and the second fundus image set respectively based on the stitching priority to obtain a corresponding first stitched fundus image and a second stitched fundus image; an intersection processing unit, used to perform image stitching on the first fundus image set and the second fundus image set based on the stitching priority to obtain a corresponding first stitched fundus image and a second stitched fundus image A method for performing intersection analysis on the second stitched fundus map after stitching, processing the first stitched fundus map based on the intersection analysis result to obtain a first intersection image, and processing the second stitched fundus map based on the intersection analysis result to obtain a second intersection image; a cropping and analysis unit, used to crop the first intersection image and the second intersection image respectively to obtain the corresponding first cropped image set and second cropped image set, performing lesion analysis on the first cropped image set and the second cropped image set respectively, and generating corresponding first analysis data and second analysis data; a result generating unit, used to generate a lesion analysis result based on the first analysis data and the second analysis data.

[0016] Through the technical solution provided by the present invention, the present invention has at least the following technical effects:

[0017] By combining the actual problems encountered in the process of using intelligent models to analyze the patient's fundus lesions, data consistency processing is performed in each link such as obtaining the patient's fundus image, determining the analysis area, and cropping the model input image, so that the patient lesions under analysis are highly consistent at any time, thereby effectively improving the accuracy and reliability of lesion-based disease analysis and meeting actual needs.

[0018] Other features and advantages of the embodiments of the present invention will be described in detail in the subsequent detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the following specific implementations, they are used to explain the embodiments of the present invention, but do not constitute a limitation on the embodiments of the present invention. In the accompanying drawings:

[0020] Figure 1 is a specific implementation flow chart of the lesion analysis method based on fundus images provided by an embodiment of the present invention;

[0021] Figure 2 is a schematic diagram of performing wide-angle image analysis provided by an embodiment of the present invention;

[0022] Figure 3 is a schematic diagram of cropping the first spliced ​​fundus image provided by an embodiment of the present invention;

[0023] Figure 4 It is a structural schematic diagram of a lesion analysis device based on fundus images provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0024] The specific implementation of the embodiment of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the embodiment of the present invention, and is not used to limit the embodiment of the present invention.

[0025] The terms "system" and "network" in the embodiments of the present invention can be used interchangeably. "Multiple" means two or more than two. In view of this, "multiple" can also be understood as "at least two" in the embodiments of the present invention. "And / or" describes the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / ", unless otherwise specified, generally indicates that the previous and subsequent associated objects are in an "or" relationship. In addition, it should be understood that in the description of the embodiments of the present invention, the words "first", "second", etc. are only used to distinguish the purpose of description, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying order.

[0026] The background technology of the present invention is first introduced below.

[0027] In the existing fundus image capture process, the operator takes the image of the patient according to the predetermined operating specifications or relevant standards. However, in the actual application process, due to the great subjectivity of manual operation and the reasons such as jitter, there are slight deviations in the fundus images of the patient taken at different times. If the images taken by the patient at different times are directly input into the intelligent model, it may lead to huge deviations in the lesion detection results. For example, in the first shot, the shooting angle shifted to the left, resulting in excessive inclusion of some lesions on the left. In the second shot, the shooting angle shifted to the right, resulting in a significant reduction in the area of ​​the lesion on the left, and an erroneous diagnosis conclusion was reached that the area of ​​the lesion was greatly reduced and the condition was greatly alleviated.

[0028] To solve the above technical problems, please see Figure 1 The embodiment of the present invention provides a lesion analysis method based on fundus images, the method comprising:

[0029] S10: Acquire a first fundus image set and a second fundus image set, wherein the first fundus image set is taken during an initial period of fundus detection, and the second fundus image set is taken during a subsequent period of fundus detection;

[0030] S20: Determine a stitching priority of each image in the first fundus image set and the second fundus image set;

[0031] S30: performing image stitching on the first fundus image set and the second fundus image set respectively based on the stitching priority to obtain a corresponding first stitched fundus image and a second stitched fundus image;

[0032] S40: performing intersection analysis on the second spliced ​​fundus image based on the first spliced ​​fundus image, processing the first spliced ​​fundus image based on the intersection analysis result to obtain a first intersection image, and processing the second fundus image set based on the intersection analysis result to obtain a second intersection image;

[0033] S50: cropping the first intersection image and the second intersection image respectively to obtain a corresponding first cropped image set and a corresponding second cropped image set, performing lesion analysis on the first cropped image set and the second cropped image set respectively to generate corresponding first analysis data and second analysis data;

[0034] S60: Generate a lesion analysis result based on the first analysis data and the second analysis data.

[0035] In a possible implementation, first obtain a first fundus image set taken by the patient in the first period and a second fundus image set taken in the second period. It should be noted that, based on actual treatment needs, the patient may need to take fundus image sets of multiple periods to continuously analyze the efficacy of medication or determine whether the condition has worsened, etc. Based on the above implementation provided by the embodiment of the present invention, those skilled in the art can think of applying it to the lesion analysis scenario of fundus image sets taken in multiple periods, which should also belong to the protection scope of the embodiment of the present invention, and will not be elaborated on here. Since doctors are actually more concerned about the changes between the lesion conditions in the fundus image taken for the first time and the lesion conditions in the fundus image taken for the last time, it is preferred that the first fundus image set is taken in the initial period of fundus detection, and the second fundus image set is taken in the last period of fundus detection. In an embodiment of the present invention, the above-mentioned first fundus image set and the second fundus image set both include 7 views taken according to the set fundus image shooting specifications.

[0036] After obtaining the fundus image sets taken in the above two periods, a 7-view stitching operation is immediately performed on them for subsequent analysis. In an embodiment of the present invention, in order to ensure the best stitching effect, the existing standard stitching method can be used to stitch them. For example, the stitching order of each fundus image is determined in sequence from the middle to the periphery, and the stitching is performed in this order. However, in the specific implementation process, there may be a situation where the shooting quality of some fundus images is poor. The use of traditional stitching methods may cause the pictures with poor shooting quality to be covered on the pictures with better shooting quality, and reduce the effect of lesion analysis.

[0037] In order to solve the above technical problems, on the one hand, each fundus image is preliminarily sorted according to its stitching priority, and then the best stitching priority is determined in combination with the clarity and stitching weight of each image. In an embodiment of the present invention, the stitching priority of each image in the first fundus image set and the second fundus image set is determined, including: determining the initial stitching order of each image, and determining the stitching weight of each image based on the initial stitching order; evaluating the lesion quality of each image and determining the corresponding quality evaluation result; and determining the stitching priority of each image in the first fundus image set and the second fundus image set based on the stitching weight and the quality evaluation result.

[0038] Specifically, the initial stitching order of each image is first determined. It should be noted that the stitching priority analysis can be performed separately for multiple images in each fundus image set. The stitching weight of each image can be determined according to the initial stitching order. For example, in an embodiment of the present invention, the middle image is determined as order 1, and then the macular image and the optic disc image are determined as orders 2 and 3 respectively. Then, according to a predetermined rule (for example, in a clockwise direction), the order of other surrounding images is determined to be 4-7 in sequence, and the stitching weight of each image is determined in the above order (that is, the importance of each image), for example, the weight of the central image is determined to be 25%. Since doctors are more concerned about the lesion condition in the fundus image, and the quality of other images is not concerned, in an embodiment of the present invention, the lesion quality of each image is further evaluated to generate a quality evaluation result. Finally, the stitching priority of each image is determined according to the stitching weight and quality evaluation result of each image.

[0039] In an embodiment of the present invention, a comprehensive analysis is performed on the stitching importance and shooting quality of each image to determine the optimal stitching priority by taking into account the stitching priority and clarity of the image. Stitching the images according to this priority can effectively ensure the stitching effect of the image, improve the clarity of the lesion display, and improve the accuracy of lesion analysis.

[0040] After determining the stitching priority, the fundus image set is stitched, and the corresponding first stitched fundus image and second stitched fundus image are obtained. Although doctors are required to shoot according to the specifications, due to factors such as hand shaking and shooting deviation, there may be slight deviations between the fundus images taken before and after, and this deviation may cause the size of the same lesion to be displayed inconsistently twice, resulting in a huge change in the lesion analysis data. In order to avoid inconsistency in the lesion area of ​​the two analyses, an intersection analysis is performed to ensure that the lesion analysis of the fundus images taken at different times is for the same lesion area, ensuring consistency before and after, and ensuring the accuracy of the lesion analysis.

[0041] In an embodiment of the present invention, the intersection analysis of the second stitched fundus image based on the first stitched fundus image includes: obtaining a first optic disc feature and a first macular area of ​​the first stitched fundus image, and obtaining a second optic disc feature and a second macular area of ​​the second stitched fundus image; performing image registration processing on the first stitched fundus image and the second stitched fundus image based on the first optic disc feature, the second optic disc feature, the first macular area, and the second macular area; performing intersection analysis on the registered first stitched fundus image and the second stitched fundus image to generate an intersection analysis result.

[0042] It is easy for technicians in this field to know that for two images, even if they are taken by the same person, the same device or even at the same time, it is very difficult to perform pixel-level registration of the two images due to various reasons. The two images taken have more or less perspective changes, which will inevitably lead to a certain degree of "ghosting" in the spliced ​​image.

[0043] In order to improve the stitching efficiency and reduce the stitching difficulty, in a possible implementation, after obtaining the first stitched fundus image and the second stitched fundus image, firstly extract the first optic disc feature and the first macular area of ​​the first stitched fundus image, and extract the second optic disc feature and the second macular area of ​​the second stitched fundus image. Then, the first stitched fundus image and the second stitched fundus image are registered by combining feature points + areas, so as to quickly register the two images within an acceptable error range, and then perform intersection analysis on the two registered images to quickly generate intersection analysis results.

[0044] However, during the shooting process, if the deviation between the two stitched fundus images is too large, the intersection area of ​​the image edge will be smaller, the loss of the analyzable lesion area will be reduced, and the reliability and accuracy of the lesion analysis will be reduced. Therefore, in an embodiment of the present invention, the method also includes: after obtaining the intersection analysis result, obtaining the degree of offset of the first stitched fundus image and the second stitched fundus image based on the intersection analysis result; judging whether the offset degree is greater than a preset offset threshold; if so, obtaining the offset area and the offset orientation; generating and feeding back a corresponding reshoot instruction based on the offset area and the offset orientation.

[0045] Specifically, after obtaining the intersection analysis results, the offset degree of the first spliced ​​fundus image and the second spliced ​​fundus image is further analyzed. If the offset degree is too large, for example, the offset degree of the overall image is greater than the preset offset threshold, the offset area and offset orientation can be immediately obtained, and the corresponding reshoot instruction can be generated. Furthermore, if the technician finds that the offset degree of the intermediate image is greater than the acceptable range, the fundus image set taken at that time can be directly discarded, and an immediate reshoot is required to ensure that the deviation of the fundus image sets taken in the previous and next two periods is within an acceptable range.

[0046] In an embodiment of the present invention, by analyzing the degree of offset of the fundus image sets taken in two previous periods, it is effectively ensured that the degree of offset of the fundus images taken in two previous periods is within an acceptable range, thereby avoiding excessive loss of lesions in the process of taking the intersection, and ensuring the reliability and accuracy of lesion analysis.

[0047] At this time, the first stitched fundus image is processed according to the intersection analysis result to obtain a first intersection image. Specifically, the first stitched fundus image is cropped according to the edge features of the intersection area of ​​the first stitched fundus image to obtain the corresponding first intersection image. Based on the same principle, the second stitched fundus image is processed to obtain a second intersection image.

[0048] In an embodiment of the present invention, by performing intersection processing on images taken at different times, it is possible to effectively avoid huge deviations in lesion detection results due to slight deviations in fundus images taken at different times, thereby ensuring that the lesion area analyzed in the two tests is the same lesion area, thereby ensuring the reliability of the analysis results.

[0049] In actual application, although in general implementation scenarios, doctors or hospitals will require that fundus images taken of patients at different times be taken with equipment of the same specifications, due to the existence of shooting devices with different shooting wide-angles, if the fundus images of the patient taken at two times are taken with equipment with different shooting wide-angles (for example, equipment with different wide-angles is used in the same hospital, or equipment with different wide-angles is used in different hospitals), it will cause mismatch in the fundus images during stitching, cropping and recognition, and further lead to abnormal lesion identification and analysis results.

[0050] In order to solve the above technical problems, in an embodiment of the present invention, the method also includes: before performing the intersection analysis, determining whether the shooting wide angles of the first stitched fundus image and the second stitched fundus image meet a preset consistency requirement; if the preset consistency requirement is not met, performing image normalization processing to obtain a processed first image and a processed second image; and performing intersection analysis on the processed first image and the processed second image.

[0051] In a possible implementation, after obtaining the first spliced ​​fundus image and the second spliced ​​fundus image, it is further determined whether they meet the preset consistency requirements. In an embodiment of the present invention, the determination of whether the shooting wide angles of the first spliced ​​fundus image and the second spliced ​​fundus image meet the preset consistency requirements includes: performing image wide-angle analysis to determine the first shooting wide angle of the first spliced ​​fundus image and the second shooting wide angle of the second spliced ​​fundus image; determining whether the first shooting wide angle and the second shooting wide angle are consistent; if the first shooting wide angle and the second shooting wide angle are equal, determining that the preset consistency requirements are met; if the first shooting wide angle and the second shooting wide angle are not equal, determining that the preset consistency requirements are not met; or: determining a preset reference wide angle; determining whether the first shooting wide angle and the second shooting wide angle are equal to the preset reference wide angle; if the first shooting wide angle and the second shooting wide angle are equal to the preset reference wide angle, determining that the preset consistency requirements are met; if the first shooting wide angle and / or the second shooting wide angle are not equal to the preset reference wide angle, determining that the preset consistency requirements are not met.

[0052] In general, the wide-angle parameters of the shooting device taken before and after the patient should be consistent. In this case, the fundus image sets taken at the two times can be directly analyzed without considering the actual wide-angle parameters of the shooting device, that is, as long as the shooting wide-angle of the fundus image sets taken twice is consistent, the subsequent analysis can be directly performed. Therefore, in the first embodiment, the image wide-angle analysis is first performed on the two fundus images.

[0053] In an embodiment of the present invention, the performing of wide-angle image analysis includes: performing optic disc identification on the target stitched fundus image to obtain optic disc information; determining the optic disc area based on the optic disc information; obtaining an outer arc of the target stitched fundus image, and determining the center and radius based on the outer arc; determining the image area of ​​the target stitched fundus image based on the center and the radius; and determining the shooting wide angle of the target stitched fundus image based on the optic disc area and the image area.

[0054] For details, see Figure 2 In a specific implementation, the optic disc is firstly identified on the target spliced ​​fundus image to obtain the optic disc information, which includes but is not limited to the optic disc position, optic disc area and other information. In an embodiment of the present invention, the target spliced ​​fundus image is characterized by any one of the first spliced ​​fundus image or the second spliced ​​fundus image, which will not be elaborated herein.

[0055] After obtaining the above-mentioned optic disc information, the optic disc area is determined. At this time, the outer arc of the target fundus image after stitching is further obtained. For example, the edge of the target fundus image after stitching is extracted by an edge extraction algorithm to form an outer arc. At this time, the corresponding center and radius are determined according to the outer arc. For example, 3 points are randomly selected on the outer arc, and the corresponding center and radius are calculated according to the coordinate information of the 3 points. The image area of ​​the captured fundus image can be approximately calculated according to the center and radius. At this time, the optic disc ratio can be obtained according to the above-mentioned optic disc area and image area. According to the ratio of the optic disc area in the whole image in the fundus image captured by shooting devices with different shooting wide angles, the shooting wide angle of the current image can be determined. For example, in the fundus images taken by a camera with a shooting wide angle of 30°, the optic disc accounts for 6%-8%; in the fundus images taken by a camera with a shooting wide angle of 45°, the optic disc accounts for 4%-6%; in the fundus images taken by a camera with a shooting wide angle of 55°, the optic disc accounts for 2%-4%; in the fundus images taken by a camera with a shooting wide angle of 90°, the optic disc accounts for less than 1%.

[0056] After performing image wide-angle analysis, the first shooting wide-angle of the first spliced ​​fundus image and the second shooting wide-angle of the second spliced ​​fundus image can be determined. If the first shooting wide-angle is equal to the second shooting wide-angle, it can be determined that the fundus images taken by the patient in the two periods are the same in size, and lesion analysis can be performed directly. Otherwise, it can be determined that the fundus images taken by the patient in the two periods are inconsistent in wide angle, and image normalization processing is required before lesion analysis can be performed, otherwise it will lead to abnormal analysis results.

[0057] For doctors, the size of the fundus image displayed by a 30° shooting wide angle is more reasonable, and the image details are clearer, so they often use a shooting device with a shooting wide angle of 30° to shoot fundus images. However, with the needs of different doctors and different practical application scenarios, doctors may expect to analyze the patient's fundus image at a certain standard shooting wide angle to obtain the best fundus image shooting effect and the most comprehensive fundus information. Therefore, in the second embodiment, after performing the image wide-angle analysis, a preset reference wide angle is further determined. For example, the preset reference wide angle is a standard wide angle pre-specified by the doctor. After obtaining the first shooting wide angle and the second shooting wide angle, it is immediately determined whether they are equal to the preset reference wide angle. If there is any fundus image whose shooting wide angle is not equal to the preset reference wide angle, it can be determined that it does not meet the preset consistency requirements. At this time, it is necessary to perform image normalization processing on the fundus image that does not meet the requirements.

[0058] In an embodiment of the present invention, the performing of image normalization processing includes: determining a reference image and a to-be-processed image from the first stitched fundus image and the second stitched fundus image; determining an optic disc center of the reference image; determining a cropping ratio for the to-be-processed image based on the first shooting wide angle and the second shooting wide angle; performing a cropping operation on the to-be-processed image based on the optic disc center and the cropping ratio to obtain a normalized image set; and determining a corresponding processed first image and the processed second image based on the reference image set and the normalized image set.

[0059] In a possible implementation, if it is detected that the first shooting wide angle and the second shooting wide angle are not equal, a reference image and a to-be-processed image are further determined from the first spliced ​​fundus image and the second spliced ​​fundus image, for example, the first spliced ​​fundus image taken for the first time is used as the reference image, and the subsequently taken fundus images are all used as the to-be-processed images. Then, the optic disc center of the reference image is determined, and in the subsequent image scaling and cropping process, the image is scaled and cropped in equal proportion with the optic disc center as the center of the circle, thereby ensuring the cropping accuracy of fundus images taken at different times.

[0060] At this time, the cropping ratio for the image to be processed is further determined according to the first shooting wide angle and the second shooting wide angle. For example, in a specific embodiment, the fundus image with a shooting wide angle of 45° is registered with the fundus image with a shooting wide angle of 30°. First, the optic disc center of the fundus image with a shooting wide angle of 30° is determined, and then the circular coverage of the fundus image with a 45° wide angle is gradually expanded with the optic disc center as the center, and the pixel mean square error between the circular coverage and the fundus image with a shooting wide angle of 30° is calculated in real time, and the value with the minimum mean square error is found. At this time, the magnification corresponding to the value is the cropping ratio for registering the fundus image with a wide angle of 45° to the fundus image with a shooting wide angle of 30°.

[0061] Finally, a cropping operation is performed on the image to be processed according to the optic disc center of the reference image and the determined cropping ratio to obtain a normalized image, thereby obtaining a processed first image and a processed second image.

[0062] In the second embodiment, the shooting wide angles used in the two sets of fundus images taken by the patient do not meet the preset reference wide angle. Therefore, in the subsequent normalization process, the first spliced ​​fundus image and the second spliced ​​fundus image are respectively normalized according to the preset reference wide angle to obtain the processed first image and the processed second image to meet actual needs.

[0063] After all fundus image sets are configured to have imaging areas corresponding to a uniform and standard shooting wide angle, intersection analysis is performed on them to obtain accurate intersection analysis results.

[0064] In an embodiment of the present invention, by performing corresponding processing on the fundus image according to its actual shooting situation, it is ensured that the fundus images taken at different times are based on the same shooting standard, which effectively guarantees the accuracy and reliability of the subsequent image stitching and cropping process and ensures the accuracy of lesion diagnosis.

[0065] After obtaining the first intersection image and the second intersection image, they are cropped respectively. In some conventional cropping methods, the fundus image can be cropped by a sliding window method. However, in the actual application process, the technicians found that there are at least the following technical problems: on the one hand, when training the intelligent recognition model of the lesion, the technicians pre-select a large number of complete retinal images and input the training according to the input format of the intelligent recognition model. After the fundus image is cropped by the sliding window, there are a large number of blank areas or abnormal areas at the edge of the fundus image. When it is input into the intelligent model for lesion recognition, the lesion recognition effect is greatly reduced due to the large difference from the training image; on the other hand, after the image cropped by the above sliding window is input into the intelligent model, some images lack the front and rear blood vessel information, resulting in some blood vessels being identified as lesions, thereby reducing the recognition accuracy.

[0066] To solve the above technical problems, please see Figure 3 In an embodiment of the present invention, the cropping of the first intersection image to obtain a first cropped image set includes: acquiring an optic disc and a macula of the first intersection image; determining an upper dividing line based on an optic disc vertex of the optic disc and a macula vertex of the macula, determining a lower dividing line based on an optic disc bottom point of the optic disc and a macula bottom point of the macula, and determining a vertical dividing line based on an optic disc center point of the optic disc; determining seven segmentation areas based on the upper segmentation line, the lower segmentation line, and the vertical segmentation line; and cropping the first intersection image based on the seven segmentation areas to obtain a first cropped image set.

[0067] In a possible implementation, after obtaining the intersection image of each shot, seven segmentation areas are constructed, and the seven segmentation areas are used to crop it. Specifically, the optic disc and macula of the first intersection image are obtained, for example, the optic disc area and the macula area are identified from the first intersection image using a deep neural detection model, and then the upper segmentation line, the lower segmentation line and the vertical segmentation line are determined according to the relevant features of the optic disc and the macula, such as the vertex, the bottom point, the center point, etc., wherein the vertical segmentation line passes through the center point of the optic disc and is perpendicular to the upper segmentation line and the lower segmentation line, and the first intersection image is divided into 4 areas by the above three lines.

[0068] Then determine the radius of the cropping circle. Specifically, the radius of the fundus image in the reference image can be used as the radius of the cropping circle, or the radius of the fundus image taken with a wide angle of 30° can be used as the radius of the cropping circle. At this time, the seven divided areas to be cropped are further determined according to the cropping rules of the fundus image and the radius of the above-mentioned cropping circle. For example, in the embodiment of the present invention, the radius of the cropping circle is used as the radius, and the area covered by the center of the optic disc is determined as the first area; the area covered by the center of the macula is determined as the second area; the area covered by the point where 1-1.5 optic disc diameters are located on the temporal side of the macula is determined as the third area; the area covered by the area located above the second area and circumscribed to the vertical dividing line and the upper dividing line is determined as the fourth area; based on the same principle as the fourth area, the fifth area, the sixth area and the seventh area can be obtained respectively, thereby realizing the standardization process of each image cropping, and on the basis of accurate and reliable fundus image splicing, it is ensured that the 7-area images cropped for the fundus images taken in each period are highly consistent.

[0069] Finally, the cropped image set consisting of the 7-zone map is subjected to lesion recognition, for example, the cropped image set consisting of the 7-zone map is input into an intelligent model, for example, the intelligent model is an intelligent recognition model generated by pre-training based on a conventional deep neural network model or the like, the conventional deep neural network model includes but is not limited to a convolutional neural network model, a recurrent neural network model, a U2-Net model, etc., which will not be described in detail here. Through the intelligent model, the lesion information of each fundus image can be effectively recognized. For example, in an embodiment of the present invention, after performing lesion analysis on the first cropped image set through a preset deep learning model, first analysis data including lesion density, total lesion area, number of lesions and other data can be generated; based on the same principle, second analysis data of lesions in the fundus image of the patient in the second shooting period can be analyzed, and the first analysis data strictly corresponds to the second analysis data. Therefore, based on the above-mentioned first analysis data and second analysis data, accurate and reliable data support can be provided for the changes in the patient's condition at different periods. Lesion analysis is performed on this basis, and accurate and reliable lesion analysis results for the patient can be obtained. For example, the patient's lesions can be quantitatively graded according to the ETDRS grading standard, thereby obtaining accurate grading results.

[0070] In an embodiment of the present invention, by introducing the seven-zone segmentation method in the cropping process of the fundus image, the vascular information in the fundus image can be maximized, which greatly reduces the recognition error of the intelligent model; at the same time, through the above-mentioned seven-zone segmentation method, the image content can be maximized in the cropped image, which greatly reduces the formal difference between the image input for analysis in the intelligent model and the training image, thereby effectively improving the recognition accuracy of the cropped image by the intelligent model, meeting the actual needs of doctors and improving the accuracy of lesion recognition.

[0071] See also Figure 4 Based on the same inventive concept, an embodiment of the present invention provides a lesion analysis device based on fundus images, the device comprising: an image set acquisition unit, used to acquire a first fundus image set and a second fundus image set, the first fundus image set being taken during an initial period of fundus detection, and the second fundus image set being taken during a subsequent period of fundus detection; a priority determination unit, used to determine the stitching priority of each image in the first fundus image set and the second fundus image set; a stitching unit, used to perform image stitching on the first fundus image set and the second fundus image set respectively based on the stitching priority, to obtain corresponding first stitched fundus images and second stitched fundus images; an intersection processing unit, used to perform image stitching on the first fundus image set and the second fundus image set based on the stitching priority, to obtain corresponding first stitched fundus images and second stitched fundus images; and The first stitched fundus image performs intersection analysis on the second stitched fundus image, processes the first stitched fundus image based on the intersection analysis result to obtain a first intersection image, and processes the second stitched fundus image based on the intersection analysis result to obtain a second intersection image; a cropping and analysis unit is used to crop the first intersection image and the first intersection image respectively to obtain the corresponding first cropped image set and second cropped image set, perform lesion analysis on the first cropped image set and the second cropped image set respectively, and generate corresponding first analysis data and second analysis data; a result generation unit is used to generate a lesion analysis result based on the first analysis data and the second analysis data.

[0072] The optional implementation modes of the embodiments of the present invention are described in detail above in conjunction with the accompanying drawings. However, the embodiments of the present invention are not limited to the specific details in the above implementation modes. Within the technical concept of the embodiments of the present invention, various simple modifications can be made to the technical scheme of the embodiments of the present invention, and these simple modifications all belong to the protection scope of the embodiments of the present invention.

[0073] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, the embodiments of the present invention will not further describe various possible combinations.

[0074] Those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, and the program is stored in a storage medium, including a number of instructions to enable a single-chip microcomputer, a chip or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0075] In addition, various implementations of the embodiments of the present invention may be arbitrarily combined, and as long as they do not violate the concept of the embodiments of the present invention, they should also be regarded as the contents disclosed in the embodiments of the present invention.

Claims

1. A lesion analysis method based on fundus images, characterized in that: The method comprises: Acquire a first fundus image set and a second fundus image set, wherein the first fundus image set is taken during an initial period of fundus detection, and the second fundus image set is taken during a subsequent period of fundus detection; Determining a stitching priority of each image in the first fundus image set and the second fundus image set; Based on the stitching priority, respectively perform image stitching on the first fundus image set and the second fundus image set to obtain a corresponding first stitched fundus image and a second stitched fundus image; performing an intersection analysis on the second spliced ​​fundus image based on the first spliced ​​fundus image, processing the first spliced ​​fundus image based on the intersection analysis result to obtain a first intersection image, and processing the second spliced ​​fundus image based on the intersection analysis result to obtain a second intersection image; Cropping the first intersection image and the second intersection image respectively to obtain a corresponding first cropped image set and a corresponding second cropped image set, and performing lesion analysis on the first cropped image set and the second cropped image set respectively to generate corresponding first analysis data and second analysis data; A lesion analysis result is generated based on the first analysis data and the second analysis data.

2. The method according to claim 1, characterized in that The determining the stitching priority of each image in the first fundus image set and the second fundus image set includes: Determining an initial stitching order for each image, and determining a stitching weight for each image based on the initial stitching order; Evaluate the lesion quality of each image and determine the corresponding quality evaluation result; The stitching priority of each image in the first fundus image set and the second fundus image set is determined based on the stitching weight and the quality assessment result.

3. The method according to claim 1, characterized in that The performing intersection analysis on the second spliced ​​fundus image based on the first spliced ​​fundus image includes: Acquire a first optic disc feature and a first macular area of ​​the first spliced ​​fundus image, and acquire a second optic disc feature and a second macular area of ​​the second spliced ​​fundus image; performing image registration processing on the first stitched fundus image and the second stitched fundus image based on the first optic disc feature, the second optic disc feature, the first macular area, and the second macular area; An intersection analysis is performed on the registered first spliced ​​fundus image and the second spliced ​​fundus image to generate an intersection analysis result.

4. The method according to claim 3, characterized in that The method further comprises: After obtaining the intersection analysis result, obtaining the degree of deviation between the first spliced ​​fundus image and the second spliced ​​fundus image based on the intersection analysis result; Determining whether the degree of deviation is greater than a preset deviation threshold; If yes, obtain the offset area and offset direction; A corresponding retake instruction is generated and fed back based on the offset area and the offset orientation.

5. The method according to claim 1, characterized in that The method further comprises: Before performing the intersection analysis, determining whether the shooting wide angles of the first spliced ​​fundus image and the second spliced ​​fundus image meet a preset consistency requirement; If the preset consistency requirement is not met, performing image normalization processing to obtain a processed first image and a processed second image; An intersection analysis is performed on the processed first image and the processed second image.

6. The method according to claim 5, characterized in that The determining whether the shooting wide angles of the first spliced ​​fundus image and the second spliced ​​fundus image meet a preset consistency requirement includes: Perform image wide-angle analysis to determine a first shooting wide angle of the first spliced ​​fundus image and a second shooting wide angle of the second spliced ​​fundus image; determine whether the first shooting wide angle and the second shooting wide angle are consistent; if the first shooting wide angle and the second shooting wide angle are equal, determine that the preset consistency requirement is met; if the first shooting wide angle and the second shooting wide angle are not equal, determine that the preset consistency requirement is not met; or: Determine a preset reference wide angle; determine whether the first shooting wide angle and the second shooting wide angle are equal to the preset reference wide angle; if the first shooting wide angle and the second shooting wide angle are equal to the preset reference wide angle, determine that the preset consistency requirement is met; if the first shooting wide angle and / or the second shooting wide angle are not equal to the preset reference wide angle, determine that the preset consistency requirement is not met.

7. The method according to claim 6, characterized in that The performing of image wide-angle analysis comprises: Perform optic disc recognition on the target spliced ​​fundus image to obtain optic disc information; determining an optic disc area based on the optic disc information; Acquire the outer arc of the target spliced ​​fundus image, and determine the center and radius based on the outer arc; Determine the image area of ​​the target spliced ​​fundus image based on the center of the circle and the radius; The shooting wide angle of the target stitched fundus image is determined based on the optic disc area and the image area.

8. The method according to claim 7, characterized in that The performing of image normalization processing comprises: Determine a reference image and an image to be processed from the first spliced ​​fundus image and the second spliced ​​fundus image; determining the optic disc center of the reference image; Determining a cropping ratio for each fundus image in the to-be-processed image set based on the first shooting wide angle and the second shooting wide angle; Performing a cropping operation on the image to be processed based on the optic disc center and the cropping ratio to obtain a normalized image; A corresponding processed first image and processed second image are determined based on the reference image and the normalized image.

9. The method according to claim 1, characterized in that: The step of cropping the first intersection image to obtain a first cropped image set includes: acquiring the optic disc and macula of the first intersection image; Determine an upper dividing line based on the optic disc vertex of the optic disc and the macula vertex of the macula, determine a lower dividing line based on the optic disc base point of the optic disc and the macula base point of the macula, and determine a vertical dividing line based on the optic disc center point of the optic disc; Determine seven partitions based on the upper partition line, the lower partition line, and the vertical partition line; The first intersection image is cropped based on the segmented seven regions to obtain a first cropped image set.

10. A lesion analysis device based on fundus images, characterized in that: The device is applied to the method according to any one of claims 1 to 9, and the device comprises: An image set acquisition unit, used to acquire a first fundus image set and a second fundus image set, wherein the first fundus image set is taken during an initial period of fundus detection, and the second fundus image set is taken during a subsequent period of fundus detection; a priority determination unit, configured to determine a stitching priority of each image in the first fundus image set and the second fundus image set; a stitching unit, configured to perform image stitching on the first fundus image set and the second fundus image set respectively based on the stitching priority, to obtain a corresponding first stitched fundus image and a second stitched fundus image; an intersection processing unit, configured to perform an intersection analysis on the second spliced ​​fundus image based on the first spliced ​​fundus image, process the first spliced ​​fundus image based on the intersection analysis result to obtain a first intersection image, and process the second spliced ​​fundus image based on the intersection analysis result to obtain a second intersection image; a cropping and analysis unit, configured to crop the first intersection image and the second intersection image respectively to obtain a corresponding first cropped image set and a corresponding second cropped image set, and to perform lesion analysis on the first cropped image set and the second cropped image set respectively to generate corresponding first analysis data and second analysis data; A result generating unit is used to generate a lesion analysis result based on the first analysis data and the second analysis data.

Images