Apparatus, method, and related products for identifying lesions of the peripheral fundus

Abstract

The application discloses a device, method and related product for identifying lesions of the fundus periphery. The device comprises a processor and a memory storing program instructions for identifying lesions of the fundus periphery, which, when executed by the processor, causes the device to perform the following operations: obtaining an ultra-wide-angle fundus image to be identified for lesions; detecting the fundus image using a detection model to obtain a fundus edge image containing a fundus edge region; performing a cropping operation on the fundus edge image to obtain intermediate fundus edge images at multiple scales; and identifying the intermediate fundus edge images at the multiple scales using an identification model, respectively, to determine an identification result of identifying lesions of the fundus periphery. The application can improve the accuracy of identifying lesions of the fundus periphery.

Description

TECHNICAL FIELD

[0001] The present application generally relates to the technical field of fundus lesion recognition. More specifically, the present application relates to an apparatus, a method, a device and a computer readable storage medium for recognizing lesions in the periphery of the fundus. BACKGROUND

[0002] It is found through research that more than 2.2 billion people suffer from visual impairment or blindness, of which more than 1 billion people can effectively prevent the disease from worsening through early prevention or effective treatment. Therefore, early detection and correct identification of fundus diseases are of great significance to prevent visual impairment. With the development of artificial intelligence, it is expected to use machine learning or deep learning to process fundus images to provide a better solution for prevention and identification. For example, using deep learning to process ultra-wide-angle fundus images is used for identification and prevention of fundus diseases. Compared with ordinary fundus images, ultra-wide-angle fundus images can obtain a wider range of retinal field of view, improve the visibility of fundus images, and significantly reduce the complexity of fundus disease identification process, providing more valuable information and data for fundus disease identification and prevention.

[0003] However, due to the wide coverage of ultra-wide-angle fundus images and the presence of many interference objects (such as eyelashes, eyelids and masks), it not only increases the difficulty of model learning, but also ignores smaller lesions, so that the fundus lesion recognition result is inaccurate. In addition, due to the wide coverage of ultra-wide-angle fundus images, some early lesions that are easily overlooked can be found around the fundus, but the current method does not have research on marginal lesion identification. SUMMARY

[0004] In order to at least partially solve the technical problems mentioned in the background, the scheme of the present application provides a scheme for identifying lesions in the periphery of the fundus. Using the scheme of the present application, the difficulty of model learning can be reduced, and an accurate identification result of identifying lesions in the periphery of the fundus can be obtained. To this end, the present application provides a solution in the following aspects.

[0005] In a first aspect, the present application provides an apparatus for identifying lesions in the periphery of the fundus, comprising: a processor; and a memory storing program instructions for identifying lesions in the periphery of the fundus, when the program instructions are executed by the processor, the apparatus performs the following operations: obtaining an ultra-wide-angle fundus image to be identified for lesion identification; using a detection model to detect the fundus image to obtain a fundus edge image containing a fundus edge region; performing a cropping operation on the fundus edge image to obtain intermediate fundus edge images under multiple scales; and using an identification model to identify the intermediate fundus edge images under the multiple scales respectively to determine an identification result of identifying lesions in the periphery of the fundus.

[0006] In one embodiment, wherein the program instructions, when further executed by the processor, cause the apparatus to obtain the fundus margin image containing the fundus margin region by: detecting a target region in the fundus image using a detection model; and obtaining the fundus margin image containing the fundus margin region based on the target region.

[0007] In another embodiment, wherein the program instructions, when further executed by the processor, cause the apparatus to obtain the fundus margin image containing the fundus margin region by: locating the target region in the fundus image, determining a position of the target region; and obtaining the fundus margin image containing the fundus margin region based on the position of the target region.

[0008] In yet another embodiment, wherein the program instructions, when further executed by the processor, cause the apparatus to determine the position of the target region by: taking a horizontal boundary in the fundus image as a reference boundary, calculating a distance of the target region from the reference boundary; comparing the distance of the target region from the reference boundary with a locating threshold; and determining the position of the target region according to a comparison result that the target region is located at an upper part, a lower part or a middle part of the fundus image.

[0009] In yet another embodiment, wherein the locating threshold comprises a first locating threshold and a second locating threshold, and wherein the program instructions, when further executed by the processor, cause the apparatus to determine the position of the target region according to a comparison result that the target region is located at an upper part, a lower part or a middle part of the fundus image by: in response to the distance of the target region from the reference boundary being less than the first locating threshold or the distance of the target region from the reference boundary being greater than the second locating threshold, determining that the target region is located at the upper part or the lower part of the fundus image; or in response to the distance of the target region from the reference boundary being between the first locating threshold and the second locating threshold, determining that the target region is located at the middle part of the fundus image.

[0010] In yet another embodiment, wherein the program instructions, when further executed by the processor, cause the apparatus to obtain the fundus margin image containing the fundus margin region by: determining a fundus margin region based on the position of the target region; and obtaining the fundus margin image containing the fundus margin region according to the fundus margin region.

[0011] In yet another embodiment, wherein when the program instructions are further executed by the processor, the apparatus is caused to perform the following operation to determine the fundus edge region: in response to the target region being located at an upper or lower portion of the fundus image, comparing distances of the target region to respective horizontal boundaries of the fundus image and determining a side farthest from the respective horizontal boundaries as the fundus edge region; or in response to the target region being located at a middle portion of the fundus image, comparing distances of the target region to respective boundaries of the fundus image and determining a side farthest from the respective boundaries as the fundus edge region.

[0012] In yet another embodiment, wherein when the program instructions are further executed by the processor, the apparatus is caused to perform the following operation to obtain the fundus edge image containing the fundus edge region: setting a target distance, and cropping the fundus edge region at the target distance away from the target region to obtain the fundus edge image containing the fundus edge region.

[0013] In yet another embodiment, wherein when the program instructions are further executed by the processor, the apparatus is caused to perform the following operation to determine the identification result of identifying the lesion of the fundus periphery: using an identification model to respectively identify the intermediate fundus edge images at multiple scales to obtain identification results corresponding to the multiple scales; and determining the identification result of identifying the lesion of the fundus periphery based on the identification results corresponding to the multiple scales.

[0014] In yet another embodiment, wherein when the program instructions are further executed by the processor, the apparatus is caused to perform the following operation to determine the identification result of identifying the lesion of the fundus periphery: in response to multiple identification results in the identification results corresponding to the multiple scales being identical, determining the identical identification result corresponding to the multiple identification results as the identification result of identifying the lesion of the fundus periphery; or in response to the identification results corresponding to the multiple scales each being different, determining an average value of the identification results corresponding to the multiple scales as the identification result of identifying the lesion of the fundus periphery.

[0015] In a second aspect, the present application also provides a method for identifying a lesion of a fundus periphery, comprising: acquiring a super-wide-angle fundus image to be subjected to lesion identification; using a detection model to detect the fundus image to obtain a fundus edge image containing a fundus edge region; performing a cropping operation on the fundus edge image to obtain intermediate fundus edge images at multiple scales; and using an identification model to respectively identify the intermediate fundus edge images at the multiple scales to determine an identification result of identifying the lesion of the fundus periphery.

[0016] In a third aspect, the present application also provides a computer readable storage medium having stored thereon computer readable instructions for identifying a lesion at the periphery of the fundus, which, when executed by one or more processors, implement the method of the preceding second aspect.

[0017] Through the scheme of the present application, the fundus edge image is obtained by detecting the fundus image using the detection model, and the fundus edge image is cropped into multiple scales, and then the intermediate fundus edge image under the multiple scales is identified by the identification model to determine the identification result of the lesion at the periphery of the fundus. Based on this, the present application embodiment can filter out the interference in the fundus image and reduce irrelevant information by detecting the fundus edge image through the detection model, thereby reducing the learning difficulty of the identification model. Further, the present application embodiment can identify the smaller lesions of the fundus edge image by cropping the fundus edge image into multiple scales, thereby obtaining an accurate identification result for the lesion at the periphery of the fundus, so as to identify the lesion at the periphery of the fundus as soon as possible and take effective intervention in time. BRIEF DESCRIPTION OF DRAWINGS

[0018] The above and other objects, features and advantages of the exemplary embodiments of the present application will be more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0019] Figure 1 is an exemplary schematic diagram showing a super-wide-angle fundus image and a normal fundus image;

[0020] Figure 2 is an exemplary structural block diagram of an apparatus for identifying a lesion at the periphery of the fundus according to an embodiment of the present application;

[0021] Figure 3 is an exemplary schematic diagram showing a target region in multiple position states of a fundus image according to an embodiment of the present application;

[0022] Figure 4 is an exemplary schematic diagram showing a fundus edge image containing a fundus edge region obtained according to an embodiment of the present application;

[0023] Figure 5 is an exemplary schematic diagram showing intermediate fundus images of multiple scales according to an embodiment of the present application;

[0024] Figure 6 is an exemplary schematic diagram showing identification of intermediate fundus images of three scales according to an embodiment of the present application;

[0025] Figure 7is an exemplary flow chart illustrating a method for identifying lesions at the periphery of the fundus according to an embodiment of the present application; and

[0026] Figure 8 is an exemplary structural block diagram of an apparatus for identifying lesions at the periphery of the fundus according to an embodiment of the present application. DETAILED DESCRIPTION

[0027] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings. It should be understood that the embodiments described in the specification are only some of the embodiments of the present application provided for the convenience of clear understanding of the solutions and to meet the legal requirements, and not all the embodiments of the present application can be implemented. Based on the embodiments disclosed in the specification, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0028] Figure 1 is an exemplary schematic diagram showing a super-wide-angle fundus image and a normal fundus image. As Figure 1 The left image in the figure shows a super-wide-angle fundus image, and the right image in the figure shows a normal fundus image. As described in the background section, compared with the normal fundus image, the super-wide-angle fundus image can obtain a wider range of retinal field of view, but there are the following defects when using deep learning to process the super-wide-angle fundus image:

[0029] In the first aspect, since the super-wide-angle fundus image has a wide range of retinal field of view, its corresponding long and wide pixel values usually reach more than 3000, and the learning of the model in the deep learning technology is limited by the display memory, and usually the size of the data picture used for learning is scaled to about 512 pixels. This will cause some small lesions to become invisible, not only increasing the learning difficulty of the model, but also making the result of identifying the lesions inaccurate.

[0030] In the second aspect, there are many interference objects in the super-wide-angle fundus image, such as eyelashes, eyelids and masks, etc. This makes the eye fundus markers tend to be complex, which also increases the learning difficulty of the model.

[0031] In the third aspect, since the super-wide-angle fundus image has a wide range of retinal field of view, some easily overlooked early lesions of the surrounding fundus can be found, and thus the development thereof can be controlled to avoid further deterioration. However, the current method does not have research on the identification of marginal lesions.

[0032] In view of this, the embodiments of the present application provide a scheme for identifying lesions at the periphery of the fundus, which can reduce the learning difficulty of the model and improve the accuracy of identifying the lesions at the periphery of the fundus, so as to facilitate the timely intervention of the lesions at the periphery of the fundus.

[0033] Figure 2 is an exemplary structural block diagram illustrating an apparatus 200 for identifying lesions of the periphery of the fundus according to an embodiment of the present application. As shown in Figure 2 the apparatus 200 can include a processor 201 and a memory 202. The processor 201 can include, for example, a general-purpose processor (“CPU”) or a dedicated graphics processor (“GPU”), and the memory 202 stores program instructions executable on the processor. In some embodiments, the memory 202 can include, but is not limited to, a resistive random access memory (“RRAM”), a dynamic random access memory (“DRAM”), a static random access memory (“SRAM”), and an enhanced dynamic random access memory (“EDRAM”).

[0034] Further, the memory 202 can store program instructions for identifying lesions of the periphery of the fundus, which when executed by the processor, cause the apparatus to perform the following operations: obtaining a wide-angle fundus image to be identified for lesions, detecting the fundus image using a detection model to obtain a fundus edge image containing a fundus edge region, then performing a cropping operation on the fundus edge image to obtain intermediate fundus edge images at multiple scales, and further identifying the intermediate fundus edge images at multiple scales using an identification model to determine an identification result of identifying lesions of the periphery of the fundus. That is, the apparatus 200 of the present embodiment identifies lesions of the periphery of the fundus by detecting the obtained wide-angle fundus image 203 to obtain a fundus edge image, then cropping the fundus edge image into intermediate fundus edge images at multiple scales, and further identifying the intermediate fundus edge images at different scales to determine the identification result 204 of identifying lesions of the periphery of the fundus.

[0035] In one embodiment, the aforementioned ultra-wide angle fundus image can be obtained by, for example, an ultra-wide angle optical camera. After obtaining the fundus image, the fundus image can be detected by using a detection model to obtain a fundus edge image containing a fundus edge region. Specifically, first, the target region in the fundus image can be detected by using the detection model, and then the fundus edge image containing the fundus edge region can be obtained based on the target region. More specifically, the target region can be located in the fundus image, the position of the target region can be determined, and then the fundus edge image containing the fundus edge region can be obtained based on the position of the target region. In one implementation scenario, the aforementioned detection model can be, for example, a Faster R-CNN network model, and the aforementioned target region can include the region where the macula and optic disc are located (for example, as shown in FIG. 1). Figure 4

[0036] It can be understood that the aforementioned Faster R-CNN network model includes four parts, namely a feature extraction part, an RPN part, a Proposal Layer part, and an ROI Pooling part. Among them, the aforementioned feature extraction part includes multiple convolution and pooling layers to extract a feature map of the fundus image. The aforementioned RPN part is used to obtain the approximate position of the target region from the feature map of the fundus image. Further, the aforementioned Proposal Layer part is used to obtain a more accurate position according to the approximate position of the target region, and the aforementioned ROI Pooling part can use the aforementioned obtained more accurate position to determine the target, such as the macula and the optic disc, from the feature map of the fundus image. Among them, the accurate position of the target region can include the coordinates of the target (such as the macula and the optic disc), and the position of the target region can be determined according to the coordinates of the macula and the optic disc, for example, the center of the target region can be determined.

[0037] In an implementation scenario, when the fundus image of the examinee is collected, the eyeball of the examinee will rotate, so that the position of the target region will change, and the position of the corresponding fundus edge region will also change. Therefore, in the embodiments of the present application, based on the coordinates of the target obtained by the aforementioned detection model, the position of the target region in the fundus image can be determined, for example, it can be determined whether the target region is in the upper, lower or middle part of the fundus image, and then the fundus edge image containing the fundus edge region can be obtained.

[0038] ​Specifically, in one embodiment, the horizontal boundary in the fundus image is first used as a reference boundary. The distance from the target region to the reference boundary is calculated, and this distance is compared with a positioning threshold. Then, based on the comparison result, the target region is determined to be above, below, or in the middle of the fundus image, thus determining the position of the target region. The aforementioned positioning threshold may include a first positioning threshold and a second positioning threshold. In an implementation scenario, if the distance from the target region to the reference boundary is less than the first positioning threshold or greater than the second positioning threshold, the target region is determined to be above or below the fundus image. Alternatively, if the distance from the target region to the reference boundary is between the first and second positioning thresholds, the target region is determined to be in the middle of the fundus image.

[0039] In the implementation scenario, the aforementioned first and second positioning thresholds can be divided based on the length of the vertical boundary (e.g., Figure 3 (As shown). Assuming the length of the vertical boundary is h, the aforementioned first positioning threshold can be set to 1 / 3h, and the aforementioned second positioning threshold can be set to 2 / 3h. In this scenario, when the distance between the target region and the reference boundary is less than 1 / 3h, the target region is determined to be above the fundus image; when the distance between the target region and the reference boundary is greater than 2 / 3h, the target region is determined to be below the fundus image; when the distance between the target region and the reference boundary is between 1 / 3h and 2 / 3h, the target region is determined to be in the middle of the fundus image.

[0040] Based on the location of the target region in the fundus image as determined above, the fundus edge region can be determined. In one embodiment, in response to the target region being located above or below the fundus image, the distances of the target region from each horizontal boundary of the fundus image are compared, and the side furthest from the corresponding horizontal boundary is determined as the fundus edge region. Alternatively, in response to the target region being located in the middle of the fundus image, the distances of the target region from each boundary of the fundus image are compared, and the side furthest from the corresponding boundary is determined as the fundus edge region.

[0041] In other words, when determining the location of the target region in this embodiment, it can first determine whether the target region is above, below, or in the middle of the fundus image based on the vertical distance of the target region from the reference boundary. When it is determined that the target region is above or below the fundus image, the distances of the target region from the upper and lower boundaries of the fundus image are compared, and the side with the greatest vertical distance from the upper and lower boundaries is determined as the fundus edge region. In an exemplary scenario, when the target region is above the fundus image, since the target region is farthest from the lower boundary of the fundus image, the side farthest from the lower boundary of the fundus image is determined as the fundus edge region.

[0042] In one implementation scenario, when the target region is in the middle of the fundus image, the distances of the target region to each boundary of the fundus image need to be compared, i.e., the distances of the target region to the upper and lower boundaries and the left and right boundaries of the fundus image are compared, and the side farthest from the upper and lower boundaries and the left and right boundaries (i.e., the maximum value of the distances to each boundary) is selected as the fundus edge region. In one exemplary scenario, when the target region is in the middle of the fundus image, and the distance of the target region to the left boundary of the fundus image is the largest, the side away from the left boundary of the fundus image is determined as the fundus edge region. It can be understood that when the distances of the target region to each boundary are calculated in the embodiments of the present application, the distances from the center of the target region to each boundary are calculated.

[0043] After the fundus edge region is determined, the fundus edge region can be cropped to obtain a fundus edge image containing the fundus edge region. In one embodiment, a target distance can be set, and the fundus edge region can be cropped at the target distance away from the target region to obtain a fundus edge image containing the fundus edge region. The aforementioned target distance can be, for example, a distance of a preset multiple (e.g., 3) of the distance from the center of the macula in the target region to the center of the optic disc. As an example, assuming that the distance from the center of the macula in the target region to the center of the optic disc is a d , the aforementioned target distance can be, for example, 3 * a d . In this scenario, the fundus edge region can be cropped at a distance of 3 * a d away from the center of the target region to obtain a fundus edge image containing the fundus edge region. How to obtain a fundus edge image containing the fundus edge region will be described in detail later. Figures 3-4

[0044] Based on the fundus edge image obtained as described above, the device of the embodiments of the present application further performs a cropping operation on the fundus edge image to obtain intermediate fundus edge images at multiple scales. In some embodiments, the fundus edge image can be uniformly cropped in the horizontal direction or the vertical direction to obtain intermediate fundus edge images at multiple scales (e.g., as shown in FIG. 6). Figure 5 Then, the identification model can be used to identify the intermediate fundus edge images at multiple scales respectively to determine the identification result of identifying the lesions around the fundus. Specifically, the identification model is used to identify the intermediate fundus edge images at multiple scales respectively to obtain identification results corresponding to multiple scales, so as to determine the identification result of identifying the lesions around the fundus based on the identification results corresponding to multiple scales.

[0045] ​In one implementation scenario, the identification model can include multiple, each of which can be, for example, a ResNet-50 network model, and each of which respectively identifies an intermediate fundus edge image of one scale type, thereby obtaining identification results corresponding to multiple scales. Wherein, in response to multiple identification results in the identification results corresponding to multiple scales being the same, the same identification result corresponding to the multiple identification results is determined as the identification result of identifying the lesion on the fundus periphery. Or, in response to the identification results corresponding to multiple scales each being different, the average of the identification results corresponding to multiple scales is determined as the identification result of identifying the lesion on the fundus periphery. In other words, when there are multiple same identification results in the identification results output by each identification model, the multiple same identification results are taken as the final identification result. When the identification results output by each identification model are all different, the average of the multiple identification results is taken as the final identification result. Details will be described later in combination with Figure 6 The foregoing identification process is described in detail.

[0046] As can be known from the foregoing description, the embodiments of the present application can filter out the interference information in the fundus image by target positioning on the ultra-wide-angle fundus image and obtaining the fundus edge image based on the positioned target region, and reduce the identification difficulty of model learning. Further, the embodiments of the present application cut the fundus edge image into intermediate fundus edge images of different scales, and use multiple identification models to respectively identify the intermediate fundus edge images of different scales. Then, the accurate identification result of the lesion on the fundus periphery is determined according to the identification results under different scales. Based on this, it can be ensured that the smaller lesion in the fundus edge image can also be identified, so as to obtain the accurate identification result of the lesion on the fundus periphery, so as to identify the lesion on the fundus periphery as soon as possible and take effective intervention in time.

[0047] Figure 3 is an exemplary schematic diagram showing the multiple position states of the target region in the fundus image according to the embodiments of the present application. As Figure 3 As shown in the figure, when the eyeball of the examinee changes, the target region (as shown by multiple small rectangular boxes in the figure) is in eight position situations in the fundus image, including the target region being in the upper part (upper left, upper right), lower part (lower left, lower right) and middle part of the fundus image. As described previously, when determining the position of the target region, first, the horizontal boundary in the fundus image can be taken as the reference boundary, then the distance of the target region to the reference boundary is calculated, and then the position of the target region is determined according to the comparison result of the distance and the first positioning threshold and the second positioning threshold. Wherein, the distance of the target region to the reference boundary refers to the distance of the center of the target region to the reference boundary, and the first positioning threshold and the second positioning threshold can be set to 1 / 3h and 2 / 3h respectively, h representing the length of the vertical boundary of the fundus image.

[0048] As an example, the upper boundary in the fundus image is selected as the reference boundary, and it is assumed that the distance from the center of the target region to the reference boundary is h1. When h1<1 / 3h, it is determined that the target region is located in the upper part of the fundus image; when 1 / 3h

[0049] For the target region located in the middle part of the fundus image, the distances of the target region from the upper and lower boundaries and the left and right boundaries of the fundus image need to be compared, and the side farthest from the upper, lower, left and right boundaries is selected as the fundus edge region. For example, it is assumed that the distances of the target region located in the middle part of the fundus image from the upper and lower boundaries and the left and right boundaries are h4, h5 and w1, w2 respectively. When w1 is the largest, it is determined that the side where w1 is located is the fundus edge region (for example, the dashed box B in the figure).

[0050] According to the foregoing, after the fundus edge region is determined, the fundus edge region can be cropped at a target distance (for example, 3*a d , a d representing the distance from the macular center in the target region to the optic disc center) away from the target region to obtain a fundus edge image containing the fundus edge region. In the following, the target region located in the middle part of the fundus image will be taken as an example to describe in detail how to obtain the fundus edge image containing the fundus edge region. Figure 4

[0051] Figure 4 is an exemplary schematic diagram illustrating the obtaining of the fundus edge image containing the fundus edge region according to an embodiment of the present application. As shown in FIG. a) of the figure, a fundus image of a super wide angle is input into a detection model (for example, a Faster R-CNN network model) 401 for detection, and a target region 402 containing the optic disc and the macula is obtained. Figure 4 Figure 4 ​​The target region is shown in FIG. (b). The target region includes a macular region (shown by the rectangular box C) and an optic disc region (shown by the rectangular box D). Further, by locating the target region, the position of the target region in the fundus image can be determined. More details about determining the position of the target region in the fundus image can be found in the above description of the method of locating the target region. Figure 3 According to the description, the present application will not be described here. For example, FIG. (c) shows the target region in the middle of the fundus image.

[0052] In this scenario, the distances of the target region from each boundary of the fundus image are calculated respectively, and the side with the maximum distance is determined as the fundus edge region. Specifically, the distances of the center of the target region from each boundary of the fundus image are calculated respectively, where the center of the target region is the center of the line connecting the center of the macular region and the center of the optic disc region, and can be determined by the coordinates of the macular and the optic disc. As shown in FIG. (b), the center of the target region is the center of the line connecting the center of the macular region and the center of the optic disc region. Thus, the distances of the center of the target region from the upper, lower, left and right boundaries of the fundus image can be calculated respectively. As an example, assume that the distances of the center of the target region from the upper, lower, left and right boundaries of the fundus image are a t , a b , a I and a r , respectively. When the distance a I is the maximum, the side with a I is determined as the fundus edge image.

[0053] Figure 4 Further shown in FIG. (d) is that the fundus edge region is cropped at a target distance away from the center of the target region (e.g. shown by the solid rectangular box). As previously described, the target distance can be the distance of the macular center to the optic disc center in the target region multiplied by a preset multiple. In the scenario of the line connecting the center of the macular region and the center of the optic disc region, the aforementioned target distance can be the distance of the line length (or distance) multiplied by a preset multiple, where the preset multiple can be, for example, 3. As an example, when the line distance is a d , the target distance can be, for example, 3*a d . Thus, the fundus edge region is cropped at 3*a d away from the target region to obtain a fundus edge image containing the fundus edge region, such as shown in FIG. (e) of Figure 4 .

[0054] Figure 5 are exemplary schematic diagrams showing intermediate fundus images of various scales according to embodiments of the present application. As shown in FIG. (a) of Figure 5 is the determined fundus edge image (i.e. the above-mentioned Figure 4The fundus edge image in FIG. 1(a) can be regarded as an intermediate fundus edge image in a scale type (as shown in FIG. 1(e)). Figure 5 FIG. 1(b) shows an intermediate fundus edge image in a scale type after the fundus edge image in FIG. 1(a) is cropped along the horizontal direction. Further, Figure 5 FIG. 1(c) shows an intermediate fundus edge image in a scale type after the fundus edge image in FIG. 1(b) is cropped along the horizontal direction. That is, Figure 5 FIG. 1(d) shows intermediate fundus edge images in three scales. It should be understood that, Figure 5 are merely exemplary and not limiting. For example, in some embodiments, the fundus edge image in FIG. 1(a) can also be cropped along the vertical direction to obtain an intermediate fundus edge image in a scale type. Figure 5 FIG. 1(d) shows intermediate fundus edge images in three scales. It should be understood that,

[0055] Figure 6 FIG. 1(e) is an exemplary schematic diagram showing identification of intermediate fundus images in three scales according to an embodiment of the present application. As shown in FIG. 1(e), Figure 6 FIG. 1(e) shows intermediate fundus images in three scales, which are intermediate fundus images in scale one, scale two and scale three, respectively. The scale one contains one intermediate fundus image, the scale two contains two intermediate fundus images, and the scale three contains four intermediate fundus images. In an implementation scenario, three identification models can be used to identify them respectively. For example, the three identification models are identification model 601-1, identification model 601-2 and identification model 601-3, and the identification model 601-1 to the identification model 601-3 identify the intermediate fundus images in the scale one, the scale two and the scale three respectively to obtain respective identification results in multiple scales. It can be understood that the respective identification results in multiple scales are determined based on the identification result of each identification model for each intermediate fundus image in each scale. When each scale contains one intermediate fundus image, the identification result of the intermediate fundus image is the identification result of the corresponding scale. When each scale contains multiple intermediate fundus images, the maximum value of the identification results of the multiple intermediate fundus images is taken as the identification result of the corresponding scale.

[0056] For example, in one example scenario, assuming that one of the intermediate fundus images in scale one corresponds to an identification result of P1, then the identification result corresponding to scale one is P1. Assuming that two of the intermediate fundus images in scale two correspond to identification results of {P2, P3} and P2 is greater than P3, then the identification result corresponding to scale two is P2. Similarly, assuming that four of the intermediate fundus images in scale three correspond to identification results of {P4, P5, P5, P6} and P4 is the maximum value, then the identification result corresponding to scale three is P4. As described above, according to the identification results under each scale, the identification result of the lesion on the fundus periphery can be determined, for example, when there are multiple identification results that are the same, the same identification result is taken as the final identification result; when each identification result is different, the average value thereof can be taken as the final identification result. Taking the foregoing identification results as an example, assuming that the identification result P1 of scale one and the identification result P2 of scale two are the same, then the identification result of the lesion on the fundus periphery is P1 or P2. Assuming that the identification result P1 of scale one, the identification result P2 of scale two, and the identification result P4 of scale three are all different, then the identification result of the lesion on the fundus periphery is the average value of P1, P2, and P4. Based on this, smaller lesions in the fundus image can be identified to improve the accuracy of the identification result.

[0057] Figure 7 is an example flow block diagram illustrating a method 700 for identifying a lesion on a fundus periphery according to an embodiment of the present application. As shown in Figure 7 As shown in FIG. 7, at step 702, an ultra-wide-angle fundus image to be subjected to lesion identification is obtained. In one embodiment, the foregoing ultra-wide-angle fundus image can be obtained by photographing the fundus via an ultra-wide-angle optical camera. Based on the obtained ultra-wide-angle fundus image, at step 704, a detection model is used to detect the fundus image to obtain a fundus edge image containing a fundus edge region. In one embodiment, the detection model can be, for example, a Faster R-CNN network model. By using the detection model, a target region (including a macular region and a disc region) in the fundus image can first be obtained, and then the target region in the fundus image can be located to determine the position of the target region. For example, it can be determined whether the target region is in the upper, lower, or middle part of the fundus image to obtain the fundus edge image containing the fundus edge region based on the specific position of the target region. For more details about determining the position of the target region and obtaining the fundus edge image containing the fundus edge region, reference can be made to the foregoing description of the fundus edge image containing the fundus edge region. Figures 3-4 The foregoing description is not repeated here.

[0058] Subsequently, at step 706, the fundus edge image is subjected to a cropping operation to obtain intermediate fundus edge images under multiple scales. In an implementation scenario, the fundus edge image can be cropped along the horizontal direction or the vertical direction to obtain intermediate fundus edge images under multiple scales (for example,Figure 5 The intermediate fundus edge image under three scales is exemplarily shown in FIG. 7B (the intermediate fundus edge image under three scales is exemplarily shown in FIG. 7B). Further, at step 708, the intermediate fundus edge images under multiple scales are respectively identified using the identification model to determine the identification result of identifying the lesions of the fundus periphery. In an embodiment, the identification model can include multiple identification models, and each identification model can be, for example, a ResNet-50 network model. The intermediate fundus edge images under each scale are identified by each identification model to obtain the identification result of each scale, and then the identification result of identifying the lesions of the fundus periphery is obtained. More details about obtaining the identification result of the lesions of the fundus periphery can be referred to the description of the above-mentioned Figure 6 .

[0059] It can be understood that the detection model and the identification model described above need to be trained when used. In the training process, the loss value of the model can be calculated by a loss function, and the weight is adjusted by the loss value to complete the training. In an embodiment, for the detection model, its loss function can be expressed as the following formula:

[0060]

[0061] wherein, L cls represents a cross-entropy loss function, which is used to calculate the loss value of the predicted class probability and the real class label p i ; L reg represents a Smooth L1 loss function, which is used to calculate the loss value of the predicted detection box position and the real detection box position label. In an embodiment, L cls represents that the cross-entropy loss function and the Smooth L1 loss function can be respectively expressed as the following formula:

[0062]

[0063]

[0064] In some embodiments, the loss function of the identification model can be a cross-entropy loss function, for example, as shown in the above-mentioned formula (2).

[0065] Figure 8 FIG. 8 is an exemplary structural block diagram of a device 800 for identifying the lesions of the fundus periphery according to an embodiment of the present application. It can be understood that the device implementing the scheme of the present application can be a single device (for example, a computing device) or a multifunctional device including various peripheral devices.

[0066] As Figure 8As shown in the figure, the device of the present application can include a central processing unit or central processing unit ("CPU") 811, which can be a general-purpose CPU, a special-purpose CPU, or other execution units for information processing and program running. Further, the device 800 can also include a mass storage 812 and a read-only memory ("ROM") 813, wherein the mass storage 812 can be configured to store various types of data, including various wide-angle fundus images, algorithm data, intermediate results, and various programs required for running the device 800. The ROM 813 can be configured to store data and instructions required for power-on self-test, initialization of various functional modules in the system, driver program for basic input / output of the system, and booting of the operating system of the device 800.

[0067] Optionally, the device 800 can also include other hardware platforms or components, such as the tensor processing unit ("TPU") 814, the graphics processing unit ("GPU") 815, the field programmable gate array ("FPGA") 816, and the machine learning unit ("MLU") 817 shown. It can be understood that although various hardware platforms or components are shown in the device 800, they are merely exemplary and not limiting, and a person skilled in the art can add or remove corresponding hardware according to actual needs. For example, the device 800 can only include a CPU, related storage devices, and interface devices to implement the method for identifying lesions around the fundus of the present application.

[0068] In some embodiments, in order to facilitate the transmission and interaction of data with external networks, the device 800 of the present application further includes a communication interface 818, so that the device 800 can be connected to a local area network / wireless local area network ("LAN / WLAN") 805 through the communication interface 818, and then can be connected to a local server 806 or connected to the Internet 807 through the LAN / WLAN. Alternatively or additionally, the device 800 of the present application can also be directly connected to the Internet or a cellular network based on wireless communication technology through the communication interface 818, such as based on the third generation ("3G"), fourth generation ("4G"), or fifth generation ("5G") wireless communication technology. In some application scenarios, the device 800 of the present application can also access the servers 808 and databases 809 of external networks as needed, in order to obtain various known algorithms, data, and modules, and can remotely store various data, such as various types of data or instructions for presenting, for example, fundus images, fundus edge images, intermediate fundus edge images under multiple scales, and identification results, etc.

[0069] The peripherals of the device 800 can include a display device 802, an input device 803, and a data transmission interface 804. In an embodiment, the display device 802 can include, for example, one or more speakers and / or one or more visual displays configured for audio and / or image / video display of the identification of the lesions of the fundus periphery according to the present application. The input device 803 can include, for example, a keyboard, a mouse, a microphone, a gesture capture camera, and other input buttons or controls configured for receiving input of audio data and / or user instructions. The data transmission interface 804 can include, for example, a serial interface, a parallel interface, or a universal serial bus interface (“USB”), a small computer system interface (“SCSI”), Serial ATA, FireWire, PCI Express, and a high-definition multimedia interface (“HDMI”), and the like, configured for data transmission and interaction with other devices or systems. According to the scheme of the present application, the data transmission interface 804 can receive the ultra-wide-angle photographed fundus image and transmit the fundus image or various other types of data or results to the device 800.

[0070] The above CPU 811, mass storage 812, ROM 813, TPU 814, GPU 815, FPGA 816, MLU 817, and communication interface 818 of the device 800 of the present application can be connected to each other through a bus 819, and data interaction is achieved with the peripherals through the bus. In an embodiment, through the bus 819, the CPU 811 can control other hardware components and their peripherals in the device 800.

[0071] The above description in combination with the drawings Figure 8 A device for identifying lesions of the fundus periphery according to the present application is described. It should be understood that the device structure or architecture herein is only exemplary, and the implementation and implementation entity of the present application are not limited thereto, but can be changed without departing from the spirit of the present application.

[0072] According to the above description in combination with the drawings, those skilled in the art can also understand that the embodiments of the present application can also be implemented by software programs. Therefore, the present application also provides a computer program product. The computer program product can be used to implement the device for identifying lesions of the fundus periphery according to the present application in combination with the above description. Figure 7 The method for identifying lesions of the fundus periphery described above.

[0073] It should be noted that, although the operations of the methods of the present application are described in a particular order in the drawings, this is not meant to imply that the operations must be performed in that particular order, or that all illustrated operations must be performed to achieve the desired result. The steps depicted in the flowcharts can be changed in order of execution. Additionally or alternatively, certain steps can be omitted, combined into a single step, and / or broken into multiple steps.

[0074] It should be understood that the terms "first", "second", "third", and "fourth" and the like in the description and in the claims, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. The terms "including", "containing", "comprising", "having" and the like are used herein to mean including but not limited to.

[0075] It is also to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used in this specification and the claims, the singular forms "a", "an" and "the" include plural referents unless the content clearly dictates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0076] While the embodiments of the present application have been illustrated and described, it is understood that the application is not limited to the details of construction or the arrangements of components as set forth above and / or in the accompanying drawings. Any skilled person in the art of the present application can make any modifications and changes without departing from the spirit and scope of the present application, and the patent protection scope of the present application should be subject to the scope defined by the appended claims.

Claims

1. An apparatus for identifying a lesion on the periphery of an eye fundus, comprising: a processor; and a memory storing program instructions for identifying a lesion on the periphery of an eye fundus, which when executed by the processor, cause the apparatus to perform the following operations: obtaining an ultra-wide angle eye fundus image to be identified for a lesion; detecting the eye fundus image using a detection model to obtain an eye fundus edge image containing an eye fundus edge region; performing a cropping operation on the eye fundus edge image to obtain intermediate eye fundus edge images at multiple scales; and identifying the intermediate eye fundus edge images at the multiple scales respectively using an identification model to determine an identification result of identifying a lesion on the periphery of an eye fundus; wherein when the program instructions are further executed by the processor, the apparatus is caused to perform the following operations to obtain an eye fundus edge image containing an eye fundus edge region: detecting a target region in the eye fundus image using a detection model, wherein the target region includes a macula and a region where a disc is located; positioning the target region in the eye fundus image to determine a position of the target region by determining that the target region is located at an upper portion, a lower portion, or a middle portion of the eye fundus image; and obtaining an eye fundus edge image containing an eye fundus edge region based on the position of the target region; wherein the apparatus performs the following operations to obtain an eye fundus edge image containing an eye fundus edge region: determining an eye fundus edge region based on that the target region is located at the upper portion, the lower portion, or the middle portion of the eye fundus image; and obtaining an eye fundus edge image containing the eye fundus edge region according to the eye fundus edge region; and performing the following operations to determine an eye fundus edge region: in response to that the target region is located at the upper portion or the lower portion of the eye fundus image, comparing distances of the target region to respective horizontal boundaries of the eye fundus image and determining a side farthest from a corresponding horizontal boundary as the eye fundus edge region; or in response to that the target region is located at the middle portion of the eye fundus image, comparing distances of the target region to respective boundaries of the eye fundus image and determining a side farthest from a corresponding boundary as the eye fundus edge region.

2. The apparatus of claim 1, wherein when the program instructions are further executed by the processor, the apparatus is caused to perform the following operations to determine the position of the target region: calculating a distance of the target region to a reference boundary in the eye fundus image, taking the horizontal boundary as the reference boundary; comparing the distance of the target region to the reference boundary with a positioning threshold; and determining the position of the target region by determining that the target region is located at the upper portion, the lower portion, or the middle portion of the eye fundus image according to a comparison result.

3. The apparatus of claim 2, wherein the positioning threshold includes a first positioning threshold and a second positioning threshold, and when the program instructions are further executed by the processor, the apparatus is caused to perform the following operations to determine that the target region is located at the upper portion, the lower portion, or the middle portion of the eye fundus image: ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ determining that the target region is located at the top or bottom of the fundus image in response to the distance of the target region from the reference boundary being less than the first positioning threshold or the distance of the target region from the reference boundary being greater than the second positioning threshold; or determining that the target region is located at the middle of the fundus image in response to the distance of the target region from the reference boundary being between the first positioning threshold and the second positioning threshold.

4. The apparatus of claim 1, wherein the program instructions, when further executed by the processor, cause the apparatus to obtain a fundus margin image containing the fundus margin region by: cropping the fundus margin region at a target distance away from the target region to obtain a fundus margin image containing the fundus margin region.

5. The apparatus of claim 1, wherein the program instructions, when further executed by the processor, cause the apparatus to determine an identification result of identifying a lesion on a fundus periphery by: identifying the intermediate fundus margin images at the plurality of scales respectively using an identification model to obtain identification results corresponding to the plurality of scales; and determining the identification result of identifying the lesion on the fundus periphery based on the identification results corresponding to the plurality of scales.

6. The apparatus of claim 5, wherein the program instructions, when further executed by the processor, cause the apparatus to determine the identification result of identifying the lesion on the fundus periphery by: determining, in response to a plurality of the identification results in the identification results corresponding to the plurality of scales being identical, an identical identification result corresponding to the plurality of the identification results as the identification result of identifying the lesion on the fundus periphery; or determining, in response to the identification results corresponding to the plurality of scales each being different, an average of the identification results corresponding to the plurality of scales as the identification result of identifying the lesion on the fundus periphery.

7. A method for identifying a lesion on a fundus periphery, comprising: obtaining a fundus image of an ultra-wide angle to be subjected to lesion identification; detecting the fundus image using a detection model to obtain a fundus margin image containing a fundus margin region; cropping the fundus margin image to obtain intermediate fundus margin images at a plurality of scales; and identifying the intermediate fundus margin images at the plurality of scales respectively using an identification model to determine an identification result of identifying a lesion on a fundus periphery; wherein the fundus margin image containing the fundus margin region is obtained by: detecting a target region in the fundus image using a detection model, wherein the target region includes a region where a macula and an optic disc are located; positioning the target region in the fundus image to determine a position of the target region by determining that the target region is located at a top, a bottom, or a middle of the fundus image; and obtaining the fundus margin image containing the fundus margin region based on the position of the target region; wherein the fundus margin image containing the fundus margin region is obtained based on the position of the target region by: determining the fundus margin region based on the target region being located at the top, the bottom, or the middle of the fundus image; and ​ obtaining a fundus margin image containing the fundus margin region according to the fundus margin region; and determining the fundus margin region based on the target region being located at the top, bottom or middle of the fundus image comprises: in response to the target region being located at the top or bottom of the fundus image, comparing distances of the target region to each horizontal boundary of the fundus image and determining the side farthest to the corresponding horizontal boundary as the fundus margin region; or in response to the target region being located at the middle of the fundus image, comparing distances of the target region to each boundary of the fundus image and determining the side farthest to the corresponding boundary as the fundus margin region. 8.A computer readable storage medium having stored thereon computer readable instructions for identifying a lesion of a fundus periphery, the computer readable instructions, when executed by one or more processors, implement the method according to claim 7.

Images