Fully automatic portable selfie fundus camera

Abstract

The present invention provides a fully automatic portable Selfie fundus camera, comprising a housing; a lens barrel located inside the housing, the lens barrel being provided with a lighting assembly, a focusing assembly, an eyepiece objective lens, an optical lens group, and an imaging detector; a motion assembly for driving the lens barrel to move inside the housing along the front-to-back axial direction, the left-to-right horizontal direction, and the up-and-down vertical direction of the lens barrel; a surface mount assembly sealingly connected to the front portion of the housing, the surface mount assembly comprising a surface mount body and a window through hole formed on the surface mount body for accommodating the eyes of a subject when the surface mount body is in contact with the eyes of the subject, wherein the moving shooting range of the eyepiece objective lens driven by the motion assembly does not exceed the range of the window through hole.

Description

Technical Field

[0001] The present invention relates to the field of ophthalmic instruments, and in particular to a fully automatic portable Selfie fundus camera. Background Art

[0002] The retina is the only tissue in the human body where capillaries and nerves can be directly observed. By observing the retina, it is possible to detect not only eye health problems but also systemic diseases such as diabetic complications and hypertension. Fundus cameras are specialized devices used to photograph the retina.

[0003] Existing fundus cameras can automatically capture fundus images. This automated process primarily involves automatically aligning the primary lens with the pupil, automatically adjusting the axial distance (working distance) between the lens and pupil, and automatically adjusting the focal length. The camera consists of a main camera, an auxiliary camera, and numerous auxiliary optical components. The main camera is mounted on a platform that can move in the X, Y, and Z directions and is used to capture the fundus. The auxiliary camera, mounted near the main camera, captures the face and external parts of the eye, primarily for searching the eye and automatically aligning the pupil. The auxiliary optical components are used for focusing, adjusting the working distance, and more.

[0004] Existing fundus cameras require complex and expensive hardware modules to solve the problems of aligning the lens with the pupil, fixing the axial distance between the lens and the pupil, and focusing. They are also complicated to use and require professional assistance in the shooting process, which hinders the popularization of fundus cameras. Summary of the Invention

[0005] In view of this, the present invention provides a fully automatic portable Selfie fundus camera, comprising:

[0006] case;

[0007] A lens barrel is located inside the housing and is provided with an illumination assembly, a focusing assembly, an eyepiece objective lens, an optical lens group, and an imaging detector;

[0008] A motion assembly, configured to drive the lens barrel to move inside the housing along the front-rear axis, the left-right horizontal direction, and the up-down vertical direction of the lens barrel;

[0009] A face mount assembly is sealingly connected to the front portion of the housing. The face mount assembly includes a face mount body and a window through-hole formed on the face mount body for accommodating the eyes of the subject when the face mount body is in contact with the eyes of the subject. The moving shooting range of the eyepiece objective lens driven by the motion assembly does not exceed the range of the window through-hole.

[0010] Optionally, the fundus camera further includes:

[0011] At least one positioning shooting target is provided on a side of the surface-mounted body facing the lens barrel. The positioning shooting target is provided adjacent to the window through-hole and is used to align the lens barrel with the shooting image when the fundus camera is started.

[0012] Optionally, a raised portion protruding toward the center of the window through hole is provided in the middle of the upper edge of the window through hole; and one positioning and shooting target is provided, which is located on a surface of the raised portion facing the lens barrel.

[0013] Optionally, the positioning shooting target corresponds to the center of the eyebrows of the person being photographed whose eyes are in contact with the face-mounted body.

[0014] Optionally, a protrusion protruding toward the window through hole is provided on the upper edge of the window through hole, for aligning the lens barrel to capture an image when the fundus camera is started.

[0015] Optionally, a positioning and shooting target is provided on a surface of the raised portion facing the lens barrel.

[0016] Optionally, a surface of the face-sticking body facing away from the lens barrel is constructed into a shape that fits the facial contour around the eyes of the subject.

[0017] Optionally, the motion component includes:

[0018] a first track assembly arranged along the left-right horizontal direction;

[0019] a base movably arranged on the first rail assembly along the first rail assembly, the lens barrel being fixedly arranged on the base;

[0020] a second track assembly arranged along the vertical direction, the lens barrel and the base being movably arranged on the second track assembly;

[0021] A third rail assembly is arranged along the front-rear axial direction, and the lens barrel and the base are movably arranged on the third rail assembly.

[0022] Optionally, the fundus camera further includes a positioning component disposed inside the housing and configured to detect a moving position of the lens barrel within the housing.

[0023] Optionally, the positioning component includes:

[0024] A first positioning member and a second positioning member are provided on the left and right horizontal sides of the base to detect the moving position of the lens barrel in the left and right horizontal directions;

[0025] a third positioning member, provided on the base, for detecting the moving position of the lens barrel in the up and down vertical directions;

[0026] The first positioning member, the second positioning member and the third positioning member are all located within a movable covering range of the lens barrel on the base.

[0027] According to the fundus camera provided by the present invention, the lighting component, focusing component, eyepiece objective lens, optical lens group and imaging detector used for imaging are integrated in a lens barrel to achieve miniaturization of the optical path structure, reduce the volume of the fundus camera, and improve portability; the surface-mounted component of the fundus camera is provided with a window through-hole for accommodating the eyes of the subject, and the user can wear the fundus camera by himself, place the eye at the window through-hole position, and the motion component drives the lens barrel to search for the pupil within the window through-hole range and adjust the working distance to capture the fundus image. This solution reduces the complexity and difficulty of use of the fundus camera hardware, allowing users to independently capture fundus images and promoting the popularization of fundus cameras. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0029] Figure 1 is a structural diagram of a fundus camera in an embodiment of the present invention;

[0030] Figure 2 is a schematic diagram of a surface mount assembly of a fundus camera in an embodiment of the present invention;

[0031] Figure 3 is a schematic diagram of the lens and positioning components;

[0032] Figure 4 This is a flow chart of a fully automatic fundus image capturing method according to an embodiment of the present invention;

[0033] Figure 5 Label the diagram for the pupil;

[0034] Figure 6 Flowchart of a preferred method for fully automatic fundus image shooting in an embodiment of the present invention;

[0035] Figure 7 Schematic diagram of a pupil larger than the illumination beam;

[0036] Figure 8 Schematic diagram of a pupil smaller than the illumination beam;

[0037] Figure 9 Schematic diagram of taking fundus images when the pupil is smaller than the illumination beam;

[0038] Figure 10 Imaging of the illumination beam reflected by the cornea;

[0039] Figure 11 Schematic diagram of the distance between the lens barrel and the eyeball;

[0040] Figure 12 Mark the schematic diagram for the light spot;

[0041] Figure 13 To achieve the imaging of the illumination beam reflected by the cornea at the working distance;

[0042] Figure 14 Label the diagram for the video disc;

[0043] Figure 15 A schematic diagram of moving the lens position according to the light spot when capturing fundus images;

[0044] Figure 16 Schematic diagram of two fundus images with unusable areas;

[0045] Figure 17 Schematic diagram of the synthesis method of fundus images;

[0046] Figure 18 The present invention is a structural diagram of a lighting lamp;

[0047] Figure 19 Schematic diagram of imaging of reflected light when detecting camera status;

[0048] Figure 20 Schematic diagram of imaging the raised part of the face-mounted component when detecting the camera status;

[0049] Figure 21 A schematic diagram of imaging the raised portion of a surface patch component with a target when detecting the camera status;

[0050] Figure 22 The image of the area between the eyes is collected to detect the user's usage status. DETAILED DESCRIPTION

[0051] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0052] In the description of the present invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings and are intended solely to facilitate and simplify the description of the present invention. They are not intended to indicate or imply that the devices or components referred to must have, be constructed, or operate in a specific orientation, and therefore should not be construed as limitations on the present invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0053] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "installed," "connected," and "connected" should be understood in a broad sense. For example, they can refer to fixed connections, detachable connections, or integral connections; mechanical connections or electrical connections; direct connections or indirect connections through an intermediate medium; internal connections between two components; wireless connections or wired connections. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0054] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0055] Figure 1 A fully automatic portable Selfie fundus camera is shown. The camera includes a surface mount assembly 01, a motion assembly, a positioning assembly 03, and a lens barrel 1. The lens barrel 1 is internally equipped with an illumination assembly, a focusing assembly, a lens (an objective lens), an optical lens assembly, and an imaging detector 10. The internal structure of the lens barrel 1 can be found in Chinese patent document CN111134616A. The actual product also includes a housing, with the motion assembly and lens barrel 1 located inside the housing. The surface mount assembly 01 is sealingly connected to the front portion of the housing. The surface mount assembly includes a surface mount body and a window through-hole formed in the surface mount body for accommodating the subject's eyes when in contact with the face of the subject. The surface mount assembly 01 serves as the component that contacts the subject's eyes, and the lens barrel 1 collects the subject's fundus retinal images through the through-hole of the surface mount assembly 01.

[0056] The side of the face patch body facing away from the lens barrel 1 is constructed to fit the facial contours around the eyes of the subject. Specifically, the face patch component 01 is formed inwardly into a concave shape to fit the arc of the human head, and the size of its through-holes is at least large enough to accommodate both eyes of the subject when the face patch component is in contact with the eyes. The face patch component 01 facing inward (inside the housing, lens barrel) has at least one specific position for detecting various camera functions. In a specific embodiment, combined with Figure 1 and Figure 2 As shown, Figure 2The inward-facing side of the faceplate assembly 01 is shown. A raised portion 012 is located at the upper edge of the middle portion of the through-hole 011. The lens of the lens barrel 1 can be aligned with this portion to capture the image. A more preferred approach is to place a pattern or simple graphic on this raised portion 012 as a target. This specific location has various uses, including checking whether the camera's lighting and focusing components are functioning properly, and verifying that the subject's eyes are properly aligned with the faceplate assembly 01. These details will be discussed in detail below.

[0057] The motion component is used to control the movement of the lens barrel 1 in three-dimensional space. Figure 1 Taking the coordinate system in the figure as an example, it can move on the three axes X, Y, and Z in the figure. It should be noted that when the lens barrel 1 moves to the extreme position in the Z direction, the end will not extend out of the surface mounting component 01. As a specific embodiment, the motion component includes three track assemblies, the first group of tracks 021 is used to control the movement of the lens barrel 1 on the X axis, the second group of tracks 022 is used to control the movement of the lens barrel 1 on the Y axis, and the third group of tracks not shown in the figure is used to control the movement of the lens barrel 1 on the Z axis. Specifically, the lens barrel 1 together with the second group of tracks 022 are set on a platform (base), the first group of tracks 021 can drive the base to move as a whole, and the third group of tracks can drive the base and the first group of tracks 021 to move, so that the whole is close to or away from the surface mounting component 01.

[0058] The positioning component 03 is used to detect the movement of the lens barrel 1. Specifically, the positioning component 03 can be an electromagnetic sensor, which senses the movement of the lens barrel 1 to the position of the positioning component 03 according to the electromagnetic induction signal. Figure 3 As shown, in this embodiment, three positioning components 03 are provided, wherein two positioning components are provided on both sides of the movable base to detect the movement of the lens barrel 1 on the X-axis, and the third positioning component is provided on the base to detect the movement of the lens barrel 1 on the Y-axis, that is, the positioning component 03 is used to detect the movement of the lens barrel 1 in the XY plane.

[0059] According to the fundus camera provided by the present invention, the lighting component, focusing component, eyepiece objective lens, optical lens group and imaging detector used for imaging are integrated in a lens barrel to achieve miniaturization of the optical path structure, reduce the volume of the fundus camera, and improve portability; the surface-mounted component of the fundus camera is provided with a window through-hole for accommodating the eyes of the subject, and the user can wear the fundus camera by himself, place the eye at the window through-hole position, and the motion component drives the lens barrel to search for the pupil within the window through-hole range and adjust the working distance to capture the fundus image. This solution reduces the complexity and difficulty of use of the fundus camera hardware, allowing users to independently capture fundus images and promoting the popularization of fundus cameras.

[0060] The embodiment of the present invention provides a method for fully automatic fundus image shooting, which can be executed by the fundus camera itself or by an electronic device such as a computer or server (as a control method). Figure 4 As shown, the method includes the following steps:

[0061] S300, move the fundus camera lens to align with the pupil.

[0062] S400: Control the lens to approach the eyeball and collect an image, which is an imaging of the illumination light beam reflected by the cornea.

[0063] S500: Determine the working distance using the above image.

[0064] S600: Adjust the focus and collect fundus images, and use the fundus images to determine the shooting focus.

[0065] S700, fundus images were captured using the shooting focal length at the working distance.

[0066] In a preferred embodiment, before the above step S100, a step of detecting the camera status and the user's usage status may be performed, such as Figure 6 As shown, the method may also include:

[0067] S100: Check whether the motion, lighting, and focusing components of the fundus camera are functioning properly. This step is optional and can be performed when the fundus camera is powered on. If any component is detected as abnormal, subsequent capture operations will be terminated and a corresponding abnormality notification will be displayed.

[0068] S200: Detect whether the head of the user fits the face-mounted assembly of the fundus camera. This step is optional. If it is detected that the head of the user does not fit the face-mounted assembly of the fundus camera, a voice module may be used to prompt the user to correctly wear the fundus camera.

[0069] With respect to step S100 above, an embodiment of the present invention provides a fundus camera detection method. This method can be performed by the fundus camera itself as a self-test method, or it can be performed by an electronic device such as a computer or server. As a product detection method, the method includes the following steps:

[0070] S1, control the motion component to adjust the position of the lens, and detect whether the lens can move to the position of each positioning component. This method is suitable for execution when the fundus camera is just started. First, the lens (according to the above embodiment, the lens and the lens barrel are integrated, and the moving lens barrel is the moving lens) is moved to the initial position. Then, combined with Figure 3As shown, the motion assembly adjusts the position of the lens and detects whether the lens can move to the positions of the three positioning assemblies. If it can move to these positions, the motion assembly is considered to be functioning properly and step S2 can be executed. Otherwise, step S6 can be executed. Step S1 can be called the motion assembly XY axis motion detection step.

[0071] S2, controls the motion component to move the lens to the set position, turns on the lighting component and controls the focusing component to adjust to the first focal length, and captures the first image. The purpose of this step is to detect whether the functions of the focusing component and the lighting component are normal. In theory, there is no need to limit the lens to a certain position, so there are multiple options for the set position described in this step. However, in an actual working environment, the external environment is uncertain. For example, it may be a relatively bright environment. If the external environment is relatively bright when the first image is captured in this step, the image content may be disturbed. In order to be applicable to the actual working environment, this step moves the lens to a specific part of the surface patch component (such as the above-mentioned raised part), captures as little external environment as possible, and the proportion of the surface patch component in the image is greater than the proportion of the external environment. Of course, it is also possible to modify the shape of the surface patch component and its through hole so that the image captured in this step does not contain the external environment at all.

[0072] By setting the appropriate focal length, the lighting component can be imaged. Figure 18 The structure of the lighting lamp in a lens barrel is shown. There are 4 lamp beads on a ring structure. Turn on these 4 lamp beads and use the first focal length to perform imaging. It is expected to obtain a Figure 19 The image shown.

[0073] In a preferred embodiment, to prevent the background from affecting the image of the lighting assembly in the captured image, the first focal length is set to image the lighting assembly but not the surface mount assembly. This ensures that only the lighting assembly is visible in the first image, without objects such as raised portions of the surface mount assembly, thereby improving the accuracy of image recognition in subsequent steps.

[0074] S3, judging whether the focusing assembly and the lighting assembly are normal according to the image features of the lighting assembly in the first image. If the focusing assembly functions normally, the set focal length should be able to obtain Figure 19 The image shown has distinct features, depending on the actual shape of the lighting assembly. For example, in this embodiment, the first image should have four distinct, distinct dots, representing the image of the four lamp beads. If the focal length is adjusted to a different focal length than the first one, the dots in the image will appear larger and blurry, or smaller. If the lighting assembly is not powered on, no shape will appear in the image.

[0075] Using a machine vision algorithm or a neural network algorithm, the first image can be identified to determine whether the image contains features that meet expectations. If the focusing assembly and lighting assembly are determined to be normal, step S4 is executed; otherwise, step S6 is executed. Steps S2-S3 can be referred to as the focusing assembly and lighting assembly detection steps.

[0076] S4, controls the motion component to adjust the lens to the set depth position, controls the focusing component to adjust to the second focal length, and shoots to obtain the second image. In this step, it is necessary to image a known object. As a preferred embodiment, the raised portion of the surface-mounted component is used as the known object. Specifically, first align the lens with the raised portion of the surface-mounted component on the XY plane. In this implementation, step S2 has already aligned this portion, and no further adjustment is required in this step. In other embodiments, if this portion is not aligned in step S2, adjustments are made in this step. This step requires adjusting the depth, that is, adjusting the position of the lens on the Z axis, which can be understood as adjusting the shooting distance to the known object, and then setting the focal length.

[0077] In order to image the external object, the focal length at this time is different from the focal length used in step S2, and the focal length in this step should be suitable for the current lens position (depth position), and it is expected to obtain a focal length such as Figure 20 The image shown.

[0078] In a preferred embodiment, to prevent the illumination component's image from affecting the image of the object being photographed, the second focal length is set to image the surface mount component but not the illumination component. This ensures that the second image only contains the object being photographed, such as the raised portion of the surface mount component, without the illumination component, improving the accuracy of image recognition in subsequent steps.

[0079] S5, judging whether the imaging function is normal based on the image features of the object in the second image. The second image is an image captured when the motion component is moving normally in the XY axis, the lighting component, and the focusing component are all normal. The purpose of this step is to detect whether the motion component is moving normally in the Z axis. If the motion component can adjust the lens to the set depth position in step S4, the second image should be able to show a clear object, such as Figure 20 The raised portion of the face patch assembly is shown.

[0080] Using either a machine vision algorithm or a neural network algorithm, the second image can be identified to determine whether the image contains features that meet expectations. If the motion component's Z-axis motion is determined to be normal, the test ends and the functioning of the fundus camera's main components is confirmed to be normal. Otherwise, step S6 is executed. Steps S4-S5 can be referred to as the motion component Z-axis motion detection step.

[0081] S6: Determine if the fundus camera is abnormal. Based on the component experiencing the abnormality, provide the user with a detailed indication of the fault location. The fundus camera may include a voice module or information display module to broadcast or display the corresponding fault information to the user.

[0082] According to the fundus camera detection method provided by an embodiment of the present invention, a positioning component is used to verify whether the motion component can properly adjust the lens position. After confirming that the motion component is normal, the focal length is adjusted to allow the illumination component to form an image. By evaluating the captured image, it is possible to determine whether the focusing and illumination components are functioning properly. Finally, the depth of the lens is adjusted by the motion component, and the focal length is adjusted to form an image of the object being photographed. The object features in the image are then evaluated to verify whether the motion component can properly adjust the depth of the lens. This automatically determines whether the various key components of the fundus camera are functioning properly. This solution allows for self-service inspection of the device's operating status in a remote, unattended environment, thereby improving the convenience of taking fundus photographs and promoting the widespread use of fundus cameras.

[0083] In a preferred embodiment, a target is provided on the raised portion of the surface patch component, that is, the object to be photographed is a target on a predetermined portion of the surface patch component. The specific content of the target is not limited, and one or more clearly defined patterns or shapes are all feasible. The second image obtained is as follows: Figure 21 As shown, a circular target 81 is included, and step S5 specifically includes:

[0084] S51, identifying whether there is a clear image of the target in the second image;

[0085] S52: When there is a clear image of the target in the second image, it is determined that the imaging function is normal.

[0086] The results of target recognition using machine vision algorithms or neural network algorithms are more accurate. If the target wheel does not exist in the image or the outline is unclear, it will be more easily recognized, thereby further improving the accuracy of camera function judgment.

[0087] With respect to step S200 above, an embodiment of the present invention provides a method for detecting the usage status of a fundus camera, for detecting whether the user is correctly wearing the fundus camera described in the above embodiment. This method can be performed by the fundus camera itself as a self-test method, or it can be performed by an electronic device such as a computer or server. This method is suitable for execution after determining that the important components of the camera are functioning properly according to the above detection method. The method includes the following steps:

[0088] S1, obtain the first image captured by the lens through the window of the surface mount component. In this solution, the lens is Figure 2The through hole 011 shown captures images of the external environment, and the surface patch component should be prevented from blocking the lens (the surface patch component is not within the imaging range). When the person being photographed correctly wears the fundus camera, the eyes fit into the surface patch component 01, and the window (through hole 011) contains the human eyes and the surrounding skin, and the lens captures the corresponding first image. In this step, the lighting component needs to be kept in a closed state, that is, no light beam is irradiated outward through the lens. In this solution, since the clarity of the captured image is not required to be high, the focal length used to capture the image can be a fixed value, and the imaging plane can be roughly set on the surface of the human body. Of course, you can also turn on the lighting component first, perform automatic focus, and then turn off the lighting component after setting the imaging plane more accurately on the surface of the human body.

[0089] S2: Determine whether the brightness of the first image meets the set standard. If the subject's eyes fit snugly within the faceplate assembly 01 without any significant gaps, the captured first image should be quite dark. The brightness of the first image is determined. If it meets the set standard, proceed to step S3; otherwise, proceed to step S6.

[0090] There are many ways to determine whether the brightness of an image meets the set standards. For example, the brightness value can be calculated based on the pixel values ​​of the image and then compared with the threshold. A neural network algorithm can also be used to pre-train the neural network using images of different brightness to enable it to have the ability to classify or regress image brightness. Here, the neural network is used to recognize the first image and output the recognition result about the brightness.

[0091] In a preferred embodiment, the first image is converted into a grayscale image, and then the brightness of the grayscale image is identified.

[0092] S3, turn on the lighting component, and obtain the second image captured by the lens through the window of the face-mounted component. At this time, the status of the lens and the subject does not change, only the lighting light source is turned on, and the lighting beam is illuminated outward through the lens. At this time, the lighting beam hits the eyes or skin of the subject and is reflected. In a preferred embodiment, the position of the lens is set to align with the center of the window of the face-mounted component, and the light source used is infrared light. If the human head is in contact with the face-mounted component, the lens is aimed at the area between the eyes, and the image can be captured. Figure 22 The image shown.

[0093] S4, determining whether the head of the person is in contact with the face-mounted assembly based on the second image. If the head of the person being photographed is in contact with the face-mounted assembly, since the human skin will reflect the illumination beam, Figure 22 An obvious light spot will appear in the image shown, and human skin features will appear around the light spot. By judging whether the image has the characteristics of brighter center and darker edges, it can be determined whether the human head fits the face patch component.

[0094] Assuming that no object is attached to the surface mount assembly during steps S1-S2, the camera is placed in a dark room, or the surface mount assembly is obscured by another object, the brightness of the first image will still be determined to meet the set standard, requiring further determination in steps S3-S4. If no object is attached to the surface mount assembly, no light spot will appear in the captured second image. If another object obscures the surface mount assembly, a light spot will appear in the second image. However, due to differences in material and surface shape, the reflection of the illumination beam will differ from that of a human body. Therefore, the characteristics of the light spot can be used to determine whether it is a human body.

[0095] In other optional embodiments, the lens may be aimed at other positions, such as the eyeball, when capturing the first and second images. In step S4, features of the eyeball may be identified in the image to determine whether the image is human.

[0096] In a preferred embodiment, step S4 can first determine whether the brightness of the second image meets a set standard. Similar to identifying the brightness of the first image, the second image can be converted to a grayscale image and then the brightness value calculated, or recognition can be performed using a neural network. If a gap exists between the surface patch assembly and the human body, causing light leakage, the brightness of the second image will be different from that when illuminated only by the camera's own light source due to the influence of ambient light. After eliminating light leakage, it is then determined whether the features in the second image match those of human skin.

[0097] When it is determined that the human head fits the surface patch assembly, step S5 is executed; otherwise, step S6 is executed.

[0098] S5, start capturing the fundus image. Specifically, it is necessary to automatically find the pupil, adjust the working distance, adjust the focus to set the imaging plane on the fundus, and finally capture the fundus image.

[0099] S6, prompting the user to wear the fundus camera correctly. For example, a voice module can be set in the fundus camera to prompt the user how to wear the fundus camera correctly, etc., and then the process can return to step S1 and re-judge.

[0100] According to the fundus camera usage status detection method provided by an embodiment of the present invention, an image is collected when the lighting component is turned off. The brightness of the image can be used to preliminarily judge whether the surface patch component is well covered by an object. Then, an image is collected when the lighting component is turned on. The image features can further judge whether the covering object is a human body. In this way, it is automatically determined whether the person being photographed is wearing the fundus camera correctly and whether the fundus camera is being used in a suitable environment. This solution can automatically trigger the fundus camera to take fundus photos. There is no need for manual intervention to trigger the shooting, and no need for professional personnel to operate it. This can improve the convenience of taking fundus photos and promote the popularization of fundus cameras.

[0101] When the camera starts shooting, the pupil and the eyepiece lens in actual application scenarios will not be completely aligned. At this time, the camera needs to determine the position of the lens relative to the pupil by the image of the pupil on the sensor, and then move the lens to the front of the pupil before shooting. With respect to the above step S300, an embodiment of the present invention provides a method for automatically aligning the lens of a fundus camera. This method can be executed by the fundus camera itself or by an electronic device such as a computer or server (as a control method). The method includes the following steps:

[0102] S1: Identify the image captured by the fundus camera lens to determine whether the pupil is present. Specifically, when the user wears the fundus camera, the system continuously (e.g., frame by frame) captures images of the pupil. If the pupil can be identified in the image, it indicates that the pupil is within the imaging range. In this case, fine-tune the lens to fully align with the pupil and then take the photo. If the pupil cannot be identified in the image, it indicates that the position of the lens and the pupil is significantly deviated. This may be due to an improper initial lens position, an improper wearing method, etc.

[0103] There are various ways to identify pupil images in images. For example, machine vision algorithms can be used to detect pupil outline and position based on graphical features in the image. However, because fundus cameras use infrared light for illumination before the final capture, pupil imaging is not very clear. Furthermore, reflections from the cornea also pose significant challenges to pupil detection. Computer vision algorithms are prone to misjudgment in this situation. Therefore, in a preferred embodiment, a deep learning algorithm is used to address this issue.

[0104] First, a large number of pupil images are collected. These images are taken from different individuals, at different directions and distances from the fundus camera's eyepiece, and at different times. The pupils in each image are then annotated to generate training data for the neural network. This annotated data is used to train a neural network model (such as the YOLO network). After training, the neural network model's recognition results include a detection box that represents the location and size of the pupil in the image.

[0105] like Figure 5 As shown, in a specific embodiment, a square frame 51 is used to mark the pupil in the training data, and the recognition result of the trained neural network model will also be a square detection frame. In other embodiments, a circular frame can also be used for marking, or other similar marking methods are feasible.

[0106] Regardless of the pupil detection method used, in this step it is only necessary to identify whether there is a pupil in the image. If there is no pupil in the image, step S2 is executed; otherwise, step S3 is executed.

[0107] S2 controls the fundus camera lens to move near the current position to search for the pupil. The motion assembly moves the lens barrel, for example, in a spiral trajectory, gradually spreading from the current position to the surrounding area. It should be noted that this embodiment only involves movement within the aforementioned XY plane and does not currently discuss Z-axis movement. Z-axis movement is related to the optimal working distance of the fundus camera and will be discussed in detail in subsequent embodiments.

[0108] If the pupil cannot be found after moving to the extreme position, the user is prompted to adjust the wearing state. If the pupil is found, the system further determines whether the user's eye is far away from the lens, exceeding the range of movement of the motion component. For example, it determines whether the movement distance of the lens exceeds the movement threshold. If the movement distance exceeds the movement threshold, the user is prompted to slightly move their head within the face mount component to adjust to the movement range of the lens. The search continues, and if the movement distance does not exceed the movement threshold, step S3 is executed.

[0109] S3: Determine whether the pupil in the image meets the set conditions. Specifically, multiple set conditions can be set, such as conditions on size, conditions on shape, etc.

[0110] In an optional embodiment, the set conditions include a size threshold, and the system determines whether the pupil size in the image is greater than the size threshold. If the pupil size in the image is greater than the size threshold, it is determined that the pupil meets the set conditions. Otherwise, the user is prompted to close their eyes and rest for a period of time to allow the pupil to dilate before resuming the image. Because fundus images are generally taken sequentially, the pupil will constrict after the first eye is photographed. Therefore, the system will also prompt the user to close their eyes and rest to allow the pupil to recover.

[0111] In another optional embodiment, the set conditions include morphological features, and the pupil shape in the image is determined to meet the set morphological features. If the pupil shape in the image meets the set morphological features, it is determined that the pupil meets the set conditions; otherwise, the user is prompted to open their eyes wide, try not to blink, etc. The set morphological features are circular or approximately circular. If the detected pupil does not meet the preset morphological features, for example, it may be flat, this is generally caused by the user's eyes not being open.

[0112] In a third optional embodiment, the above-mentioned neural network model is required to perform pupil detection, and the recognition result of the neural network model also includes confidence information of the pupil, that is, a probability value used to indicate that the model determines that a pupil exists in the image. The set conditions include a confidence threshold, which determines whether the confidence information obtained by the neural network model is greater than the confidence threshold. When the confidence information is greater than the confidence threshold, it is determined that a pupil that meets the set conditions exists; otherwise, the user is prompted to open their eyes wide and remove obstructions such as hair. The confidence of the pupil obtained by the neural network model is relatively low, indicating that although the pupil exists in the image, it may be interfered with by other objects. In order to improve the shooting quality, the user is prompted to make adjustments.

[0113] The above three embodiments can be used one by one or in combination. When the pupil in the image meets the set conditions, step S4 is executed, otherwise wait for the user to adjust their state and continue to judge until the set conditions are met.

[0114] S4, moving the fundus camera lens to align with the pupil according to the position of the pupil in the image. The lens barrel is moved by the above-mentioned motion component, and the movement direction and distance depend on the deviation between the pupil in the image and the lens. The center point of the captured image is regarded as the center point of the lens, and the center point of the pupil in the image is identified. Regarding the method of identifying the center point of the pupil in the image, for example, when using the above-mentioned neural network model to detect the pupil, the center point of the detection frame can be regarded as the center point of the pupil. Step S4 specifically includes:

[0115] S41, determining a moving distance and a moving direction according to a deviation between a center position of the detection frame and a center position of the image;

[0116] S42, moving the fundus camera lens to align with the pupil according to the determined moving distance and moving direction.

[0117] According to an embodiment of the present invention, a method for photographing fundus images is provided. By judging the pupil state in the image, it is possible to automatically determine whether the current pupil state of the subject is suitable for photographing the fundus image. When the state is not suitable for photographing the fundus image, a corresponding prompt can be issued to the subject to enable the subject to adjust their state. When the state is suitable for photographing the fundus image, the position of the pupil is identified and automatically aligned, and then the photograph is taken, thereby avoiding photographing unusable fundus images. The entire process does not require the participation of professionals, and users can take photos independently.

[0118] In actual application scenarios, a special situation may occur, that is, the size of the pupil may be smaller than the size of the annular illumination beam. In this case, aligning the pupil and the eyepiece lens will result in no light entering the pupil, so the captured image is black.

[0119] To solve this problem, with respect to step S700 above, an embodiment of the present invention provides a preferred method for photographing fundus images, which includes the following steps:

[0120] S51 , determining whether the pupil size in the image is smaller than the size of the annular illumination beam of the fundus camera illumination assembly. Figure 7 A case is shown where the size of the pupil 72 is larger than the size of the annular beam 71 , in which case step S52 is executed.

[0121] Figure 8 The diagram shows a case where the sizes of the two annular illumination beams are larger than the pupil size. The illumination light source is a complete annular illumination lamp or a light source formed by a plurality of illumination lamps arranged in an annular shape. The inner diameter of the annular beam 71 is larger than the diameter of the pupil 72 .

[0122] When the pupil size is smaller than the annular illumination beam size, it meets the following conditions: Figure 8 In the situation shown, step S53 is executed.

[0123] S52: Capture a fundus image at the current lens position. This is an image captured when the fundus is well illuminated by light.

[0124] S53, respectively moving the lens and the pupil in multiple directions to produce an offset, so that the annular illumination beam is partially illuminated in the pupil, and obtaining multiple fundus images. Figure 9 Taking the movement shown as an example, in this embodiment, the lens is moved in two horizontal directions respectively. When the lens moves to one side, a part 73 of the annular light beam 71 is irradiated into the pupil 72, and a fundus image is taken at this time; when the lens moves to the other side, another part 74 of the annular light beam 71 is irradiated into the pupil 72, and another fundus image is taken at this time.

[0125] Figure 9 The illustrated movement and lighting are merely examples for purposes of illustrating the capture scenario. In practice, the camera can be moved in more directions to capture more fundus images. However, fundus images captured using these movement and lighting conditions may be partially overexposed. Such fundus images cannot be directly used as the capture result, so step S54 is executed.

[0126] In addition, in order to reduce overexposed areas, in a preferred embodiment, the following method is used for moving and shooting:

[0127] S531, determine the edge position of the pupil. Specifically, a machine vision algorithm or the above-mentioned neural network model can be used to obtain Figure 9 The left edge point 721 and the right edge point 722 of the middle pupil 72 .

[0128] S532: Determine the moving distance based on the pupil edge position. Specifically, the moving distance of the motion component can be calculated based on the positional relationship between the current lens center position O (image center position) and the left edge point 721 and the right edge point 722.

[0129] S533, moving the lens in multiple directions according to the determined moving distance, wherein the determined moving distance makes the edge of the annular illumination beam coincide with the edge of the pupil. Figure 9 As shown, the outer edge of the annular beam 71 coincides with the edge of the pupil 72, so that the portion of the annular beam 71 entering the fundus is located at the edge of the fundus, reducing the impact on imaging in the central area of ​​the fundus.

[0130] S54, multiple fundus images are fused into one fundus image. In this step, usable areas are extracted from each fundus image, and these fundus images are stitched and fused to form a complete fundus image. There are many ways to stitch and fuse. As an optional embodiment, step S54 specifically includes:

[0131] S541a, calculating the displacement deviations of the multiple fundus images according to the lens movement distances corresponding to the acquired fundus images;

[0132] S542a, selecting a valid area from the multiple fundus images;

[0133] S543a, stitching the multiple effective areas according to the displacement deviation to obtain a stitched fundus image. Further, performing fusion processing on the stitching locations of the effective areas using an image fusion algorithm.

[0134] As another optional embodiment, step S54 specifically includes:

[0135] S541b, detecting corresponding feature points in multiple fundus images;

[0136] S542b, calculating the spatial transformation relationship of the plurality of fundus images according to the positions of the feature points;

[0137] S543b, setting the multiple fundus images in the same coordinate system according to the spatial transformation relationship;

[0138] S544b: Select valid areas from multiple fundus images in the same coordinate system and stitch them together to obtain a stitched fundus image.

[0139] According to an embodiment of the present invention, a method for capturing fundus images is provided. When the fundus camera lens is aligned with the pupil, the pupil size in the comparison image and the size of the annular light beam emitted by the camera itself are first determined. If the pupil size is too small, resulting in the illumination beam not being able to normally illuminate the fundus, the lens is moved away from the current alignment position so that the annular illumination beam partially illuminates the pupil, and fundus images are acquired at multiple offset positions. Finally, a single fundus image is fused based on the multiple fundus images. This solution can capture fundus images even when the subject's pupil is relatively small, does not require professional personnel to participate in the shooting process, reduces the requirements for the subject's pupil state, and improves shooting efficiency.

[0140] The following describes the movement of the camera lens (lens barrel) along the Z axis. This movement is related to the optimal working distance of the fundus camera. Regarding steps S400-S500 above, this embodiment provides a method for adjusting the working distance of a fundus camera. This method can be performed by the fundus camera itself or by an electronic device such as a computer or server (as a control method). The method includes the following steps:

[0141] S1, control the lens to approach the eyeball and collect an image, which is an image of the illumination beam reflected by the cornea. This step is performed according to the solution of the above embodiment, when the lens is aligned with the pupil in the XY plane. In this step, controlling the lens to approach the eyeball means controlling the lens to move toward the eyeball on the Z axis through the motion component. At the initial distance, the light source of the illumination component passes through the optical lens, and the reflected light irradiated on the cornea of ​​the eye is imaged on the CMOS, which will produce the following image: Figure 10 As shown in the results, in this embodiment, the light source is four light balls distributed in a cross shape on the four sides of the lighting assembly, and the imaging of this light source also shows four light spots accordingly. In other embodiments, the lighting source can be as follows Figure 8 The shape shown in the figure will show light spots of corresponding shape or arrangement in the collected image.

[0142] S2, detect whether the characteristics of the light spot in the image meet the set characteristics. Figure 11 As shown, as lens barrel 1 moves toward eyeball 01 along the Z axis, the image of corneal reflected light changes. Specifically, the position, size, and clarity of the image are related to the distance between the eyepiece objective and the cornea. The closer the distance, the greater the angle between the incident light and the corneal normal, the more severe the reflected light scattering effect, the larger the spot size, the more divergent it is, and the lower the brightness.

[0143] There are various ways to identify light spot features in an image. For example, machine vision algorithms can be used to detect the light spot's outline and position based on the image's graphical features. However, due to the wide range of variations in light spot clarity, size, and other aspects, computer vision algorithms are prone to misjudgment. Therefore, in a preferred embodiment, a deep learning algorithm is used to address this issue.

[0144] First, a large number of images of light spots are collected. These images are taken from different people, at different directions and distances from the fundus camera's eyepiece, and at different times. The light spots in each image are then labeled to generate training data for the neural network. This labeled data is used to train a neural network model (such as the YOLO network). After training, the neural network model's recognition results include a detection box that represents the location and size of the light spot in the image.

[0145] like Figure 12 As shown, in a specific embodiment, a square frame 121 is used to mark the light spot in the training data, and the recognition result of the trained neural network model will also be a square detection frame. In other embodiments, a circular frame can also be used for marking, or other similar marking methods are feasible.

[0146] Regardless of the light spot detection method used, it is sufficient to identify the light spot features in the current image that meet the set features in this step. The set features can be size-related, such as when the light spot size in the image is smaller than the set size, the set features are met; or the light spot disappears, such as when a machine vision algorithm or neural network cannot detect the light spot in the image, the set features are met.

[0147] If the light spot in the image meets the set characteristics, step S3 is executed; otherwise, the process returns to step S1 to continue moving the lens and capturing images.

[0148] S3, determine that the working distance has been reached. When it is determined that the characteristics of the light spot in the image meet the set characteristics, the distance between the lens and the eyeball can be considered to have reached the working distance. In a specific embodiment, based on the hardware parameters, a distance compensation can be performed on the basis of this distance. The direction and distance value of the compensation are related to the hardware parameters. For example, Figure 13 An image showing a light spot that meets the set characteristics is shown. At this time, the distance between the lens 1 and the eyeball 01 is WD. On this basis, the lens is controlled to continue moving toward the eyeball by a preset distance d to achieve a more accurate working distance WD+.

[0149] At this working distance, further adjusting the focus allows the fundus image to be captured. The method for adjusting the focus will be described in detail in subsequent embodiments.

[0150] According to the working distance adjustment method provided by an embodiment of the present invention, the imaging of the illumination light beam reflected by the cornea is collected and identified, and the distance between the lens and the eyeball is judged and adjusted based on the light spot characteristics in the image. No additional optics or hardware need to be set on the fundus camera. Only a suitable illumination light beam needs to be set to accurately position the working distance, thereby reducing the cost of the fundus camera and improving the efficiency of working distance adjustment.

[0151] Considering that the user may slightly turn their head during the movement of the lens toward the eyeball, which may cause the lens to no longer be aligned with the pupil, the position of the lens is also adjusted on the XY plane during the working distance adjustment process to maintain alignment with the pupil. This embodiment provides a preferred method for adjusting the working distance, which includes the following steps:

[0152] S1A, captures the image of the illumination beam reflected by the cornea;

[0153] S2A, calling a neural network to detect the light spot in the image and determine whether there is a light spot in the image. If there is no light spot in the image, execute step S6A; otherwise, execute step S3A.

[0154] S3A identifies the center of the light spot in the image and determines whether it coincides with the center of the image. The center of the detection frame generated by the neural network is considered the center of the light spot. The center of the image is considered the center of the lens. If the center of the image coincides with the center of the light spot, the lens is aligned with the pupil, and step S5A is executed. If they do not coincide, the lens is out of alignment, and step S4A is executed.

[0155] S4A, fine-tune the lens position based on the offset between the center point of the light spot and the center point of the image. Detection-adjustment-redetection is a feedback process. As a preferred embodiment, a smooth adjustment algorithm is used here:

[0156] Adjustment(i)=a*Shift(i)+(1-a)Adjustment(i-1),

[0157] Adjustment(i-1) represents the displacement of the last lens adjustment, Shift(i) represents the offset (the difference between the pupil center and the image center), and Adjustment(i) represents the displacement required for this lens adjustment. a is a coefficient between 0 and 1. Because the lens position is a two-dimensional coordinate on the XY plane, Adjustment and Shift are both two-dimensional vectors.

[0158] After the center point of the light spot and the center point of the image are adjusted to coincide with each other, step S5A is performed.

[0159] In step S5A, the lens is controlled to move closer to the eyeball to reduce the distance. The process then returns to step S1A. As the lens is repeatedly moved closer to the eyeball, the spot size in the corresponding image decreases from large to small. To accurately capture the critical point where the spot disappears, the above judgment and adjustment can be made for each frame of the image until an image is detected where the spot disappears.

[0160] S6A, controls the lens to continue to move toward the eyeball by a preset distance to reach the working distance.

[0161] In a preferred embodiment, while executing the above-mentioned adjustment process, it will also detect whether the light spot in the image is complete. When the light spot is incomplete, for example, only half, it means that the user is blinking or the eyes are not open. At this time, the system will prompt the user through voice to open the eyes, try not to blink, etc.

[0162] According to the working distance adjustment method provided by an embodiment of the present invention, while adjusting the distance between the lens and the eyeball, the lens position will also be fine-tuned according to the position of the light spot in the image, thereby keeping the lens aligned with the pupil when adjusting the working distance. This solution does not require any additional optics or hardware to be set on the fundus camera. It only needs to set a suitable illumination light beam to accurately position the working distance and keep the lens aligned with the pupil, thereby reducing the cost of the fundus camera and improving the efficiency of fundus image capture.

[0163] After the automatic alignment and automatic adjustment of the working distance in the above embodiment, it is still necessary to set the appropriate focal length to capture a clear fundus image. The following describes the technical solution for automatically adjusting the focal length. With respect to the above step S600, this embodiment provides a method for adjusting the focal length of a fundus camera. This method can be performed by the fundus camera itself, or by an electronic device such as a computer or server (as a control method). The method includes the following steps:

[0164] S1, adjust the focus and collect fundus images. This step is performed when the fundus camera lens is aligned with the pupil and reaches the working distance. At this time, the position of the lens and the eyeball is as follows: Figure 13 As shown. It should be noted that, when adjusting the lens position and working distance in the above embodiment to capture images, a fixed focal length is also required. For example, when adjusting the working distance, the focal length can be fixed to a zero diopter position. If the subject has normal refraction, fundus images can be captured directly after the working distance is adjusted. However, in actual applications, the subject's actual diopter must be considered to set an appropriate focal length.

[0165] Before the fundus camera is exposed to capture fundus images, such as during the aforementioned automatic alignment and working distance determination processes, infrared light is used for imaging. The light source used for image acquisition during this time is still infrared light. Although the current focal length does not allow for a clear fundus image, the image captured at this point already captures basic fundus features, at least showing the optic disc. Therefore, the captured image is called a fundus image.

[0166] S2, identifying the optic disc area in the fundus image. Because the optic disc area is the area with the most texture and the highest brightness in the fundus, it is most suitable for focusing.

[0167] There are various ways to identify the optic disc in fundus images. For example, machine vision algorithms can be used to detect the optic disc's outline and location based on graphical features in the fundus image. However, infrared imaging is relatively blurry, making optic disc identification a significant challenge. Computer vision algorithms are prone to misjudgment in this situation. Therefore, in a preferred embodiment, a deep learning algorithm is used to address this issue.

[0168] First, a large number of fundus images are collected, taken from different individuals and at different focal lengths. The optic disc in each image is then labeled, generating training data for the neural network. This labeled data is then used to train a neural network model (such as the YOLO network). After training, the neural network model's recognition results include a detection box that represents the location of the optic disc in the fundus image.

[0169] like Figure 14 As shown, in a specific embodiment, a square frame 141 is used to mark the optic disc in the training data, and the recognition result of the trained neural network model will also be a square detection frame. In other embodiments, a circular frame can also be used for marking, or other similar marking methods are feasible.

[0170] S3. Determine the shooting focal length based on the clarity of the optic disc area. Specifically, starting from an initial focal length, the focal length can be continuously changed using a gradient ascending method while capturing corresponding fundus images. It is determined whether the clarity of the optic disc meets a preset standard. Once the preset standard is met, the current focal length is determined to be the optimal focal length, and further searching is unnecessary. Alternatively, all available focal lengths within the adjustable focal length range can be used, and corresponding fundus images are captured. The fundus image with the highest optic disc clarity is determined from all fundus images, and the focal length at which this image was captured is determined to be the optimal focal length.

[0171] In a specific embodiment, a traversal approach is used to first adjust the focus within a set focal length range of 800-1300 in a first set step size of 40 and acquire a first set of fundus images. This progresses to fundus images at a focal length of 800, then at a focal length of 840, then at a focal length of 880, and so on, to a fundus image at a focal length of 1300. The optic disc area is identified in each of these fundus images, and the clarity of each fundus image is determined. In this embodiment, the clarity is calculated by calculating the average of the pixel values ​​within the optic disc area. Then, a fundus image with the highest clarity can be determined from the first set of fundus images. In this case, the focal length X (first focal length) used when acquiring this fundus image can be used as the shooting focal length.

[0172] To achieve better shooting effects, further focal length searches can be performed. For example, another traversal can be performed near the aforementioned focal length X. The second set step size used in this traversal process is smaller than the first set step size, for example, the second set step size is 10. This can further generate a second set of fundus images, namely, fundus images at focal lengths X+10, X+20, X-10, X-20, and so on. The optic disc area is then identified in each of these fundus images, and the clarity of each fundus image is determined. For example, if the fundus image at focal length X-20 is determined to have the highest clarity, then the focal length X-20 (the second focal length) is used as the shooting focal length.

[0173] Regarding further searching the focal length range, as a preferred embodiment, the first focal length X can be set as the midpoint to increase the focal length range with the first set step length as the maximum value and reduce the first set step length as the minimum value, and the range is X±40.

[0174] According to the focus adjustment method provided by an embodiment of the present invention, fundus images are collected at different focal lengths, and whether the current focal length is suitable for capturing fundus images is determined by the clarity of the optic disc in the fundus image. No additional optics or hardware need to be set on the fundus camera. Only an image recognition algorithm needs to be set to find the optimal focus position, thereby reducing the cost of the fundus camera and improving the efficiency of focus adjustment.

[0175] Considering that the user may slightly turn their head during the focus adjustment process, which may cause the lens to no longer be aligned with the pupil, the position of the lens is also adjusted on the XY plane to maintain alignment with the pupil. Furthermore, at this stage, the fundus image is about to be captured. If the subject blinks or closes their eyes at this time, the image will not be captured successfully. Therefore, blink and / or eye closure detection is required during this process. This embodiment provides a preferred focus adjustment method, which includes the following steps:

[0176] S1A, fundus image acquisition using the current focal length.

[0177] S2A, using the fundus image to determine whether the subject has blinked and / or closed their eyes. If the subject blinks and / or closes their eyes, a prompt is provided, such as a voice prompt to the user not to blink or close their eyes, and the process returns to step S1A; otherwise, the process proceeds to step S3A. Blink and eye closure detection can also be implemented using machine vision algorithms or neural network algorithms. When the subject blinks or closes their eyes, the captured image will be completely black or very blurry, with relatively distinct features. Various detection methods can be used, which will not be detailed here.

[0178] S3A, identify whether there is a light spot formed by the illumination beam reflected by the cornea in the fundus image. Different from the method of keeping the lens aligned with the pupil when adjusting the working distance in the above embodiment, after reaching the working distance, if it is in the aligned state, the illumination beam reflected by the cornea should not be within the imaging range, and the above light spot should no longer appear in the fundus image, especially it is impossible to form a complete imaging of the light spot. Even if a light spot appears, it will be a part of the entire light spot. In a specific embodiment, a light source formed by multiple lighting lamps arranged in a ring is used. The complete light spot is as follows Figure 12 If light spots appear in the fundus image when adjusting the focus, it will be as shown below. Figure 15 The situation shown here is only a partial light spot 151. If the light source itself were a complete ring light, this would appear as a stripe in the image.

[0179] When there is a light spot in the fundus image, step S4A is executed; otherwise, step S5A is executed.

[0180] S4A, fine-tune the lens position based on at least the position of the light spot to remove the light spot, thereby keeping the lens aligned with the pupil. When the light spot appears at different positions, its size and brightness will be different. As a preferred embodiment, the vector offset can be calculated based on the position, size and brightness of the light spot in the image. Figure 15 For example, the image center is used as the origin (0,0) to establish a coordinate system, and the image radius is R. Calculate the approximate circular area of ​​each light spot 151. In this embodiment, the approximate circular area is the smallest circular area containing the light spot 151. For example, the center coordinates of the approximate circular area of ​​the i-th light spot are (x i ,y i ), the radius is r i Then we can conclude that the direction that the i-th light spot needs to move is v i =(x i ,y i ), the distance to be moved is where k = x i 2 +y i 2 , and then we can conclude that the current spot needs to move vi m i , summing up the amounts that all the light spots need to move, the vector 152 that the lens needs to move is obtained as ∑vm.

[0181] After the lens is aligned with the pupil again, the process returns to step S1A.

[0182] S5A, identifying the optic disc area in the fundus image and determining whether the clarity of the optic disc area meets the set standard. In this embodiment, the Mobilenet-Yolov3 neural network model is used to identify the optic disc. The optic disc area output by the neural network is the area containing the optic disc and the background. An edge detection algorithm (such as Sobel or Laplace) is then used to detect the optic disc edge within this optic disc area to obtain an accurate optic disc image. The mean value of the optic disc image is then calculated as the clarity value.

[0183] For example, the obtained clarity value can be compared with a threshold to determine whether it meets the set standard. If the clarity of the optic disc area does not meet the set standard, step S6A is executed. If the clarity of the optic disc area does meet the set standard, it is determined that the current focal length is suitable for capturing fundus images. The infrared light can then be turned off and white light exposure can be used to capture the fundus image.

[0184] S6A, adjust the focal length, and then return to step S1A. According to the initial focal length used in step S1A, for example, if the initial focal length is the minimum value of the adjustable focal length, the focal length is increased in fixed steps or variable steps, otherwise the focal length is decreased.

[0185] After using the solutions provided by the above embodiments to align the lens with the pupil, adjust to the optimal working distance and determine the focal length, start capturing fundus images. When capturing fundus images, it is necessary to use an illumination component for exposure (the light source used by the camera of this embodiment is white light). However, during the exposure and shooting process, the subject may still affect the shooting quality of the fundus image, such as pupil shrinkage, eyelid occlusion, blinking, light leakage from the face-mounted component, etc. When these situations occur, unusable areas will appear in the captured fundus image. In order to improve the shooting success rate, with respect to the above step S700, this embodiment provides a fundus image shooting method, which can be executed by the fundus camera itself, or by an electronic device such as a computer or server (as a control method), and the method includes the following steps:

[0186] S1, maintaining the lens state unchanged and capturing multiple fundus images. Specifically, according to the methods of the above embodiments, the lens is fixed in position in the XY plane, aligned with the pupil, and positioned at a distance on the Z axis, with a fixed focal length. While maintaining the lens position, working distance, and focal length unchanged, the illumination assembly is exposed and multiple fundus images are captured.

[0187] S2. Determine the quality of each of the multiple fundus images. There are various methods for analyzing fundus image quality. For example, reference may be made to the fundus image detection method provided in Chinese patent document CN108346149A. In this embodiment, a neural network model is used to analyze image quality. The neural network model can perform classification tasks to classify image quality, such as outputting a classification result of high quality or poor quality; it can also perform regression prediction tasks to quantify image quality, such as outputting a score of 1-10 to express an evaluation of image quality.

[0188] To train the model, a large number of retinal images exposed to white light are collected in advance. The images are manually labeled as good or poor (for classification models) or scored (for example, on a scale of 1 to 10, for regression prediction models). These fundus images and the annotations or scores serve as training data for the neural network model. Once the model converges, it can be used to identify fundus image quality.

[0189] S3, determine whether the quality of each fundus image meets the set standard. If any fundus image meets the set standard, then the fundus image is used as the shooting result (output shooting result). If the quality of multiple fundus images does not meet the set standard, execute step S4.

[0190] S4, using multiple fundus images to synthesize a fundus image as the shooting result. Multiple fundus images taken continuously may not have good overall quality, but each image may have some areas of good quality. By stitching and fusing these available areas, a high-quality and complete fundus image can be obtained.

[0191] According to the fundus image shooting method provided by an embodiment of the present invention, while keeping the lens state unchanged and shooting multiple fundus images, the quality of the multiple fundus images is determined separately. When it is determined that all fundus images are unusable, the multiple fundus images are used to synthesize a complete fundus image. Even if the person being photographed interferes with the shooting process, it is possible to use the existing fundus images to obtain a higher-quality fundus image, thereby reducing the number of reshoots, reducing the user's usage difficulty, and improving the success rate of shooting fundus images.

[0192] Furthermore, an embodiment of the present invention provides a method for synthesizing fundus images, the method comprising the following steps:

[0193] S41: Acquire multiple fundus images captured with the lens state unchanged. These fundus images may have areas of poor quality and areas of good quality. Of course, if some fundus images are of extremely poor quality, such as images with a score of 0 that are completely black or completely white, these completely unusable images can be directly removed.

[0194] S42: Extract high-quality regions from each of the multiple fundus images. In this step, brightness can be calculated based on the pixel values ​​of the fundus images. By comparing the brightness with a brightness threshold, regions with high brightness and regions with low brightness are removed, thereby eliminating overexposed and underexposed regions and extracting regions with moderate brightness, i.e., high-quality regions. Sharpness can also be calculated based on the pixel values ​​of the fundus images. By comparing the sharpness with a sharpness threshold, regions with low sharpness are removed, thereby removing blurred regions and obtaining high-quality regions. Alternatively, high-quality regions can be extracted based on a combination of brightness and sharpness.

[0195] The regions extracted based on the actual brightness and / or sharpness of the fundus image are usually regions with irregular boundaries, e.g. Figure 16 The two high-quality areas shown are: the area on the left is from the upper part of a fundus image, and the area on the right is from the lower part of a fundus image.

[0196] In other optional embodiments, each fundus image may be divided into grids according to a fixed division method, and then the quality of each grid area is analyzed separately to extract high-quality grids, so as to obtain high-quality areas with regular boundaries.

[0197] S43, synthesizing the fundus image using the multiple high-quality regions. Since each fundus image may have some offset, in order to synthesize the fundus image more accurately, each fundus image may be mapped to the same coordinate system according to the offset before being stitched and fused.

[0198] As a preferred embodiment, Figure 17 As shown, first, abnormal area detection is performed on multiple fundus images to extract high-quality areas. In step S43, feature point extraction (or key points) is first performed on multiple fundus images respectively, which can be the center point of the optic disc, the intersection of blood vessels, and other significant positions. Feature point matching is then performed to match the feature points between different fundus images. After these feature points are matched, the matching information is used to calculate the offset between each fundus image (projection matrix calculation). Then, multiple high-quality areas are mapped into one fundus image based on the offset. For the overlapping parts between multiple high-quality areas, such as Figure 16 The two regions shown have overlapping centers. The pixel values ​​of the overlapping regions can be determined using the pixel values ​​of multiple high-quality regions and their corresponding weights. This is a fusion process based on weighted averaging. For example, the fusion process can be expressed as q1 / (q1+q2)*image1+q2 / (q1+q2)*image2, where q1 represents the weight corresponding to the first high-quality region, q2 represents the weight corresponding to the second high-quality region, image1 represents the first high-quality region, and image2 represents the second high-quality region.

[0199] The values ​​of the above weights are set according to the overall quality of the fundus image. For example, the first high-quality area is taken from the first fundus image and the second high-quality area is taken from the second fundus image. The quality of the first fundus image obtained according to the above quality analysis method (such as the score output by the neural network) is higher than the quality of the second fundus image, then the corresponding weight q1 is greater than q2.

[0200] like Figure 16 、 17 The situation shown is only an example to illustrate the principle of this solution. In actual use, more fundus images will be taken to ensure that more high-quality areas are extracted as much as possible to ensure that the generated fundus image is complete.

[0201] According to the fundus image synthesis method provided by an embodiment of the present invention, when multiple fundus images taken of a subject all have defects, this solution is used to extract high-quality areas from the multiple fundus images, and stitch and fuse them to obtain a high-quality complete fundus image, thereby reducing the difficulty of users taking selfies of fundus images and improving the success rate of shooting.

[0202] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0203] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0204] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0205] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0206] Obviously, the above embodiments are merely examples for clarity of explanation and are not intended to limit the implementation methods. Those skilled in the art will readily appreciate that other variations or modifications based on the above descriptions are possible. It is not necessary and impossible to enumerate all implementation methods here. Obvious variations or modifications arising therefrom remain within the scope of protection of the present invention.

Claims

1. A fundus camera detection method, characterized in that: The method comprises the following steps: Step S1 is to obtain a first image captured by a lens through a window of the surface mount assembly; Step S2 is to determine whether the brightness of the first image reaches a set standard, if the brightness reaches the set standard, then execute step S3, otherwise execute step S6; Step S3 is turning on the lighting assembly to obtain a second image captured by the lens through the window of the surface mount assembly; Step S4 is determining whether the human head fits the surface patch assembly based on the second image. This is done by determining whether the second image has a characteristic of being brighter in the center and gradually darker at the edges. If it is determined that the human head fits the surface patch assembly, step S5 is executed; otherwise, step S6 is executed. Step S5 is to start taking fundus images; Step S6 is to prompt the user to wear the fundus camera correctly.

2. A fully automatic portable Selfie fundus camera system for executing the fundus camera detection method according to claim 1, characterized in that: The fully automatic portable Selfie fundus camera system includes: case; A lens barrel is located inside the housing and is provided with an illumination assembly, a focusing assembly, an eyepiece objective lens, an optical lens group, and an imaging detector; A motion assembly, configured to drive the lens barrel to move inside the housing along the front-rear axis, the left-right horizontal direction, and the up-down vertical direction of the lens barrel; A face mount assembly is sealingly connected to the front portion of the housing. The face mount assembly includes a face mount body and a window through-hole formed on the face mount body for accommodating the eyes of the subject when the face mount body is in contact with the eyes of the subject. The moving shooting range of the eyepiece objective lens driven by the motion assembly does not exceed the range of the window through-hole.

3. The fully automatic portable Selfie fundus camera system according to claim 2, characterized in that: Also includes: At least one positioning shooting target is provided on a side of the surface-mounted body facing the lens barrel. The positioning shooting target is provided adjacent to the window through-hole and is used to align the lens barrel with the shooting image when the fundus camera is started.

4. The fully automatic portable Selfie fundus camera system according to claim 3, characterized in that: A raised portion is provided in the middle of the upper edge of the window through hole and protrudes toward the center of the window through hole; one positioning shooting target is provided and is located on the side of the raised portion facing the lens barrel.

5. The fully automatic portable Selfie fundus camera system according to claim 4, characterized in that: The positioning shooting target corresponds to the position of the center of the eyebrows of the person being photographed whose eyes are attached to the face sticker body.

6. The fully automatic portable Selfie fundus camera system according to claim 2, characterized in that: A protrusion protruding toward the window through hole is provided on the upper edge of the window through hole, which is used to align the lens barrel to capture images when the fundus camera is started.

7. The fully automatic portable Selfie fundus camera system according to claim 6, characterized in that: A positioning and shooting target is provided on a surface of the raised portion facing the lens barrel.

8. The fully automatic portable Selfie fundus camera system according to any one of claims 2 to 7, characterized in that: The surface of the face-sticking body facing away from the lens barrel is constructed into a shape that fits the facial contour around the eyes of the subject.

9. The fully automatic portable Selfie fundus camera system according to any one of claims 2 to 7, characterized in that: The motion assembly includes: a first track assembly arranged along the left-right horizontal direction; a base movably arranged on the first rail assembly along the first rail assembly, the lens barrel being fixedly arranged on the base; a second track assembly arranged along the vertical direction, the lens barrel and the base being movably arranged on the second track assembly; A third rail assembly is arranged along the front-rear axial direction, and the lens barrel and the base are movably arranged on the third rail assembly.

10. The fully automatic portable Selfie fundus camera system according to claim 9, characterized in that: The utility model also comprises a positioning component which is arranged inside the shell and is used for detecting the moving position of the lens barrel in the shell.

11. The fully automatic portable Selfie fundus camera system according to claim 10, characterized in that: The positioning component includes: A first positioning member and a second positioning member are provided on the left and right horizontal sides of the base to detect the moving position of the lens barrel in the left and right horizontal directions; a third positioning member, provided on the base, for detecting the moving position of the lens barrel in the up and down vertical directions; The first positioning member, the second positioning member and the third positioning member are all located within a movable covering range of the lens barrel on the base.

Images