Information processing device, control method, and information processing program

Abstract

An information processing device J includes a light control unit 32, a still image data acquisition unit P2, and an extraction unit 33. The light control unit 32 adjusts the amount of light emitted by a light emitting unit 61 and directed to an eye of a subject. The still image data acquisition unit P2 acquires a plurality of items of still image data on the basis of captured images obtained by capturing the fundus of the subject that has received the light emitted from the light emitting unit 61. The extraction unit 33 extracts extracted image data in which the fundus appears for each of the plurality of items of still image data.

Description

Information processing device, control method, and information processing program

[0001] The present invention relates to an information processing device, a control method, and an information processing program.

[0002] Ocular fundus diseases such as glaucoma, diabetic retinopathy, and macular degeneration tend to cause delayed symptoms, leading to blindness even if treatment is initiated when symptoms first appear. However, only ophthalmologists can observe and diagnose the fundus. Therefore, not only in developing countries, but also in people living in remote areas such as mountainous regions and remote islands, as well as elderly people, who do not have easy access to an ophthalmologist, potentially face a significant risk of blindness. Delays in seeing an ophthalmologist are a unique challenge for ocular fundus diseases, unlike anterior segment diseases that can be treated even in the late stages of symptoms. Therefore, there is a growing need for remote fundus diagnosis, which allows for fundus observation even in areas without ophthalmologists.

[0003] To enable remote diagnosis, when photographing a subject's fundus in an environment without specialized ophthalmic equipment, it is necessary to obtain images of a quality usable for diagnosis and provide high-quality images to a remote ophthalmologist. To support the photographing procedure, various adapters have been developed for attaching a fundus observation lens to a camera-equipped smartphone (e.g., International Publication No. 2016 / 154558 and U.S. Patent Application Publication No. 2019 / 0191988). The adapter is useful for maintaining a constant positional relationship between the smartphone camera, the observation lens, and the subject's eye.

[0004] International Publication No. 2016 / 154558 U.S. Patent Application Publication No. 2019 / 0191988

[0005] However, even when using a conventional adapter, it is difficult to obtain images of a quality that can be used for fundus observation because the equipment is different from that used in ophthalmology.

[0006] The embodiments of the present invention have been made in consideration of the above circumstances, and aim to provide an information processing device, a control method, and an information processing program that can easily acquire fundus images suitable for fundus observation.

[0007] The information processing device includes a light control unit, a still image data acquisition unit, and an extraction unit. The light control unit adjusts the amount of light emitted by a light-emitting unit directed toward the subject's eye. The still image data acquisition unit acquires a plurality of still image data based on captured images of the subject's fundus exposed to the light emitted from the light-emitting unit. The extraction unit extracts extracted image data showing the fundus from each of the plurality of still image data.

[0008] According to the embodiments of the present invention, it is possible to provide an information processing device, a control method, and an information processing program that can easily acquire a fundus image suitable for fundus observation.

[0009] FIG. 1 is a schematic cross-sectional view showing the structure of an eye photographed by an information processing device according to a first embodiment. FIG. 2 is a schematic external view showing an example of an adapter used for fundus photography. FIG. 3A is a schematic view showing an example of a fundus image. FIG. 3B is a schematic cross-sectional view of an eyeball illustrating each position of the fundus shown in FIG. 3A. FIG. 4 is a diagram showing an overview of the system configuration of an information processing device according to a first embodiment. FIG. 5 is a block diagram showing the hardware configuration of the information processing device according to the first embodiment. FIG. 6 is a functional block diagram showing an example of the functional configuration of the information processing device according to the first embodiment. FIG. 7 is a block diagram showing an example of the configuration of a camera. FIG. 8A is a diagram showing various distances when photographing a subject's fundus. FIG. 8B is an explanatory diagram explaining photographing a subject located several meters away using light emitted from a light-emitting unit of the information processing device according to the first embodiment. FIG. 9 is a flowchart showing an example of fundus photography processing by the information processing device according to the first embodiment. FIG. 10 is a sequence diagram showing an example of fundus photography processing by the information processing device according to the first embodiment. FIG. 11 is a flowchart for explaining an example of light intensity adjustment processing in an information processing device according to a fourth embodiment. Fig. 12 is an explanatory diagram for explaining an optical system for photographing a fundus in an information processing apparatus according to the fourth embodiment. Fig. 13 is a diagram showing design values ​​of the optical system for photographing a fundus in an information processing apparatus according to the fourth embodiment.

[0010] (Configuration of Information Processing Device J According to First Embodiment) First, the structure of the eye photographed by the information processing device J will be described.

[0011] 1 is a schematic cross-sectional view showing the structure of an eye photographed by an information processing device J. An eyeball 70 has a sclera 71, a choroid 72, a retina 73, a vitreous body 74, a lens 75, a cornea 76, and an iris 78. The lens 75 is supported by a zonules of Zinn 77. The size of the pupil 79 changes as the iris 78 expands and contracts. The retina 73 is called the fundus.

[0012] The central part of the retina 73 is the macula 82. Cells responsible for visual acuity and color vision are particularly concentrated in the macula 82. The center of the macula 82 is the fovea centralis 81. Multiple nerve fibers run throughout the retina 73, and these multiple nerve fibers converge at a part called the optic disc 84 to form the optic nerve 83.

[0013] Fig. 2 is a schematic view showing an example of an adapter 90 used for fundus photography. Fig. 2 shows the adapter 90, an auxiliary tool for fundus photography, attached to a smartphone, which is a portable information processing terminal 150 used for fundus photography. The adapter 90 has a main body 91, an objective lens 92, and a lens support 93 that supports the objective lens 92. An opening 96 is formed in the main body 91 at the position of the camera 44 of the portable information processing terminal 150. A claw 95 is provided on the bottom of the adapter 90.

[0014] The lens support part 93 is supported on the upper part of the main body part 91 so as to be rotatable around a rotation axis 94, and adjusts the optical axis of the camera lens 52 provided on the camera 44 of the portable information processing terminal 150 and the optical axis of the objective lens 92. In other words, the adapter 90 holds the portable information processing terminal 150 so that the objective lens 92 and the camera 44 are positioned at predetermined photographing positions with respect to the subject's eye.

[0015] The procedure for capturing a still image of a subject's fundus using the adapter 90 and the portable information processing terminal 150 shown in FIG. 2 will be described. The doctor attaches the main body 91 of the adapter 90 to the portable information processing terminal 150 and hooks the claw portion 95 onto the bottom end of the portable information processing terminal 150. This secures the adapter 90 to the portable information processing terminal 150. In a dark room, the doctor operates the portable information processing terminal 150 to set the portable information processing terminal 150 to imaging mode. The purpose of capturing images in a dark room is to enlarge the subject's pupil 79 as much as possible to make it easier to capture images of the fundus. In this case, a mydriatic drug to dilate the pupil 79 may be administered to the subject before capturing images. When the doctor inputs a capture command into the portable information processing terminal 150 while the subject places their eye over the objective lens 92 or faces the eye directly, the portable information processing terminal 150 emits a flash light and captures a still image of the fundus.

[0016] FIG. 3A is a schematic diagram showing an example of a fundus image. FIG. 3B is a schematic cross-sectional view of an eyeball 70 illustrating each position of the fundus shown in FIG. 3A. As shown in FIG. 3A, an image that can be used for fundus observation is one in which the optic disc 84 is clearly defined. Another example is an image in which the retinal blood vessels 85, consisting of an artery 85a and a vein 85b, are clearly defined. Another example is an image in which the macula 82 is clearly defined, so that the poles of the macula 82 are clearly visible. In other words, an image that can be used for fundus observation is one in which at least one of the optic disc 84, the retinal blood vessels 85, and the macula 82 is clearly visible or clearly defined.

[0017] However, when photographing the fundus of a subject using the adapter 90 and the portable information processing terminal 150 shown in Fig. 2, an image of a quality usable for fundus observation may not be obtained. The following three reasons are considered to be the reasons why it is difficult to obtain an image of a quality usable for fundus observation.

[0018] (1) If the subject's eyeball 70 and / or eyelids move frequently due to the flash light during photography, the relative positions of the camera 44, the objective lens 92, and the subject's fundus will fluctuate.

[0019] (2) In accordance with the fluctuations in (1) above, the doctor needs to change the positions of the camera 44 and the objective lens 92 while operating the portable information processing terminal 150 to take photographs in parallel.

[0020] (3) If the portable information processing terminal 150 is shaken even slightly by hand shake or the like during the operation (2) above, the positional relationship between the objective lens 92 and the subject's fundus will also fluctuate.

[0021] The information processing device J photographs the fundus of the subject by setting the light emission amount of the light emitting unit 61 to a value lower than the default value of the portable information processing terminal 1 .

[0022] 4 is a diagram showing an outline of the system configuration of the information processing device J. The information processing device J has a mobile information processing terminal 1, a server 2, a diagnosis terminal 3, and a subject terminal 4, which are communicably connected to each other via a network 100 such as the Internet.

[0023] The mobile information processing terminal 1, the diagnostic terminal 3, and the subject terminal 4 are smartphones, tablets, or the like. The mobile information processing terminal 1 is used by medical professionals such as doctors or nurses in facilities such as hospitals or clinics in areas where there are no ophthalmologists. Hereinafter, the user operating the mobile information processing terminal 1 will be referred to as a medical professional, but is not limited to this. The diagnostic terminal 3 is used by a doctor. The subject terminal 4 is used by a subject undergoing an eye examination. The server 2 stores information received from the mobile information processing terminal 1 and the diagnostic terminal 3, and relays communication between the mobile information processing terminal 1, the diagnostic terminal 3, and the subject terminal 4.

[0024] 5 is a block diagram showing the hardware configuration of the information processing device J. The portable information processing terminal 1 has a CPU 11, a ROM 12, a RAM 13, a bus 14, an input / output interface 15, an output unit 16, an input unit 17, a storage unit 18, a communication unit 19, and a drive 20.

[0025] The CPU 11 executes various processes in accordance with programs recorded in the ROM 12 or programs loaded from the storage unit 18 into the RAM 13. The RAM 13 also stores data and the like required for the CPU 11 to execute various processes. The CPU 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output interface 15 is also connected to this bus 14.

[0026] The input / output interface 15 is connected to an output unit 16, an input unit 17, a storage unit 18, a communication unit 19, and a drive 20. The output unit 16 is composed of a display 41, a speaker 42, etc., and outputs various information as images and sounds. The input unit 17 is composed of a keyboard, a mouse, etc., and inputs various information. The storage unit 18 is composed of storage elements such as an HDD or SSD, and stores various data. The communication unit 19 communicates with other devices via a network 100.

[0027] Removable media 21, which may be a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, is appropriately attached to the drive 20. Programs read from the removable media 21 by the drive 20 are stored in the storage unit 18 as needed. The removable media 21 can also store various data stored in the storage unit 18 in the same way as the storage unit 18.

[0028] 5 shows a configuration in which the drive 20 is provided in the portable information processing terminal 1, but the drive 20 may not be provided. The server 2, the diagnosis terminal 3, and the subject terminal 4 also have hardware configurations as shown in FIG. 5, but detailed descriptions of these configurations will be omitted.

[0029] 6 is a functional block diagram showing an example of the functional configuration of the information processing device J according to the first embodiment. The CPU 11 of the mobile information processing terminal 1 functions as a control unit 31, a light adjustment unit 32, and an extraction unit 33. The communication unit 19 communicates with the server 2 using a predetermined communication protocol such as the Internet Protocol.

[0030] The memory unit 18 pre-stores the first trained model M1, various thresholds and reference values ​​used in the judgment process, etc.

[0031] The output unit 16 has a display 41 that displays images and a speaker 42 that outputs sound. The input unit 17 has a touch panel 43 that allows the user to input instructions to the portable information processing terminal 1, a camera 44 that photographs the fundus of the subject, and a microphone 45 that collects sound. The touch panel 43 is provided on the surface on which the display 41 displays images.

[0032] 7 is a block diagram showing an example of the configuration of the camera 44. The camera 44 has an imaging unit 51 and a light emitting unit 61. The imaging unit 51 has a camera lens 52 and an image sensor 53. The image sensor 53 is configured by, for example, a CCD or a CMOS.

[0033] The light-emitting unit 61 has an LED 62 that functions as a lighting means and a voltage adjustment circuit 63. The LED 62 is, for example, an SMD type. The voltage adjustment circuit 63 adjusts the DC voltage supplied from a battery (not shown) according to the control of the light adjustment unit 32 and supplies the adjusted voltage to the LED 62.

[0034] The control unit 31 controls each unit according to the operation mode set by the user. When the normal shooting mode is set, the control unit 31 shoots still images or videos according to shooting instructions. Video shooting includes not only shooting videos having multiple frames but also shooting still images repeatedly at predetermined time intervals, with the still images constituting the frames. When shooting videos, the imaging unit 51 shoots multiple still images at a frame rate of, for example, 30 fps. The frame rate value is not limited to 30 fps and can be set to another value, which is stored in the storage unit 18.

[0035] Furthermore, when the inspection mode is set, the control unit 31 causes the imaging unit 51 to capture a moving image, and determines whether or not the captured still image data contains an image in which the frame of the objective lens 92 is captured. If the control unit 31 determines that there is still image data in which the frame of the objective lens 92 is captured, the control unit 31 transitions to a standby state. When a shooting instruction is input in the standby state, the control unit 31 causes the imaging unit 51 to capture a moving image, rather than a still image, for a predetermined shooting time. The shooting time is, for example, 5 to 10 seconds. The shooting time is stored in the storage unit 18. Furthermore, the control unit 31 divides the moving image acquired from the imaging unit 51 in the inspection mode into a plurality of still image data for each frame, and transmits the plurality of still image data to the extraction unit 33.

[0036] That is, the control unit 31 constitutes an imaging instruction unit P1 that starts imaging the fundus of the subject and then ends the imaging after the imaging time has elapsed. The control unit 31 also constitutes a still image data acquisition unit P2 that acquires multiple still image data based on the captured image of the fundus of the subject that has received light emitted from the light emitter 61.

[0037] When a shooting instruction is input, the light adjustment unit 32 acquires an illuminance value indicating the brightness of the subject from the image sensor 53 and compares the acquired illuminance with a predetermined illuminance threshold. When the normal shooting mode is set and the acquired illuminance is lower than the illuminance threshold, the light adjustment unit 32 controls the voltage adjustment circuit 63 to set the light emission amount to a value (default value) preset in the portable information processing terminal 1 and cause the LED 62 to emit light. On the other hand, when the inspection mode is set, the light adjustment unit 32 controls the voltage adjustment circuit 63 to set the light emission amount of the LED 62 to a value lower than the default value of the portable information processing terminal 1 even if the acquired illuminance is lower than the illuminance threshold.

[0038] FIG. 8A is a diagram showing various distances when photographing the fundus of a subject.

[0039] The control of the light emission amount of the LED 62 by the light control unit 32 will be described. Let θ be the two-dimensional angle indicating the spread of light emitted in a certain direction at a solid angle dΩ from the LED 62, which is the light source. Let KS [lm] be the luminous flux corresponding to the light emission amount of the LED 62 at the solid angle dΩ, and let KD [cd] be the luminous intensity, which is the strength of the light emitted from the LED 62 at the solid angle dΩ. In this case, the luminous intensity KD is expressed by the following mathematical formula (1) using the luminous flux KS and the angle θ: KD = KS / [2π{1 - cos(θ / 2)}] (1)

[0040] Furthermore, the distance L from the LED 62, which is the light source, to the subject and the illuminance SD [lux] of the subject are expressed by the following equation (2) using the luminous intensity KD and the distance L. From equation (2), it can be seen that, for the same value of KD, the longer the distance L, the smaller the illuminance SD. SD = KD / L ２ ... (2)

[0041] As shown in Figure 8A, distance L3 is 20 to 77 mm, or 0.020 to 0.077 m. Distance L2 is, for example, 0.15 m. Distance L1 is the sum of distance L2 and the diameter of the eyeball 70 (approximately 2.5 cm), or approximately 0.175 m. Angle θ1 is θ1 = (0.005 / 0.15) × (180° / π) = 1.9°.

[0042] FIG. 8B is an explanatory diagram illustrating the use of light emitted by the light-emitting unit 61 to capture an image of a subject located several meters away. The distance L4 shown in FIG. 8B is, for example, 1-2 m. Consider a case in which a user, in normal capture mode, captures an image of a subject that is a distance L4 away from the mobile information processing terminal 1, as shown in FIG. 8B. In this case, if the acquired illuminance is lower than the illuminance threshold, the light-adjusting unit 32 sets the light emission amount to a default value and causes the light-emitting unit 61 to emit light. The default value is the maximum light emission amount of the LED 62. If the luminous flux KS of the LED 62 of the light-emitting unit 61 is adjustable within a range of 40-60 lm, the default value is 60 lm.

[0043] In the fundus photography of Fig. 8A, consider the illuminance SD at the subject when the light emitting unit 61 emits light in the same manner as in Fig. 8B. In Fig. 8B, the illuminance at the distance L2 is SD1, and the illuminance at the distance L4 is SD2. The luminous intensity KD at θ1 is the same for both the distance L2 and the distance L4. From equation (2), SD1 = KD / L2 ２ =KD / (0.15) ２ , SD2=KD / L4 ２ = KD / 2 ２ From KD / 1 ２ When the ratio of illuminance SD1 to illuminance SD2 is calculated, SD1 / SD2 is a value between 1 / 0.0225 and 4 / 0.0225, which corresponds to a magnification between 44 and 177 times. In other words, if the light-emitting unit 61 is made to emit light at the same light intensity as when photographing a subject several meters away in dark conditions, excessive light will be irradiated onto the eye and fundus. Therefore, in the examination mode, the light adjustment unit 32 sets the light emission intensity of the LED 62 of the light-emitting unit 61 to a value lower than the default value of the portable information processing terminal 1.

[0044] Typically, the exposure amount is calculated based on the integral of the intensity of the illuminance SD and the exposure time. The intensity of the illuminance SD is determined by the distance L and the aperture value in Equation (2). However, in a mobile information processing terminal 1 such as a smartphone, the aperture value is usually constant. Therefore, in the mobile information processing terminal 1, the exposure amount at the subject's fundus is considered to be determined by the exposure time and the illuminance SD. However, as described with reference to Figures 8A and 8B, when the value of the illuminance SD is too large, the exposure time must be shortened, but there is a limit to the shutter speed. Therefore, in order to suppress the subject's reflexive movement of the eyeball 70 and eyelids during photography and to obtain an appropriate exposure amount at the fundus, the light adjustment unit 32 adjusts the light emission amount of the light-emitting unit 61 rather than adjusting the exposure time.

[0045] Furthermore, the light adjustment unit 32 may change the reference value of the light emission amount depending on the type of the portable information processing terminal 1. For example, suppose there are two types of portable information processing terminals 1, model A and model B, as follows. Model A allows the light flux KS of the light-emitting unit 61 to be selected from three levels: 40, 50, and 60 lm within a range of 40-60 lm. Model B allows the light flux KS of the light-emitting unit 61 to be selected from four levels: 50, 60, 70, and 80 lm within a range of 50-80 lm. When the portable information processing terminal 1 is model A, the light adjustment unit 32 sets the light emission amount in the inspection mode to 50 lm, which is the middle level of the three levels. On the other hand, when the portable information processing terminal 1 is model B, the light adjustment unit 32 sets the light emission amount in the inspection mode to 50 lm, which is the smallest level of the four levels. In this way, the default value of the light emission amount of the light-emitting unit 61 is set to the maximum value of the light-emitting unit 61, and even if the default value differs for each type of mobile information processing terminal 1, the light emission amount is adjusted to an appropriate value.

[0046] When the extraction unit 33 receives multiple still image data from the imaging unit 51, it extracts extracted image data that meets predetermined criteria based on the first trained model M1. The extracted image data is still image data extracted by the extraction unit 33. Specifically, the extraction unit 33 determines, based on the first trained model M1, still image data in which at least one of the optic disc 84, retinal blood vessels 85, and macular region 82 in the fundus image is determined to be clearly connected, as extracted image data that meets predetermined conditions. The number of extracted image data may be one or more. The extraction unit 33 transmits the one or more extracted extracted image data to the server 2 via the communication unit 19. That is, the extraction unit 33 extracts extracted image data in which the fundus is visible for each of the multiple still image data. More specifically, the extraction unit 33 acquires a score by inputting still image data acquired by the still image data acquisition unit P2 into a first trained model M1 generated by training a first training image in which at least one of the subject's optic disc 84, retinal blood vessels 85, and macular region 82 appears, and extracts extracted image data based on the score. For example, when the extraction unit 33 inputs still image data into the first trained model M1 and acquires a score higher than a threshold, it acquires the still image data as extracted image data. The threshold is adjusted empirically or experimentally so that appropriate extracted image data can be acquired.

[0047] The first trained model M1 is trained with a first training image in which at least one of the optic disc 84, retinal blood vessels 85, and macular region 82 in the fundus image is clearly formed with a sensitivity of 97% or more, for example, based on 15,000 or more training data pieces in advance for 4,000 or more sample images, so that it can determine whether the still image data of the fundus is of a quality that can be used for diagnosis, i.e., whether at least one of the optic disc 84, retinal blood vessels 85, and macular region 82 is clearly formed.

[0048] Next, a description will be given of the processing for photographing a fundus by the portable information processing terminal 1. Fig. 9 is a flowchart showing an example of the fundus photographing processing of the information processing device J according to the embodiment. The user operates the touch panel 43 to set the operation mode of the portable information processing terminal 1 to the examination mode.

[0049] In S1, the control unit 31 causes the imaging unit 51 to capture a moving image, and determines whether or not the plurality of still image data includes an image that captures the frame of the objective lens 92. If the control unit 31 determines in S1 that there is still image data that captures the frame of the objective lens 92, it determines whether or not a capture instruction has been input (S2).

[0050] If a shooting instruction is input as a result of the determination in S2, the light adjustment unit 32 sets the light emission amount of the LED 62 of the light emitting unit 61 to a value lower than the default value of the mobile information processing terminal 1 (S3). In S4, the control unit 31 causes the imaging unit 51 to capture a video of the fundus for the shooting time. The imaging unit 51 transmits multiple still image data acquired by video shooting to the extraction unit 33.

[0051] In S5, when the extraction unit 33 receives a plurality of still image data from the imaging unit 51, it extracts extracted image data that satisfies predetermined conditions based on the first trained model M1. Specifically, the extraction unit 33 determines, based on the first trained model M1, still image data in which at least one of the optic disc 84, the retinal blood vessels 85, and the macular region 82 in the fundus image is determined to be clearly formed, as extracted image data that satisfies the predetermined conditions. The number of extracted image data to be extracted may be one or more.

[0052] In S6 , the extraction unit 33 transmits the extracted one or more pieces of extracted image data to the server 2 via the communication unit 19 .

[0053] In addition, in S5, if the extraction unit 33 is unable to extract extracted image data that meets the predetermined conditions, it may notify the user by voice via the speaker 42 a message indicating that the optical axis of the camera lens 52 and the optical axis of the objective lens 92 are misaligned.

[0054] Furthermore, in S5, the extraction unit 33 may perform a process of cutting out an image within the frame of the objective lens 92 from each of the plurality of still image data. In this case, the range of the image targeted for the extraction process is narrowed, thereby reducing the amount of calculation required for the extraction process and the processing load on the extraction unit 33. As a result, the extraction process by the extraction unit 33 is accelerated. Furthermore, the volume of still image data from which an image within the frame of the objective lens 92 has been cut out is smaller than the volume of still image data from which an image within the frame of the objective lens 92 has not been cut out. As a result, when the extraction unit 33 transmits the extracted image data to the server 2 via the communication unit 19 in S6, the communication processing load on the communication unit 19 is also reduced.

[0055] Next, a description will be given of fundus photographing processing by the information processing device J. Fig. 10 is a sequence diagram showing an example of fundus photographing processing by the information processing device J.

[0056] In S11 , the portable information processing terminal 1 transmits extracted image data of the fundus image of the subject to the server 2 .

[0057] In S12 , when the server 2 receives the extracted image data from the portable information processing terminal 1 , the server 2 stores the extracted image data and transmits the extracted image data to the diagnosis terminal 3 .

[0058] In S13, when the diagnostic terminal 3 receives the extracted image data from the server 2, it displays a fundus image based on the extracted image data. An ophthalmologist observes the fundus image displayed on the diagnostic terminal 3, makes a diagnosis, and inputs the diagnostic result to the diagnostic terminal 3. In S14, the diagnostic terminal 3 transmits the input diagnostic result to the server 2. In S15, when the server 2 receives the diagnostic result from the diagnostic terminal 3, it stores the diagnostic result and transmits the diagnostic result to the subject terminal 4. In S16, when the diagnostic result is received from the server 2, the subject terminal 4 displays the diagnostic result.

[0059] In the embodiment, the mobile information processing terminal 1 captures a moving image in the inspection mode, but it may also capture a single still image without capturing a moving image.

[0060] In addition, in the embodiment, an example has been described in which the dimming unit 32 can select the light emission amount in three or four levels, but this is not limited to this, and the light emission amount may be adjusted using a slide bar so that the light emission amount becomes a value lower than the default value.

[0061] Furthermore, in the embodiment, an example has been described in which the light-emitting unit 61 is provided in the portable information processing terminal 1, but the light-emitting unit 61 may be provided in the adapter 90. If dilation of the subject's pupil is essential, it is desirable to provide the light-emitting unit 61 in the portable information processing terminal 1, but if dilation of the subject is not necessary, the light-emitting unit 61 can also be provided in the adapter 90.

[0062] <Advantageous Effects of the First Embodiment> According to the information processing device J, when the portable information processing terminal 1 photographs the fundus of the subject, the light emission intensity of the light-emitting unit 61 of the portable information processing terminal 1 is set lower than the default value, thereby suppressing movement of the eyeball 70 and eyelids due to a reflex reaction when light is emitted. As a result, fluctuations in the positional relationship between the objective lens 92 and the fundus during photography are suppressed. This saves the doctor the trouble of adjusting the positional relationship between the camera 44 and the objective lens 92 during photography, eliminates the effects of camera shake, and makes it easier to obtain high-quality fundus images.

[0063] In the past, the glare of the flash could cause discomfort to subjects and lead to resistance to fundus examinations. Children, in particular, may remember fundus photography as an unpleasant test, refusing to undergo subsequent examinations and even being unable to undergo the examination at all. Furthermore, patients with corneal disorders, cataracts, and retinopathies, including hereditary retinal diseases, are more susceptible to glare than healthy individuals, and may be more aware of the discomfort caused by the flash during photography, making fundus examinations uncomfortable for the patient. In response to this issue, the information processing device J performs fundus photography using an adjusted light intensity to achieve an appropriate exposure for fundus photography. This reduces the glare experienced by the subject and prevents the subject from experiencing discomfort during fundus photography.

[0064] Furthermore, the light adjustment unit 32 may adjust the light emission amount to a value smaller than the default value in accordance with the type of portable information processing terminal 1. Even if the default value of the light emission amount of the light-emitting unit 61 is set to the maximum value of the light-emitting unit 61 and the default value differs for each type of portable information processing terminal 1, the light emission amount is adjusted to an appropriate value.

[0065] (Variation 1 of the First Embodiment) The portable information processing terminal 1 may capture images for a certain period of time with the light emission amount of the light-emitting unit 61 set lower than the default value. The light-emitting unit 61 emits light for a predetermined period of time at a light emission amount adjusted to a value lower than the default value. The imaging unit 51 captures images of the fundus at a predetermined frame rate within the certain period of time. The extraction unit 33 extracts extracted image data that satisfies predetermined conditions from multiple still image data of the fundus captured by the imaging unit 51.

[0066] According to the first modification, the brightness of the light irradiated onto the subject's eye during fundus photography does not change, which significantly reduces the movement of the eyeball 70 and eyelids. In addition, since a large number of still image data can be acquired, it is more likely that high-quality images useful for fundus observation can be obtained.

[0067] (Second Modification of First Embodiment) The portable information processing terminal 1 may change the amount of light emitted by the light-emitting unit 61 within a certain period of time, with a state lower than a default value as a reference. The light adjustment unit 32 changes the amount of light emitted by the light-emitting unit 61 within a predetermined period of time, with a value lower than the default value as a reference value. The imaging unit 51 photographs the fundus at a predetermined frame rate within the certain period of time. The extraction unit 33 extracts extracted image data that satisfies predetermined conditions from multiple still image data of the fundus photographed by the imaging unit 51.

[0068] Here, there are two possible methods for changing the light emission amount: a stepwise increase in the light emission amount from a reference value, and a stepwise decrease in the light emission amount from the light-emitting unit 61. For example, the light adjusting unit 32 increases the light emission amount of the light-emitting unit 61 from a reference value in a stepwise manner while photographing the fundus. In this case, if the subject has severe opacity of the optical intermediate body, such as a cataract, it becomes easier to find abnormalities by observing the fundus.

[0069] For example, if the subject complains of photophobia or the like, the doctor may operate the touch panel 43 to input an instruction to reduce the light emission intensity of the light-emitting unit 61. In this case, the light adjusting unit 32 gradually reduces the light emission intensity of the light-emitting unit 61 from the reference value while photographing the fundus. As a result, the subject is prevented from reflexively closing his or her eyelids or moving his or her eyeballs 70 upward or moving his or her entire face due to disgust, and a good fundus image can be obtained.

[0070] According to the second modification, a large number of still image data can be acquired while the light emission amount of the light emitted by the light-emitting unit 61 to the subject's eye is changed (increased or decreased) based on a light emission amount that is lower than the default light emission amount of the portable information processing terminal 1. This increases the possibility of obtaining an image with an appropriate exposure that is useful for fundus observation.

[0071] In other words, the information processing device J to which the present invention is applied can take various forms having the following configurations.

[0072] That is, the information processing device J has an imaging means (e.g., an imaging unit 51) that images the fundus of the subject when an imaging instruction is input, an emitting means (e.g., an emitting unit 61) that emits light when the fundus is photographed by the imaging means, and a light adjusting means (e.g., a light adjusting unit 32) that adjusts the amount of light emitted by the emitting means to a value lower than a predetermined value when an imaging instruction is input.

[0073] As a result, when the information processing device J photographs the fundus of the subject, the light emission amount of the light-emitting unit 61 of the information processing device J is set lower than the default value, thereby suppressing the movement of the subject's eyeball 70 and eyelids due to the reflex reaction when light is emitted, making it easier to obtain high-quality fundus image data.

[0074] The device may further include an extraction means (e.g., extraction unit 33) that extracts extracted image data from a plurality of still image data of the fundus photographed at predetermined time intervals by the photographing means, where the photographed fundus image satisfies predetermined conditions, and the light emitting means emits light at an amount of light emission adjusted to a low value for a predetermined fixed period of time, and the photographing means photographs the fundus at time intervals within the fixed period of time.

[0075] This prevents the brightness of the light irradiated onto the subject's eye during fundus photography from changing, thereby significantly suppressing the movement of the eyeball 70 and eyelids.

[0076] The device may further include an extraction means for extracting extracted image data from a plurality of still image data of the fundus photographed at predetermined time intervals by the photographing means, the photographed fundus image satisfying predetermined conditions, and the light adjusting means may change the amount of light emitted by the light emitting means within a predetermined fixed time period using a low value as a reference, and the photographing means may photograph the fundus at time intervals within the fixed time period.

[0077] This allows a large number of still image data to be acquired while the light emission amount of the light emitted by the light-emitting unit 61 to the subject's eye is changed based on a light emission amount that is lower than the default light emission amount of the information processing device J. This increases the likelihood of obtaining an image with an appropriate exposure that is useful for fundus observation.

[0078] The preset value may also be the maximum light emission amount of the light emitting means.

[0079] As a result, the amount of light emitted by the light emitting means is smaller than the amount of light emitted when photographing a subject several meters away in dark conditions, and movement of the subject's eyeball 70 and eyelids due to a reflex reaction when light is emitted is suppressed.

[0080] The control method is executed by an information processing device J having an imaging means, a light emitting means, and a communication means, and includes a dimming process for adjusting the amount of light emitted by the light emitting means to a value lower than a preset value when an imaging instruction is input, an imaging process for causing the imaging means to photograph the fundus of the subject when an imaging instruction is input, and a light emitting process for causing the light emitting means to emit light when the fundus is photographed by the imaging means.

[0081] The photographing method also includes a dimming step of photographing the fundus using an information processing device J having a photographing means, a light emitting means, and a communication means, and adjusting the amount of light emitted by the light emitting means to a value lower than a preset value when a photographing instruction is input, a photographing step of causing the photographing means to photograph the fundus of the subject when a photographing instruction is input, and a light emitting step of causing the light emitting means to emit light when the fundus is photographed by the photographing means.

[0082] In addition, the information processing program causes the computer to execute a dimming process of adjusting the amount of light emitted by the light-emitting means to a value lower than a preset value when an instruction to photograph is input to an information processing device J having a photographing means, a light-emitting means, and a communication means, a photographing process of causing the photographing means to photograph the fundus of the subject when an instruction to photograph is input, and a light-emitting process of causing the light-emitting means to emit light when the fundus is photographed by the photographing means.

[0083] Second Embodiment An information processing device J of the second embodiment captures a video of the fundus of a subject and acquires some still image data from the video data as an image of a quality usable for fundus observation. In the description of this embodiment, the same configuration as in other embodiments and modifications will not be described.

[0084] The information processing device J has an imaging means (e.g., an imaging unit 51) that images the fundus of the subject at predetermined time intervals when an imaging instruction is input, an extraction means (e.g., an extraction unit 33) that extracts extracted image data from a plurality of still image data of the fundus captured at time intervals by the imaging means, where the captured fundus image satisfies predetermined conditions, and a communication means (e.g., a communication unit 19) that transmits the extracted image data that satisfies the conditions extracted by the extraction means to an external device. This makes it possible to obtain a plurality of still image data of the fundus of the subject, thereby increasing the possibility of extracting high-quality extracted image data from the plurality of still image data.

[0085] The apparatus may further include an output means (e.g., output unit 16) that notifies the subject of the remaining time required for imaging when the imaging means starts imaging. This notifies the subject of the remaining time and encourages the subject to try not to move the eyeball 70 or eyelids for the remaining time. This increases the likelihood of obtaining a high-quality still image.

[0086] The image capturing unit may terminate the image capturing when a predetermined image capturing time has elapsed since the start of the image capturing, or when the extraction unit has extracted extracted image data that satisfies a condition, thereby reducing the processing load of extracting extracted image data that satisfies a condition from the plurality of still image data.

[0087] Furthermore, the extraction means may determine, based on the pre-stored first trained model M1, still image data in which at least one of the optic disc 84, the retinal blood vessels 85, and the macula 82 in the fundus image is determined to be clearly formed, as extracted image data that satisfies a predetermined condition. This allows for the acquisition of an image of a quality usable for fundus observation.

[0088] The control method is executed by an information processing device J having an imaging means and a communication means, and includes an imaging process in which, when an imaging instruction is input, the imaging means photographs the subject's fundus at predetermined time intervals; an extraction process in which, from a plurality of still image data of the fundus photographed at the time intervals, extracted image data in which the photographed fundus image satisfies predetermined conditions; and a communication process in which the extracted extracted image data that satisfies the conditions is transmitted to the outside.

[0089] Furthermore, the information processing program, upon input of a photographing instruction to an information processing device J having a photographing means and a communication means, causes the computer to execute the following steps: a photographing step of causing the photographing means to photograph the subject's fundus at predetermined time intervals; an extraction step of extracting extracted image data from multiple still image data of the fundus photographed at the time intervals, where the photographed fundus images satisfy predetermined conditions; and a communication step of transmitting the extracted image data that satisfy the extracted conditions to an external device. <Advantageous Effects of the Second Embodiment> The information processing device J can obtain multiple still image data of the subject's fundus, thereby increasing the likelihood of extracting high-quality still image data from the multiple still image data. Conventionally, when a user operates a mobile information processing terminal 150 to photograph multiple still images, the user must repeatedly input photographing instructions. Each time the user inputs a photographing instruction into the mobile information processing terminal 150, the relative positions of the camera 44, the objective lens 92, and the subject's eye may become misaligned. In contrast, with the information processing device J, the user only needs to input a single instruction to photograph the image into the portable information processing terminal 1, which then acquires multiple still images of the fundus. Therefore, the user can concentrate on adjusting the positional relationship between the camera 44 and the objective lens 92 during photographing, and can acquire a group of high-quality still image data.

[0090] Furthermore, the information processing device J can extract clear extracted image data free from camera shake and the like from a plurality of still image data, thereby eliminating the influence of camera shake.

[0091] On the other hand, it is also conceivable that the mobile information processing terminal 1 transmits the video data to an external server 2, and the server 2 performs processing to extract extracted image data that meets conditions from a plurality of still image data. However, in developing countries or remote locations, the communication environment of the network 100 may not be good. If the communication environment of the network 100 is not good, the mobile information processing terminal 1 may not be able to transmit the video data to the server 2. In response to this, the mobile information processing terminal 1 transmits extracted image data extracted from the video data to the external server 2. Because the mobile information processing terminal 1 transmits extracted image data, which has a smaller data capacity than the video data, to the server 2, there is a high possibility that the data can be transmitted even in an unfavorable communication environment.

[0092] (Variation 1 of the Second Embodiment) The portable information processing terminal 1 may notify the remaining time of shooting during video shooting. When the imaging unit 51 starts shooting in the examination mode, the output unit 16 notifies the remaining time required for shooting. For example, the control unit 31 outputs information about the remaining time (= shooting time - elapsed time since the start of shooting) as audio to the speaker 42. According to Variation 1, when the subject is notified of the remaining time of shooting, the subject tries to keep the eyeball 70 and eyelids still for the remaining time. This increases the possibility of obtaining a high-quality still image.

[0093] (Variation 2 of the Second Embodiment) The portable information processing terminal 1 may end video shooting when the shooting time has reached the shooting time or when extracted image data that satisfies the conditions has been extracted during the shooting time. In the examination mode, the control unit 31 causes the imaging unit 51 to end shooting when the elapsed time since the start of shooting has reached the shooting time or when extracted image data that satisfies the conditions has been extracted by the extraction unit 33. Variation 2 reduces the processing load of extracting extracted image data that satisfies the conditions from multiple still image data. Because the process of extracting images of a quality suitable for fundus observation from multiple still images is performed by the portable information processing terminal 1, which has lower processing power than the server 2, the effect of reducing the processing load is important for the portable information processing terminal 1.

[0094] Third Embodiment An information processing device J according to a third embodiment will be described. In the description of this embodiment, the same configuration as in the other embodiments and modifications will not be described.

[0095] Generally, the pupil diameter d in a dilated state in a dark place is approximately 4 mm to 8 mm. Also, the pupil diameter d in a constricted state in a bright place is approximately 2 mm to 4 mm. If the pupil diameter d becomes too small due to constriction, it becomes difficult to obtain a clear fundus image.

[0096] The information processing device J adjusts the amount of light emitted by the light emitting unit 61 so that the pupil diameter d does not become too small due to miosis.

[0097] The CPU 11 in the mobile information processing terminal 1 is provided with a dimming unit 32 implemented by a program read from the storage unit 18, as well as a pupil diameter calculation unit 34, a disease estimation unit 35, and an estimation result output unit 36 ​​(shown by the two-dot chain line in FIG. 6). The storage unit 18 stores a second trained model M2 used by the disease estimation unit 35.

[0098] The light adjustment unit 32 adjusts the amount of light emitted by the light-emitting unit 61 directed toward the subject's eye so that the pupil diameter d of the subject does not become smaller than a predetermined diameter. More specifically, when the pupil diameter d of the subject calculated by the pupil diameter calculation unit 34 based on the still image data is smaller than the predetermined diameter, the light adjustment unit 32 adjusts the amount of light emitted by the light-emitting unit 61 to reduce the amount of light emitted by a predetermined amount. For example, if the predetermined diameter is 3 mm and the predetermined amount is ¼ of the current amount of light emitted, when the eye contracts upon receiving light with an amount of light emitted by the light-emitting unit 61 of 50 lm and the pupil diameter d becomes smaller than 3 mm, the light adjustment unit 32 reduces the amount of light emitted by ¼ to 37.5 lm. The predetermined diameter is not limited to 3 mm and may be other values, such as 2 mm or 4 mm, and is adjusted empirically or experimentally so as to obtain clearer still image data. Furthermore, the predetermined amount may not only be an amount expressed as a percentage, but also a fixed amount such as 10 [lm], and is adjusted empirically or experimentally so that the amount of light emitted can be adjusted in a shorter time.

[0099] The pupil diameter calculation unit 34 calculates the pupil diameter d based on the still image data. More specifically, the pupil diameter calculation unit 34 acquires a contour image of the pupil 79 based on the still image data using a technique such as edge detection. The pupil diameter calculation unit 34 also reads out a reference contour image and a reference contour diameter corresponding to the reference contour image stored in the storage unit 18. The pupil diameter calculation unit 34 derives a ratio between the acquired contour image and the reference contour image by a ratio derivation process, and calculates the pupil diameter d by multiplying the reference contour diameter by the ratio. This method of calculating the pupil diameter d is one example, and the pupil diameter calculation unit 34 may calculate the pupil diameter d based on the still image data using other methods.

[0100] The disease estimation unit 35 estimates the fundus disease of the subject by inputting the extracted image data into the second trained model M2 generated by training the second training image showing the fundus disease of the subject. More specifically, the disease estimation unit 35 obtains an estimate of the fundus disease by inputting the extracted image data into the second trained model M2.

[0101] The second training image is, for example, a fundus image in which a doctor has diagnosed or suspected glaucoma, and is a two-dimensional fundus image taken with or without mydriasis. Preferably, the second training image is a fundus image in which the optic disc 84 is visible.

[0102] The estimation result output unit 36 ​​transmits the extracted image data and the estimated value acquired by the disease estimation unit 35 to the server 2. The server 2 transmits the extracted image data and the estimated value to the diagnostic terminal 3, which displays the estimation result based on the extracted image data and the estimated value on the screen. The estimation result can be divided into multiple stages and displayed in stages to make it easier for doctors and patients to understand.

[0103] Next, the light amount adjustment process will be described.

[0104] FIG. 11 is a flowchart illustrating an example of the light amount adjustment process.

[0105] The user attaches the portable information processing terminal 1 to the adapter 90 and positions the objective lens 92 directly against the subject's eye.

[0106] In response to a user's instruction, the light emitting unit 61 emits light and the imaging unit 51 starts capturing an image (S21). Still image data is acquired based on the image captured by the imaging unit 51 (S22). Subsequently, the pupil diameter d is calculated based on the still image data (S23).

[0107] If the pupil diameter d is smaller than the predetermined diameter (S24: YES), the amount of light emitted by the light-emitting unit 61 is reduced by a predetermined amount (S25), and the process returns to S22. On the other hand, if the pupil diameter d is not smaller than the predetermined diameter (S24: NO), the process proceeds to S26. The acquisition of still image data, the calculation of the pupil diameter d based on the still image data, and the adjustment of the amount of light emitted (S22-S25) are repeated until the pupil diameter d becomes larger than the predetermined diameter.

[0108] If the photographing time has not elapsed (S26: NO), the process returns to S22, and if the photographing time has elapsed (S26: YES), the process proceeds to S27.

[0109] The still image data is input to the first trained model M1, and if the output score is higher than the threshold, it is determined that the fundus is clearly visible in the still image data (S27: YES), and the still image data is extracted as extracted image data (S28). On the other hand, if it is determined that the fundus is not clearly visible in the still image data (S27: NO), the process returns to S22.

[0110] The extracted image data is input to the second trained model M2, and the fundus disease of the subject is estimated using the estimated value of the fundus disease (S28), and the estimation result is displayed on the screen of the diagnostic terminal 3 (S29).

[0111] That is, the information processing device J has a dimming unit 32 that adjusts the amount of light emitted by the light-emitting unit 61 directed at the subject's eye, a still image data acquisition unit P2 that acquires multiple still image data based on a captured image taken of the subject's fundus exposed to the light emitted from the light-emitting unit 61, and an extraction unit 33 that extracts extracted image data showing the fundus from each of the multiple still image data.

[0112] The light adjusting unit 32 adjusts the amount of light emitted by the light emitting unit 61 directed toward the subject's eye so that the pupil diameter d of the subject does not become smaller than a predetermined diameter.

[0113] The pupil diameter calculation unit 34 calculates the pupil diameter d based on the still image data, and the light adjustment unit 32 adjusts the amount of light emitted by the light emitting unit 61 to reduce by a predetermined amount when the pupil diameter d is smaller than a predetermined diameter.

[0114] The extraction unit 33 obtains a score by inputting still image data acquired by the still image data acquisition unit P2 into a first trained model M1 generated by learning a first training image that shows at least one of the subject's optic disc 84, retinal blood vessels 85, and macular region 82, and extracts extracted image data based on the score.

[0115] The disease estimation unit 35 estimates the subject's fundus disease by inputting the extracted image data into a second trained model M2 generated by learning a second training image that shows the subject's fundus disease.

[0116] The photographing instruction unit P1 starts photographing the fundus of the subject, and then ends the photographing after the photographing time has elapsed.

[0117] The control method involves a dimming unit 32 adjusting the amount of light emitted by a light-emitting unit 61 directed at the subject's eye so that the pupil diameter d of the subject does not become smaller than a predetermined diameter, a still image data acquisition unit P2 acquiring a plurality of still image data based on an image captured from the subject's fundus receiving the light emitted from the light-emitting unit 61, and an extraction unit 33 extracting extracted image data showing the fundus for each of the plurality of still image data.

[0118] The information processing program causes the computer to execute the following: code for a dimming unit 32 that adjusts the amount of light emitted by a light-emitting unit 61 directed at the subject's eye so that the pupil diameter d of the subject does not become smaller than a predetermined diameter; code for a still image data acquisition unit P2 that acquires multiple still image data based on an image captured from the subject's fundus exposed to light emitted from the light-emitting unit 61; and code for an extraction unit 33 that extracts extracted image data showing the fundus for each of the multiple still image data.

[0119] <Advantageous effects of the third embodiment> As a result, the information processing device J, by having the light adjustment unit 32 instruct the light emitting unit 61 to emit light at an amount corresponding to the pupil diameter d, can suppress eye position shifts caused by an aversion reflex when exposed to a large amount of light, as well as miosis and eyelid closure of the pupil 79, thereby reducing discomfort to the subject.In addition, by preventing the pupil diameter d from becoming smaller than a predetermined diameter, it is possible to obtain extracted image data based on clearer still image data suitable for fundus observation.

[0120] Furthermore, according to the information processing device J, by using the first trained model M1, it is possible to extract clear extracted image data free from camera shake and the like from multiple still image data, thereby eliminating the effects of camera shake.

[0121] In addition, according to the information processing device J, by using the second trained model M2, it is possible to estimate the subject's fundus disease, and to present the user with extracted image data as well as the estimated results of the fundus disease, thereby assisting in the diagnosis of the fundus disease.

[0122] According to the information processing device J, fundus images suitable for fundus observation can be easily obtained.

[0123] (Fourth embodiment) An information processing device J in the fourth embodiment will be described. Fig. 12 is an explanatory diagram for explaining an optical system for photographing a fundus in the information processing device J. Fig. 13 is a diagram showing design values ​​of the optical system. In the description of this embodiment, descriptions of the same configuration as in other embodiments and modified examples will be omitted.

[0124] The information processing device J has multiple optical systems including positive refractive powers, arranged in order from the eye side. More specifically, the information processing device J has, in the optical axis direction, an objective lens 92 and a subsequent lens group Lg subsequent to the objective lens 92. The subsequent lens group Lg has an exit lens Le through which light is emitted from the adapter 90 and a camera lens 52 onto which the light emitted from the exit lens Le is incident. The objective lens 92 and the exit lens Le are provided in a lens barrel Br of the adapter 90.

[0125] The light-emitting unit 61 may be an illumination device such as an LED 62 disposed near the aperture stop at a distance perpendicular to the optical axis extending from the fundus image to the image sensor 53, or may be configured such that a reflecting member is disposed at the same position, the illumination device projects light onto the reflecting member, and the illumination light is guided into the fundus. In the example of Fig. 12, the light-emitting unit 61 is provided inside the lens barrel Br, but this is not particularly limited.

[0126] The rear lens group Lg has an imaging function, and by moving the entire rear lens group Lg or a part of it in the forward and backward directions on the optical axis, it is possible to adjust the focus from nearsightedness to farsightedness.

[0127] The subsequent lens group Lg in the optical system magnifies or reduces the image formed by the objective lens 92 and forms the image on the image sensor 53 .

[0128] The objective lens ray height H0 at the objective lens 92 is calculated by the following formula (3) using the eyeball-objective lens distance WD, which is the distance between the eyeball 70 and the objective lens 92 required to configure the adapter 90, and the angle of view θ that can accommodate the optic disc 84 and the macula 82: H0=WD×tan θ, (tan θ=H0 / WD) (3)

[0129] Next, the objective lens focal length f1, which is the focal length of the objective lens 92, is calculated by the following formula (4). Fno in formula (4) is the F-number of the objective lens 92. f1=2×Fno×WD×tan θ, {Fno=f1 / (2×H0)} (4)

[0130] Furthermore, the object height He on the retina including the optic disc 84 and the macula 82 is calculated by the formula (5) based on the eyeball focal length fe, which is the focal length of the eyeball 70, and the angle of view θ. It is generally known that the eyeball focal length fe of the eyeball 70 in the normal viewing state is approximately 17 mm. He=fe×tan θ (5)

[0131] The image of the retina 73 is projected onto an image forming point in the lens barrel Br in the adapter 90. The image height H1 in the adapter, which is the height of the image projected in the lens barrel Br in the adapter 90, is calculated by the following formula (6): H1=He×f1 / fe (6)

[0132] The magnification M of the subsequent lens group Lg is calculated by the following equation (7) based on the image sensor image height H2, which is the image height of the image projected onto the image sensor 53 provided in the camera 44, and the image height H1 within the adapter: |M|=H2 / H1 (7)

[0133] The eyeball objective lens magnification M0, which is the combined magnification of the eyeball 70 and the objective lens 92, is calculated by the following formula (8) based on the eyeball focal length fe and the objective lens focal length f1: M0=f1 / fe (8)

[0134] The adapter 90, to which the portable information processing terminal 1 is attached and which is held by the user's hand so as to face the subject's eye, is made compact and lightweight to prevent fatigue in the user's hand. More specifically, the adapter 90 has an inner diameter of the lens barrel Br of less than 100 mm, preferably less than 60 mm. Furthermore, when the diagonal length of the image sensor 53 is 8.5 mm and the short side length is 5.1 mm, the image sensor image height H2 is preferably in the range of 0.255 mm to 1.53 mm, which is 10-60% of the half angle of view of the short side length.

[0135] (A) To set the inner diameter of the lens barrel Br to less than 100 mm and obtain an image sensor image height H2 of 0.255 mm, if the eyeball-objective lens distance WD is 35 mm, the angle of view θ is 35°, the F-number is 1.2, and the eyeball focal length fe is 17 mm, then the objective lens ray height H0 is 24.51 mm according to formula (3), the objective lens focal length f1 is 58.82 mm according to formula (4), the object height on the retina He is −11.90 mm according to formula (5), and the image height H1 inside the adapter is −41.184 mm according to formula (6). The subsequent lens group magnification M is |0.255 mm / −41.184 mm| = 0.006 according to formula (7). Furthermore, the eyeball objective lens magnification M0 is 58.82 mm / 17 mm = 3.460 according to formula (8).

[0136] (B) In order to set the inner diameter of the lens barrel Br to less than 60 mm and obtain an image sensor image height H2 of 0.255 mm, if the eyeball-to-objective lens distance WD is 25 mm, the angle of view θ is 35°, the F-number is 1.2, and the eyeball focal length fe is 17 mm, then the objective lens ray height H0 is 17.51 ​​mm according to formula (3), the objective lens focal length f1 is 42.01 mm according to formula (4), the object height on the retina He is −11.90 mm according to formula (5), and the image height H1 inside the adapter is −29.417 mm according to formula (6). The subsequent lens group magnification M for obtaining an image sensor image height H2 of 0.255 mm is |0.255 mm / −29.417 mm|=0.009 according to formula (7). Furthermore, the ocular objective lens magnification M0 is 42.01 mm / 17 mm = 2.471 according to formula (8).

[0137] (C) To obtain an image sensor image height H2 of 1.53 mm, if the eyeball-objective lens distance WD is 19 mm, the angle of view θ is 15°, the F-number is 2.0, and the eyeball focal length fe is 17 mm, then the objective lens ray height H0 is 5.09 mm according to formula (3), the objective lens focal length f1 is 20.36 mm according to formula (4), the object height on the retina He is −4.56 mm according to formula (5), and the image height H1 inside the adapter is −5.457 mm according to formula (6). The subsequent lens group magnification M for obtaining the image sensor image height H2 of 1.53 mm is |1.53 mm / −5.457 mm| = 0.280 according to formula (7). Furthermore, the eyeball objective lens magnification M0 is 20.36 mm / 17 mm = 1.198 according to formula (8).

[0138] (D) To obtain an image sensor image height H2 of 1.53 mm, if the eyeball-objective lens distance WD is 20 mm, the angle of view θ is 15°, the F-number is 1.2, and the eyeball focal length fe is 17 mm, then the objective lens ray height H0 is 5.36 mm according to formula (3), the objective lens focal length f1 is 12.86 mm according to formula (4), the object height on the retina He is −4.56 mm according to formula (5), and the image height H1 inside the adapter is −3.446 mm according to formula (6). The subsequent lens group magnification M for obtaining the image sensor image height H2 of 1.53 mm is |1.53 mm / −3.446 mm| = 0.444 according to formula (7). Furthermore, the eyeball objective lens magnification M0 is 12.86 mm / 17 mm = 0.757 according to formula (8).

[0139] The rear lens group magnification M in the information processing device J has a lower limit of 0.006 as shown in (A) and an upper limit of 0.444 as shown in (C), and preferably has a lower limit of 0.009 as shown in (B) and an upper limit of 0.280 as shown in (D). Here, the calculation was performed assuming that the diagonal length of the image sensor 53 was 8.5 mm, but the condition for the rear lens group magnification M when the diagonal length of the image sensor 53 is A is as shown in formula (9), and preferably as shown in formula (10). 0.006<|M|×8.5 / A<0.444 (9) 0.009<|M|×8.5 / A<0.280 (10)

[0140] The ocular objective lens magnification M0 is calculated by M0 = f1 / fe. Since the lower limit of the objective lens focal length f1 is 12.86 mm, the upper limit is 58.82 mm, and the ocular focal length fe is 17 mm, the conditions for the ocular objective lens magnification M0 are as shown in formula (11), with the lower limit being 0.757 shown in (D) and the upper limit being 3.460 shown in (A). Preferably, the lower limit of the objective lens focal length f1 is 20.36 mm, and the upper limit is 42.01 mm, the conditions for the ocular objective lens magnification M0 are as shown in formula (12), with the lower limit being 1.198 shown in (C) and the upper limit being 2.471 shown in (A). 0.757<|M0|<3.460 (11) 1.198<|M0|<2.471 (12)

[0141] That is, the adapter 90 holds the portable information processing terminal 1, 150 so that the objective lens 92 and the camera 44 are positioned at a predetermined photographing position relative to the subject's eye.

[0142] Furthermore, the information processing device J has an optical system that satisfies the condition 0.006<|M|×A / 8.5<0.444, where M is the composite magnification of the subsequent lens group Lg following the objective lens 92 in the optical axis direction and A is the diagonal length of the image sensor 53 provided in the camera 44.

[0143] The information processing device J has an optical system that satisfies the condition 1.198<|M0|<2.471 in the optical axis direction, where M0 is the combined magnification of the eyeball 70 and the objective lens 92.

[0144] Furthermore, the information processing device J has an optical system that satisfies the condition 12.86 mm<objective lens focal length f1<58.82 mm.

[0145] <Advantageous Effects of the Fourth Embodiment> As a result, the information processing device J has a smaller diameter for the objective lens 92 and the subsequent lens group Lg, making it possible to make it compact and lightweight so as not to tire the user's hands due to its weight, and reducing the user's hand shake. Furthermore, the information processing device J can increase the focal depth of the optical system, making it easier to achieve focus and easily obtain high-quality images. Furthermore, the information processing device J reduces the amount of refraction of light rays on the lens surface, thereby reducing the occurrence of aberrations and making it easier to obtain sufficient imaging performance.

[0146] (Others) In the embodiment and modified example, an example in which the objective lens 92 is one lens has been described, but the number of objective lenses is not limited to one and may be multiple.

[0147] Furthermore, in the embodiment and modified examples, an example has been described in which the information processing device J is used for fundus observation for glaucoma, but the information processing device J may be used for fundus diseases other than glaucoma, such as optic disc cupping enlargement, retinal optic nerve fiber layer defects, diabetes, side effects of anticancer drugs, etc. In this case, the first trained model M1 and the second trained model M2 are trained based on first training images and second training images corresponding to the fundus disease.

[0148] In addition, in the embodiment and variant examples, an example has been described in which the disease estimation unit 35 inputs extracted image data into the second trained model M2 to estimate a fundus disease, but this is not limited to this, and fundus disease may also be estimated by inputting still image data.

[0149] Furthermore, for example, the above-described series of processes can be executed by hardware or software. In other words, the above-described functional configuration is merely an example and is not particularly limited. That is, it is sufficient for the information processing device J to have the function of executing the above-described series of processes as a whole, and the type of functional block used to realize this function is not particularly limited to the above example. Furthermore, the location of the functional block is not particularly limited and may be arbitrary. For example, the functional block of the mobile information processing terminal 150 may be transferred to another device, etc. Conversely, the functional block of another device may be transferred to the server 2, etc. Furthermore, one functional block may be configured as a single piece of hardware, a single piece of software, or a combination thereof.

[0150] When the series of processes is executed by software, the programs constituting the software are installed into a computer or the like from the network 100 or a recording medium. The computer may be a computer incorporated in dedicated hardware. Alternatively, the computer may be a computer capable of executing various functions by installing various programs, such as the server 2 or a general-purpose personal computer.

[0151] The recording medium containing such a program may be configured as a removable medium 21 (not shown) that is distributed separately from the device main body in order to provide the program to users, etc., or may be configured as a recording medium that is pre-installed in the device main body and provided to users, etc. Since the program can be distributed via the network 100, the recording medium may be installed in or accessible to a computer that is connected to or can be connected to the network 100.

[0152] In this specification, the steps describing the program recorded on the recording medium include not only processes that are performed in chronological order, but also processes that are not necessarily performed in chronological order but are performed in parallel or individually. Also, in this specification, the term "system" means an overall device composed of multiple devices or multiple means, etc.

[0153] Although one embodiment of the present invention has been described above, the present invention is not limited to the above-described embodiment, and modifications, improvements, etc. within the scope of achieving the object of the present invention are included in the present invention.

[0154] <Advantageous Effects of the Present Invention> The present invention provides the following advantageous effects.

[0155] (A) With the information processing device J, the movement of the eyeball 70 and eyelids due to the reflex reaction when light is emitted is suppressed, so that the doctor does not have to spend as much time adjusting the positional relationship between the camera 44 and the objective lens 92 when taking an image, the effects of camera shake can be eliminated, and high-quality fundus images can be easily obtained.

[0156] (B) According to the information processing device J, it is possible to reduce the glare felt by the subject, and to prevent the subject from feeling uncomfortable when photographing the fundus.

[0157] (C) Since the portable information processing terminal 1 transmits the extracted image data, which has a smaller data volume than the video data, to the server 2, there is a high possibility that the data can be transmitted even in an unfavorable communication environment.

[0158] (D) By instructing the light-emitting unit 61 to emit light at an amount that corresponds to the pupil diameter d, the light adjustment unit 32 can suppress eye position shifts caused by an aversion reflex when exposed to a large amount of light, as well as pupil constriction and eyelid closure of the pupil 79. This reduces discomfort to the subject, and by preventing the pupil diameter d from becoming smaller than a predetermined diameter, it is possible to obtain extracted image data based on clearer still image data that is suitable for fundus observation.

[0159] (E) According to the information processing device J, by using the first trained model M1, it is possible to extract clear extracted image data free from camera shake and the like from multiple still image data, thereby eliminating the effects of camera shake.

[0160] (F) According to the information processing device J, by using the second trained model M2, it is possible to estimate the subject's fundus disease, and to present the user with extracted image data as well as the estimated results of the fundus disease, thereby assisting in the diagnosis of the fundus disease.

[0161] (G) The information processing device J has a small diameter objective lens 92 and a subsequent lens group Lg, making it possible to make it compact and lightweight so that the weight does not tire the user's hands, and reducing the user's hand shake.

[0162] (H) The information processing device J can increase the focal depth of the optical system, making it easier to focus and easily obtain high-quality images.

[0163] (I) In the information processing device J, the amount of refraction of light rays on the lens surface is suppressed, so the occurrence of aberrations is suppressed and it is easy to obtain sufficient imaging performance.

[0164] According to the present invention, it is possible to provide an information processing device J, a control method, and an information processing program that can easily acquire a fundus image suitable for fundus observation.

[0165] 1...portable information processing terminal, 2...server, 3...diagnosis terminal, 4...subject terminal, 11...CPU, 18...storage unit, 31...control unit, 32...light adjustment unit, 33...extraction unit, 34...pupil diameter calculation unit, 35...disease estimation unit, 36...estimation result output unit, 51...imaging unit, 52...camera lens, 53...image sensor, 61...light emitting unit, 90...adapter, 92...objective lens, J...information processing device, Le...output lens, Lg...subsequent lens group, M1...trained model, M2...trained model, P1...imaging instruction unit, P2...still image data acquisition unit

Claims

1. An information processing device having: a light control unit that adjusts the amount of light emitted by a light-emitting unit directed at a subject's eye; a still image data acquisition unit that acquires multiple still image data based on an image captured from the subject's fundus that has received the light emitted from the light-emitting unit; and an extraction unit that extracts extracted image data showing the fundus from each of the multiple still image data.

2. An information processing device as described in claim 1, wherein the light adjusting unit adjusts the amount of light emitted by the light emitting unit directed at the subject's eye so that the pupil diameter of the subject does not become smaller than a predetermined diameter.

3. An information processing device as described in claim 2, further comprising a pupil diameter calculation unit that calculates the pupil diameter based on the still image data, and wherein the light adjustment unit adjusts the amount of light emitted by the light-emitting unit to reduce the amount by a predetermined amount when the pupil diameter is smaller than a predetermined diameter.

4. The information processing device described in claim 1, wherein the extraction unit obtains a score by inputting the still image data acquired by the still image data acquisition unit into a first trained model generated by learning a first training image that shows at least one of the subject's optic disc, retinal blood vessels, and macular region, and extracts the extracted image data based on the score.

5. An information processing device as described in claim 1, further comprising a disease estimation unit, which estimates the fundus disease of the subject by inputting the extracted image data into a second trained model generated by training a second training image showing the fundus disease of the subject.

6. An information processing device according to claim 1, further comprising an imaging instruction unit, wherein the imaging instruction unit starts imaging the fundus of the subject and then ends the imaging after an imaging time has elapsed.

7. An information processing device according to claim 1, further comprising an adapter having an objective lens provided on the subject side, and a portable information processing terminal having a camera, wherein the adapter holds the portable information processing terminal so that the objective lens and the camera are positioned at a predetermined photographing position relative to the subject's eye.

8. The information processing device according to claim 7, having an optical system that satisfies the condition 0.006<|M|×A / 8.5<0.444, where M is the composite magnification of the subsequent lens group following the objective lens in the optical axis direction and A is the diagonal length of the image sensor provided in the camera.

9. The information processing device according to claim 7, having an optical system that satisfies the condition 1.198<|M0|<2.471, where M0 is the combined magnification of the eyeball and the objective lens in the optical axis direction.

10. The information processing device according to claim 7, having an optical system that satisfies the condition 12.86 mm<f1<58.82 mm, where f1 is the focal length of the objective lens.

11. A control method, in which a light adjustment unit adjusts the amount of light emitted by a light emitting unit directed at a subject's eye, a still image data acquisition unit acquires a plurality of still image data based on an image captured from the fundus of the subject receiving the light emitted from the light emitting unit, and an extraction unit extracts extracted image data showing the fundus for each of the plurality of still image data.

12. An information processing program that causes a computer to execute the following: a code for a light adjusting unit that adjusts the amount of light emitted by a light emitting unit directed at the subject's eye; a code for a still image data acquiring unit that acquires a plurality of still image data based on an image captured from the fundus of the subject that has received the light emitted from the light emitting unit; and a code for an extracting unit that extracts extracted image data showing the fundus from each of the plurality of still image data.