Image processing device, image processing method and program

Abstract

An image processing device 1X comprises, among other things, a medication information acquisition means 14X, a photographing timing determination means 15X, and a pupil information generation means 17X. The medication information acquisition means 14X acquires medication information pertaining to a medicine used for a subject. On the basis of the medication information, the photographing timing determination means 15X determines a photographing timing at which the subject is to be photographed by the photographing means. The pupil information generation means 17X generates pupil information pertaining to a pupil of the subject on the basis of an image generated by the photographing means at the photographing timing.

Description

Image processing device, image processing method, and storage medium

[0001] The present disclosure relates to the technical fields of an image processing device, an image processing method, and a storage medium that perform processing related to determining medication effects using images.

[0002] There are known systems that measure the pupils of a subject based on a photographed facial image of the subject. For example, Patent Literature 1 discloses a system that detects the area of ​​the subject's pupils based on video signals from two cameras and analyzes the pupil area, pupil diameter, and pupil position.

[0003] International Publication WO2002 / 003853

[0004] Measuring the effectiveness of a drug requires measuring blood concentrations, etc. However, because the appropriate dosage of a drug varies from person to person, it would be desirable for the subject to be able to easily measure the effectiveness of the drug themselves.

[0005] In view of the above-mentioned problems, one of the objectives of the present disclosure is to provide an image processing device, an image processing method, and a storage medium that can suitably generate information regarding the effectiveness of a subject's medication from an image of the subject.

[0006] One aspect of the image processing device is an image processing device having: a medication information acquisition means for acquiring medication information regarding medication used by a subject; an imaging timing determination means for determining the timing of imaging the subject by an imaging means based on the medication information; and a pupil information generation means for generating pupil information regarding the pupil of the subject based on an image generated by the imaging means at the imaging timing.

[0007] One aspect of the image processing method is an image processing method in which a computer acquires medication information regarding medication administered to a subject, determines a timing for photographing the subject by an imaging means based on the medication information, and generates pupil information regarding the pupil of the subject based on an image generated by the imaging means at the photographing timing. Note that the "computer" includes any electronic device (which may be a processor included in an electronic device) and may be composed of multiple electronic devices.

[0008] One aspect of the storage medium is a storage medium that stores a program that causes a computer to execute the following processes: acquire medication information regarding medication used by a subject; determine the timing of photographing the subject by an imaging means based on the medication information; and generate pupil information regarding the pupil of the subject based on an image generated by the imaging means at the photographing timing.

[0009] It is possible to suitably generate information regarding the effectiveness of the subject's medication from an image of the subject.

[0010] 1 shows a schematic configuration of a medication effect assessment system according to a first embodiment; FIG. 2 shows a state when determining medication effect when the medication effect assessment system is a single terminal device; FIG. 3 shows an example of the hardware configuration of an image processing device common to all embodiments; FIG. 4 shows an example of functional blocks of an image processing device related to medication effect assessment processing in the first embodiment; FIG. 5 shows an example of a camera image capturing a subject's face; FIG. 6 is a graph showing an example of a change in the subject's pupil size after the subject has taken a certain drug; FIG. 7 shows an example of a flowchart related to medication effect assessment processing executed by an image processing device; FIG. 8 shows a schematic configuration of a medication effect assessment system according to a second embodiment; FIG. 9 is a block diagram of an image processing device according to a third embodiment; FIG. 10 is an example of a flowchart executed by an image processing device according to the third embodiment.

[0011] Hereinafter, embodiments of an image processing device, an image processing method, and a storage medium will be described with reference to the drawings.

[0012] First Embodiment (1) System Configuration FIG. 1 shows a schematic configuration of a medication effect assessment system 100 according to the first embodiment. The medication effect assessment system 100 is a system that simply assesses the effectiveness of a medication taken by a subject 6 (also referred to as "medication effect") based on an image of the subject 6's face captured by a visible light camera. The system mainly includes an image processing device 1, an input device 2, an output device 3, a storage device 4, and a measurement device 5 including a camera (photography device) 51. The medication effect assessment system 100 may be used by the subject 6 for self-care purposes or at the direction or request of a medical professional. The medication effect assessment system 100 may be used, for example, to confirm medication effect, adjust medication dosage and medication intervals, etc. Hereinafter, "medication" is not limited to oral ingestion of a medication by the subject 6, but also includes cutaneous ingestion.

[0013] The image processing device 1 measures the pupil of the subject 6 after medication based on facial images of the subject 6 generated by the camera 51 (including videos that are a predetermined number of images obtained in time series; the same applies hereinafter), and based on the pupil measurement results, determines the effectiveness of the medication and notifies information related to the determination results of the medication effectiveness. The image processing device 1 communicates data with the input device 2, output device 3, storage device 4, and measurement device 5 via a communication network or by direct wireless or wired communication.

[0014] The input device 2 is an interface that accepts user input (manual input). The user who inputs information using the input device 2 may be the subject 6 himself / herself, or a person managing or supervising the subject 6. The input device 2 may be, for example, various user input interfaces such as a touch panel, buttons, a keyboard, a mouse, or a voice input device. The input device 2 supplies an input signal generated based on the user input to the image processing device 1.

[0015] The output device 3 outputs predetermined information based on an output signal supplied from the image processing device 1. In this case, the output signal includes at least one of a display signal and an audio signal. The output device 3 displays information based on the display signal supplied from the image processing device 1, and outputs audio information based on the audio signal supplied from the image processing device 1. The output device 3 includes at least one of a display device such as a display or a projector, and an audio output device such as a speaker, for example.

[0016] The storage device 4 is a memory that stores various information necessary for determining medication effects, etc. The storage device 4 may be an external storage device such as a hard disk connected to or built into the image processing device 1, or may be a storage medium such as a flash memory. The storage device 4 may also be a server device that performs data communication with the image processing device 1. The storage device 4 may also be composed of multiple devices.

[0017] The measurement device 5 is one or more sensors including a camera 51, which is a visible light camera. For example, the measurement device 5 may include an illuminance sensor for detecting changes in the amount of external light in the environment in which the camera image of the subject 6 is captured. The measurement device 5 supplies signals measured by each sensor to the image processing device 1. Hereinafter, the image generated by the camera 51 will also be referred to as a "camera image." The camera 51 is an example of an "imaging means."

[0018] 1 is an example, and various modifications may be made to the configuration. For example, the image processing device 1, the input device 2, the output device 3, the storage device 4, and the measurement device 5 may be implemented by a single terminal device such as a smartphone or a tablet terminal.

[0019] 2 shows a state in which a camera image of a subject 6 is captured when the medication effect determination system 100 is a single terminal device (e.g., a smartphone). As shown in FIG. 2, the subject 6 holds the medication effect determination system 100, which is a terminal device, and adjusts the orientation of the terminal device so that the subject's face is included in the capturing range of the camera 51. The medication effect determination system 100 may be fixed to a tripod or the like. Then, in the state shown in FIG. 2, the subject 6 follows instructions (guidance) output by the medication effect determination system 100 to cause the medication effect determination system 100 to capture a camera image of the subject 6 required for determining the medication effect.

[0020] (2) Hardware Configuration Fig. 3 shows the hardware configuration of the image processing device 1. The image processing device 1 includes, as hardware, a processor 11, a memory 12, and an interface 13. The processor 11, the memory 12, and the interface 13 are connected via a data bus 90.

[0021] The processor 11 executes programs stored in the memory 12 to function as a controller (arithmetic unit) that performs overall control of the image processing device 1. The processor 11 is, for example, a processor such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a TPU (Tensor Processing Unit). The processor 11 may be composed of multiple processors. The processor 11 is an example of a computer.

[0022] The memory 12 is composed of various types of volatile and non-volatile memory, such as RAM (Random Access Memory), ROM (Read Only Memory), and flash memory. The memory 12 also stores programs for executing processes performed by the image processing device 1. Some of the information stored in the memory 12 may be stored in one or more external storage devices capable of communicating with the image processing device 1, or in a storage medium that is detachable from the image processing device 1. The memory 12 may also function as at least a part of the storage device 4.

[0023] The interface 13 is an interface for electrically connecting the image processing device 1 to other devices. These interfaces may be wireless interfaces such as network adapters for wirelessly transmitting and receiving data to and from other devices, or may be hardware interfaces for connecting to other devices via cables or the like.

[0024] The hardware configuration of the image processing device 1 is not limited to the configuration shown in Fig. 2. For example, the image processing device 1 may include at least one of an input device 2, an output device 3, a storage device 4, and a measurement device 5.

[0025] (3) Overview of Medication Effect Assessment Process Next, the medication effect assessment process, which is a process related to the assessment of medication effect, will be described. In summary, the image processing device 1 determines the timing of capturing a camera image based on medication information related to the medication of the subject 6, and measures the pupil of the subject 6 based on the camera image captured at the determined timing. The image processing device 1 then assesses the medication effect and notifies information related to the assessment result based on the measured change in pupil size of the subject 6, etc. This enables the image processing device 1 to accurately assess the medication effect based on the camera image, regardless of the type and amount of medication administered, and to notify the subject 6, etc. of a highly accurate assessment result of the medication effect.

[0026] Generally, the administration of a drug causes changes in pupil size depending on the drug's effectiveness. For example, when an autonomic nervous system drug (e.g., Lexotanil) is administered, pupil dilation (dilation) occurs due to sympathetic nervous activity. Furthermore, when a parasympathomimetic drug (e.g., pilocarpine, physostigmine) is administered, pupil constriction occurs. Furthermore, when a parasympatholytic drug (e.g., homatropine, tropicamide) is administered, pupil dilation occurs.

[0027] Taking the above into consideration, the image processing device 1 determines the medication effect based on the change in pupil size. Furthermore, since the timing at which the medication effect peaks varies depending on the medication content, the image processing device 1 determines the timing of capturing a camera image to identify the change in pupil size based on the medication information. In this way, the image processing device 1 determines the medication effect with high accuracy.

[0028] Fig. 4 shows an example of functional blocks of the image processing device 1 related to the medication effect assessment process. Functionally, the processor 11 of the image processing device 1 has a medication information acquisition unit 14, an imaging timing determination unit 15, a first notification unit 16, a pupil information generation unit 17, a medication effect assessment unit 18, and a second notification unit 19. Note that in Fig. 4, blocks where data is exchanged are connected by solid lines, but the combination of blocks where data is exchanged is not limited to that shown. The same applies to other functional block diagrams described below.

[0029] When medication is administered to the subject 6 (i.e., the subject 6 uses a medicine), the medication information acquisition unit 14 acquires medication information indicating at least the details of the medication administered to the subject 6 (i.e., the type and amount of the medicine used) and the time of administration. Then, the medication information acquisition unit 14 supplies the acquired medication information to the imaging timing determination unit 15.

[0030] Here, specific examples of acquisition of medication information will be described. In a first example, the medication information acquisition unit 14 generates medication information based on input by the subject 6 to the input device 2. For example, the medication information acquisition unit 14 displays an input screen for inputting medication content and medication time on the output device 3, and acquires input information indicating the input on the input screen from the input device 2, thereby generating the medication information. Note that if setting information for medication content and medication time is registered in advance in the storage device 4, memory 12, or the like, the medication information acquisition unit 14 may generate medication information based on the setting information.

[0031] In a second example, the medication information acquisition unit 14 acquires an image of the packaging (e.g., a PTP (Press Through Pack) sheet) that contained the medication used by the subject 6, and analyzes the image using any image recognition technology to recognize the medication content. The medication information acquisition unit 14 then recognizes the generation date and time indicated in the metadata of the image as the medication time, and generates medication information indicating the recognized medication content and medication time. In this example, the subject 6 or the subject's manager takes an image of the used medication case (package) using the camera 51 or another camera after the subject 6 has used the medication, and performs an input to designate the captured image as the image to be analyzed by the medication information acquisition unit 14.

[0032] In the third example, when an application for managing medication to the subject 6 is installed in the image processing device 1, the medication information acquisition unit 14 acquires medication information from the application.

[0033] The imaging timing determination unit 15 determines the imaging timing of the subject 6 based on the medication information provided by the medication information acquisition unit 14, and when the determined imaging timing is reached, supplies an instruction signal to start imaging to the first notification unit 16. In this case, the imaging timing determination unit 15 determines each pupil measurement timing required to determine the medication effect as the imaging timing. Note that administration of medication causes a change in pupil size depending on the effectiveness of the medication, and the effectiveness of the medication varies depending on the type of medication used, etc. Taking the above into consideration, the imaging timing determination unit 15 recognizes each measurement timing required to identify a change in pupil size based on the medication information, and determines each measurement timing as the imaging timing. Then, when the determined imaging timing is reached, the imaging timing determination unit 15 supplies an instruction signal to start imaging to the first notification unit 16. The method by which the imaging timing determination unit 15 determines the imaging timing will be described later.

[0034] When the first notification unit 16 receives an instruction signal to start imaging from the imaging timing determination unit 15, it notifies the subject 6 via the output device 3 that it is time to image. In this case, the first notification unit 16 notifies the subject 6 via the output device 3 that it is time to image by supplying an output signal including at least one of an audio signal and a display signal to the output device 3 via the interface 13. This causes the subject 6 to adjust the camera 51 so that the subject's face is within the imaging range of the camera 51, and perform an operation to start imaging by the camera 51, etc.

[0035] Preferably, the first notification unit 16 instructs the subject 6 to direct his / her gaze (i.e., focus) at a location a predetermined distance away in order to stabilize the subject 6's focus. The above-mentioned "predetermined distance" is, for example, a distance of 6 m or more at which the focal distance is essentially infinity. Note that the location a predetermined distance away is not limited to a location 6 m or more away, and may be any location at which the distance to the subject 6 does not change.

[0036] In another preferred example, the first notification unit 16 may instruct the subject 6 to adjust the distance (shooting distance) between the subject 6 and the camera 51 to a predetermined distance based on the size of the iris of the subject 6 recognized from the camera image. In this case, the first notification unit 16 determines whether the iris size recognized from the camera image falls within an appropriate size range. The appropriate size range is an iris size range corresponding to a shooting distance range that is within a distance range preferable for assessing the medication effect, and is pre-stored in, for example, the storage device 4 or the memory 12. If the recognized iris size is outside the appropriate size range, the first notification unit 16 causes the output device 3 to output guidance information instructing the user to adjust the shooting distance. In this case, if the recognized iris size is smaller than the appropriate size range, the first notification unit 16 outputs guidance information (guidance) instructing the user to shorten the shooting distance, and if the recognized iris size is larger than the appropriate size range, the first notification unit 16 outputs guidance information instructing the user to increase the shooting distance. In this way, the first notification unit 16 can suitably adjust the photographing distance to a distance suitable for determining the medication effect.

[0037] When the first notification unit 16 notifies the subject 6 of the imaging timing, the pupil information generation unit 17 generates information about the pupils of the subject 6 (also referred to as "pupil information") based on camera images of the subject 6. As a result, the pupil information generation unit 17 generates pupil information for each imaging timing determined by the imaging timing determination unit 15 and supplies the generated pupil information to the medication effect assessment unit 18. The pupil information generation unit 17 calculates at least the pupil size (e.g., pupil diameter) of the subject 6 as the pupil information. In this case, the pupil information generation unit 17 may measure a predetermined number of pupil sizes based on a predetermined number of camera images obtained after the first notification unit 16 notifies the subject 6, and generate pupil information indicating the measured predetermined number of pupil sizes. In this case, the pupil information may be information indicating the average, maximum, or other representative value of the predetermined number of pupil sizes.

[0038] The medication effect assessment unit 18 assesses the medication effect of the medication administered to the subject 6 based on the pupil information generated for each imaging timing. In this case, the medication effect assessment unit 18 calculates one or more indices (also referred to as "pupil change indices") indicating changes in the pupil indicated by the pupil information. The pupil change index is, for example, the difference between the pupil size before the medication effect appears and the pupil size when the medication effect is at its peak. For example, it is calculated as the difference between the pupil size based on a camera image generated at an imaging timing immediately after medication and the pupil size at an imaging timing at the peak of the medication effect estimated based on the medication information. The difference corresponds to the degree of miosis if the pupil is constricted, or the degree of mydriasis if the pupil is dilated. The medication effect assessment unit 18 then assesses the medication effect based on the calculated pupil change index and supplies information regarding the medication effect assessment result to the second notification unit 19.

[0039] Here, the determination of medication effect by the medication effect determining unit 18 will be described.

[0040] The medication effect assessment unit 18 generates a determination result indicating, for example, whether the medication effect is within an appropriate range. Note that, when the medication effect is not within the appropriate range, the medication effect assessment unit 18 may further generate a determination result indicating whether the medication effect is excessive or insufficient.

[0041] In this case, for example, the medication effect assessment unit 18 compares the calculated pupil change index with a reference value of the pupil change index and assesses the medication effect based on the comparison result. The reference value may be a statistical reference value of the pupil change index for the age of the subject 6 (i.e., a statistical value of the pupil change index according to the attributes of the subject 6), or a previously calculated value of the pupil change index for the subject 6. The reference value may also be a threshold value for determining whether the medication effect is within an appropriate range. The medication effect assessment unit 18 may also use a model that performs inference regarding the medication effect and output information output by the model as information regarding the medication effect assessment result. The model may be a machine learning model such as an equation, a lookup table, or a neural network, and may output an inference result regarding the medication effect when, for example, calculated values ​​of various pupil change indices (or the difference between the calculated value and the reference value) are input. Parameters of the model may be stored in advance in the storage device 4, memory 12, or the like. In addition, the medication effectiveness assessment unit 18 may output information for displaying a graph or the like that allows comparison between the calculated values ​​of various pupil change indices and the corresponding reference values ​​as information regarding the assessment results of the medication effectiveness.

[0042] The second notification unit 19 displays or outputs as audio information on the medication effect determination result supplied from the medication effect determination unit 18 on the output device 3. In this case, the second notification unit 19 generates an output signal including at least one of a display signal and an audio signal indicating information on the medication effect determination result, and supplies the output signal to the output device 3 via the interface 13. The second notification unit 19 may also perform processing to notify a third party other than the subject 6 of the information on the medication effect determination result, or may perform processing to store the information on the medication effect determination result in the storage device 4, the memory 12, or the like.

[0043] The components of the medication information acquisition unit 14, the imaging timing determination unit 15, the first notification unit 16, the pupil information generation unit 17, the medication effect assessment unit 18, and the second notification unit 19 described in FIG. 4 can be realized, for example, by the processor 11 executing a program. Alternatively, the necessary programs may be recorded on any non-volatile storage medium and installed as needed to realize each component. At least some of these components may not necessarily be realized by software programs, but may also be realized by any combination of hardware, firmware, and software. At least some of these components may also be realized using a user-programmable integrated circuit, such as an FPGA (Field-Programmable Gate Array) or a microcontroller. In this case, the integrated circuit may be used to realize a program consisting of the above components. Furthermore, at least a portion of each component may be configured by an ASSP (Application Specific Standard Product), an ASIC (Application Specific Integrated Circuit), or a quantum processor (quantum computer control chip). In this way, each component may be realized by various hardware. The same applies to other embodiments described below. Furthermore, each of these components may be realized by the cooperation of multiple computers, for example, using cloud computing technology.

[0044] (4) Pupil Information First, a specific example of pupil information will be described. Fig. 5 shows an example of a camera image of the face of the subject 6.

[0045] The image processing device 1 recognizes, based on any image recognition technology, each of the areas of the eyes 9Ra, 9La, irises 9Rb, 9Lb, and pupils 9Rc, 9Lc of the subject 6 from a camera image capturing the face of the subject 6. Then, based on each of the recognized areas, the image processing device 1 calculates pupil information indicating at least the pupil size.

[0046] For example, the image processing device 1 generates pupil information indicating the pupil size (i.e., pupil diameter) corresponding to the length of the arrows 91 and 92. Note that if the image processing device 1 calculates the pupil size for each eye, it may generate pupil information indicating the average of the calculated pupil sizes. Furthermore, if only one eye of the subject 6 is displayed in the camera image, the image processing device 1 may generate pupil information indicating the pupil size of the displayed eye.

[0047] Here, the image processing device 1 may perform a normalization process to convert the pupil size from a size based on the number of pixels in the camera image to a size of a predetermined scale (e.g., actual size), and generate pupil information indicating the normalized pupil size. In this case, for example, the image processing device 1 may normalize the pupil size based on the size of the irises 9Rb and 9Lb, taking advantage of the characteristics that the size of the irises 9Rb and 9Lb varies little between individuals and does not change over time. In this case, for example, information indicating the relationship between the size of the irises 9Rb and 9Lb in the image and the size of the irises 9Rb and 9Lb after normalization is stored in advance in the storage device 4, memory 12, or the like, and the image processing device 1 performs the above-mentioned normalization by referring to this information. In another example, when a plurality of cameras with different viewpoints are provided as cameras 51 in the medication effect assessment system 100, the image processing device 1 may recognize the actual size of the pupil size by three-dimensionally reconstructing the face of the subject 6 from camera images from the plurality of cameras based on any three-dimensional reconstruction technology such as SfM (Structure From Motion).

[0048] (5) Imaging Timing Next, the imaging timing determined by the imaging timing determination unit 15 will be described. Fig. 6 is a graph showing an example of a change in pupil size of the subject 6 after the subject 6 has taken a certain drug. Here, as an example, a case will be described in which the subject 6 has taken a drug that causes pupil dilation upon administration.

[0049] The imaging timing determination unit 15 determines, as imaging timings, at least the timing immediately after the start of medication (corresponding to the medication time, time "t1" in FIG. 6 ) and the timing at which the medication effect peaks (time "t3" in FIG. 6 ). Here, the pupil size at time t1 corresponds to the pupil size when there is no medication effect, and is the minimum value "Dmin." On the other hand, the pupil size at time t3 corresponds to the pupil size when the pupil is most dilated due to the medication effect, and is the maximum value "Dmax." Note that in the case of a drug that causes miosis, the pupil size reaches its minimum value at time t3.

[0050] Preferably, in addition to the above-mentioned timings, the imaging timing determination unit 15 determines at least one of the timings when the medicine starts to take effect (time "t2" in Figure 6) and the timing when the medication effect decreases (ends) (time "t4" in Figure 6) as the imaging timing.

[0051] Here, time t2 (i.e., the timing when the medication starts to work), time t3 (i.e., the timing when the medication effect peaks), and time t4 (i.e., the timing when the medication effect decreases) each vary depending on the type (and amount) of medication used by the subject 6. Therefore, the imaging timing determination unit 15 estimates these timings based on the medication content indicated by the medication information.

[0052] First, a method for estimating the timing of the peak medication effect will be described. Based on the medication content indicated by the medication information, the imaging timing determination unit 15 recognizes the length of time from the administration time to the occurrence of the peak medication effect (the length of time from time t1 to time t3 in FIG. 6 ), and determines the time when the above-mentioned length of time has elapsed from the administration time as the timing of the peak medication effect. In this case, table information showing the correspondence between the medication content and the above-mentioned length of time (i.e., table information correlating the above-mentioned length of time to be set for each expected medication content) is pre-stored in the storage device 4, memory 12, etc. Then, the imaging timing determination unit 15 determines the above-mentioned length of time from the medication content by referring to this table information. Note that in this case, the imaging timing determination unit 15 may determine the above-mentioned length of time from the type of medication as the medication content, or may determine the above-mentioned length of time from both the type of medication and the dosage (usage amount).

[0053] Similarly, the imaging timing determination unit 15 refers to table information indicating the correspondence between the medication content and the length of time from the administration time until the medication starts to take effect (the length of time from time t1 to time t2 in FIG. 6 ), and estimates the timing at which the medication will start to take effect from the medication content. Furthermore, the imaging timing determination unit 15 refers to table information indicating the correspondence between the medication content and the length of time from the administration time until the medication effect wears off (the length of time from time t1 to time t4 in FIG. 6 ), and estimates the timing at which the medication effect wears off from the medication content. The above-mentioned table information is stored in advance in, for example, the storage device 4 or the memory 12.

[0054] In this way, the imaging timing determination unit 15 can accurately estimate the timing of each pupil measurement required for determining the medication effect based on the medication information, and can accurately determine the imaging timing.

[0055] Next, the calculation of the pupil change index will be further explained with reference to FIG. 6 . For example, the medication effect assessment unit 18 calculates, as the pupil change index, the difference between the pupil size Dmin measured based on a camera image generated at time t1 immediately after medication and the pupil size Dmax measured based on a camera image generated at time t3 when the medication effect peaks. That is, the medication effect assessment unit 18 calculates, as the pupil change index, the difference between the pupil size when there is no medication effect and the pupil size when the pupil is most dilated due to the medication effect.

[0056] Note that, instead of using the pupil size immediately after the start of medication (time "t1" in FIG. 6 ) as the pupil size when medication is ineffective, the imaging timing determination unit 15 may use the pupil size based on a camera image of the subject 6 taken at any timing before the start of medication. In this case, for example, a camera image of the subject 6 taken at any timing before the start of medication is stored in the storage device 4, memory 12, etc., and the imaging timing determination unit 15 calculates the pupil size when medication is ineffective based on the camera image. Note that the pupil size of the subject 6 when medication is ineffective may be stored in advance as registered information in the storage device 4, memory 12, etc.

[0057] Furthermore, the imaging timing determination unit 15 may determine multiple imaging timings near the estimated time t3 (i.e., the timing at which the medication effect peaks), and estimate the maximum pupil size "Dmax" (or the minimum value in the case of a drug that causes miosis) based on the pupil size measured at each imaging timing by interpolation or other statistical methods (e.g., selection of a representative value such as a maximum value). Similarly, the imaging timing determination unit 15 may determine multiple imaging timings for each of time t2 (i.e., the timing at which the medication effect begins) and time t4 (i.e., the timing at which the medication effect decreases), and determine the pupil size to be used in calculating the pupil change index based on the pupil size measured at each imaging timing.

[0058] (6) Notification of Medication Effect Determination Result Next, a specific example of notification of the medication effect determination result by the second notification unit 19 will be described.

[0059] First, a case where a judgment result indicating that the medication effect is too small (for example, when the subject 6 uses a drug that causes mydriasis but the degree of mydriasis is lower than the standard) will be described.

[0060] In a first notification example when the medication effect is determined to be insufficient, the second notification unit 19 takes into consideration the possibility that the subject 6 took an insufficient amount of medication (i.e., the subject may have taken a smaller amount than prescribed), and issues a notification urging the subject to confirm whether they took the medication as prescribed. For example, the second notification unit 19 outputs guidance such as "Did you take the medication as prescribed?" or "The medication does not seem to be working very well. Please consult your doctor at your next appointment."

[0061] In a second notification example when the medication effect is determined to be insufficient, the second notification unit 19 takes into consideration the possibility that the medication is not working, and notifies the subject 6 that the medication is not working because the prescribed amount is too small or because of the effects of drug interactions. For example, the second notification unit 19 outputs guidance such as "Did you take the medication as prescribed?", "The medication does not seem to be working very well. Please consult your doctor at your next appointment," or "You may have eaten food that does not interact well with the medication. Please record what you ate today and consult your doctor or pharmacist at your next appointment."

[0062] Furthermore, in the second notification example described above, the second notification unit 19 may acquire information (also referred to as "ingestion information") regarding a substance ingested (including oral and cutaneous ingestion) by the subject 6, and display the ingestion information in chronological correspondence with the pupil size calculated by the pupil information generation unit 17. In this case, the ingestion information may be, for example, information indicating at least the type (and amount) of the ingested substance and the time of ingestion. If an application that manages the ingestion information of the subject 6 is installed in the image processing device 1, the second notification unit 19 acquires the ingestion information from the application. In another example, the second notification unit 19 acquires an image of the substance ingested by the subject 6, captured by the camera 51 or the like, and generates the ingestion information by analyzing the image using any image recognition technology. In this case, the subject 6 performs an input to designate the captured image as the image to be analyzed by the second notification unit 19.

[0063] Furthermore, in the second notification example described above, if second notification unit 19 detects, based on the intake information, the intake of a substance that does not interact well with the drug indicated in the medication information (or a drug registered as a drug regularly taken by subject 6), it may notify subject 6 that an interaction effect will occur. In this case, for example, table information showing the correspondence between drugs and substances that affect the drug interaction is stored in storage device 4, memory 12, or the like, and if second notification unit 19 detects, based on the table information and the intake information, the intake of a substance that will cause an interaction effect, it notifies subject 6 that an interaction effect will occur.

[0064] Next, a case where a judgment result indicating that the medication effect is excessive is obtained (for example, when the subject 6 uses a drug that causes mydriasis, but the degree of mydriasis is higher than the standard) will be described.

[0065] In a first notification example when it is determined that the medication effect is excessive, the second notification unit 19 takes into consideration the possibility that the subject 6 took too much medicine (i.e., the subject may have taken a larger amount than prescribed) and issues a notification urging the subject 6 to confirm whether they took the medicine as prescribed. For example, the second notification unit 19 outputs guidance such as "Did you take the medicine as prescribed?" or "It seems that the medicine is too effective. Please consult your doctor at your next appointment."

[0066] In a second notification example when the medication effect is determined to be excessive, the second notification unit 19 may provide a notification suggesting that the medication is too effective, that the prescribed amount may be too high, that the subject 6's aging may be a factor, or that the type of medication used may not be appropriate for the subject 6. For example, the second notification unit 19 outputs guidance such as, "Did you take the medication as prescribed?", "The medication seems to be too effective. Please consult your doctor at your next appointment," or "The medication seems to be too effective. Age or multiple medications can cause over-effectiveness. Please record the medications taken today and consult your doctor or pharmacist at your next appointment." In the second notification example described above, the second notification unit 19 may acquire information about the medication used by the subject 6 and display the acquired information together with the time-series measurement results of the pupil size measured by the pupil information generation unit 17 on the output device 3. The second notification unit 19 may also associate information about the medication used by the subject 6 with the time-series measurement results of the pupil size and store them in the storage device 4, memory 12, or the like.

[0067] The subject 6 who has received the above-mentioned notification can, for example, provide feedback on the assessment result of the medication effect to the doctor and consult with the doctor to change the prescribed amount. The subject 6 can also determine the dosage of the medication tailored to the individual, reduce side effects, and adjust the medication to be used most effectively.

[0068] The second notification unit 19 may also notify a third party other than the subject 6 of information relating to the determination result of the medication effect.

[0069] For example, if the determination result that the medication effect for the subject 6 is not within the appropriate range is returned a predetermined number of times in succession, the second notification unit 19 notifies a third party other than the subject 6 of information related to the determination result of the medication effect. In this case, for example, contact information for the third party is stored in the storage device 4, memory 12, or the like, and the second notification unit 19 transmits information related to the determination result of the medication effect to the contact indicated in the contact information. This makes it possible to appropriately encourage the subject 6 to use an appropriate type or amount of medication.

[0070] Furthermore, the second notification unit 19 may notify the subject 6 of information regarding past medication effect determination results in addition to the current medication effect determination result for the subject 6 (i.e., the latest medication effect determination result supplied from the medication effect determination unit 18). For example, the second notification unit 19 may display the current medication effect determination result for the subject 6 and the past medication effect determination results for the subject 6 so that they can be compared. The second notification unit 19 may notify the subject 6 of individual medication effect determination results obtained in the past, or may notify statistical data summarizing the past determination results through statistical processing such as averaging of the individual medication effect determination results obtained in the past. The second notification unit 19 may also notify the subject 6 of medication effect determination results obtained a predetermined period ago (e.g., one year ago) along with the subject 6's current medication effect determination result. In addition, the second notification unit 19 stores information regarding the obtained medication effect judgment result in the storage device 4 or memory 12, etc., for example, each time a medication effect judgment result is obtained, and information regarding past medication effect judgment results is stored in the storage device 4 or memory 12, etc.

[0071] Here, a supplementary explanation of the effect of the notification by the second notification unit 19 will be provided. For example, if the subject 6 is elderly, by notifying the subject 6 of the current medication effect assessment result compared with data from a predetermined period of time ago (e.g., one year ago), the second notification unit 19 can appropriately alert the subject 6 or a third party to any tendency for the medication's effectiveness to change with age. Generally, aging causes liver and kidney function to decline, which increases the time required for metabolism and excretion, potentially resulting in the medication being too effective. In another example, if the subject 6 is experiencing cognitive decline or a mental illness, the second notification unit 19 can notify a third party of the medication effect assessment result, indicating that the subject 6 took too little or that a drug combination may have caused adverse effects. In this case, even if the subject 6 is not aware of whether the dosage was appropriate or whether the medication is effective, notifying a third party can provide a sense of security to the third party or the subject 6, and encourage third parties to provide support for maintaining the subject 6's condition. Furthermore, for subjects 6 suffering from mental illness, notifications from the second notification unit 19 can be used to prevent overdoses, etc., thereby reducing the frequency of hospitalization for the subject 6 and preventing the condition from worsening.

[0072] (5) Processing Flow FIG. 7 is an example of a flowchart relating to the medication effect assessment process executed by the image processing device 1.

[0073] First, the image processing device 1 determines whether or not the subject 6 has used medicine (i.e., administered medication) (step S11). If the image processing device 1 determines that medicine has been used (step S11; Yes), it acquires medication information indicating the dosage content and time of administration of the medicine used (step S12). The image processing device 1 determines whether or not medicine has been used in step S11 and acquires the medication information in step S12, for example, based on user input via the input device 2 (including registration of an image immediately after administration). On the other hand, if it determines that medicine has not been used (step S11; No), the image processing device 1 continues to determine whether or not medicine has been used in step S11.

[0074] After acquiring the medication information, the image processing device 1 determines the timing of photographing based on the medication information (step S13). As a result, the image processing device 1 determines the timing of photographing, such as the timing immediately after medication or the timing at which the medication effect estimated based on the medication information reaches its peak.

[0075] The image processing device 1 then determines whether one of the image capture timings determined in step S13 has arrived (step S14). If the image processing device 1 determines that one of the image capture timings determined in step S13 has arrived (step S14; Yes), it notifies the subject 6 of the image capture timing via the output device 3 (step S15). This causes the image processing device 1 to prompt the subject 6 to perform operations related to the camera 51 (including adjusting the shooting distance, etc.) required for pupil measurement. In this case, the image processing device 1 may output guidance for stabilizing the focal length, guidance for adjusting the shooting distance, etc. via the output device 3.

[0076] Then, the image processing device 1 generates pupil information indicating at least the pupil size of the subject 6 based on the camera image captured by the camera 51 after the notification in step S15 (step S16). On the other hand, if the image processing device 1 determines that it is not time to capture an image (step S14; No), it continues to determine whether it is time to capture an image in step S14.

[0077] Next, the image processing device 1 determines whether there is an unprocessed photographing timing (step S17). In other words, the image processing device 1 determines whether the processes of steps S15 and S16 have been completed for all photographing timings determined in step S13. If there is an unprocessed photographing timing (step S17; Yes), the image processing device 1 returns to step S14.

[0078] On the other hand, if there is no unprocessed imaging timing (step S17; No), the image processing device 1 determines the medication effect based on the pupil information generated in step S16 (step S18). Then, the image processing device 1 outputs information related to the medication effect determination result (step S19). In this case, the image processing device 1 may display or output the information related to the medication effect determination result by the output device 3, store it in the storage device 4, or transmit it to the contact information of a third party other than the subject 6.

[0079] <Second embodiment> Fig. 8 shows a schematic configuration of a medication effect assessment system 100A according to the second embodiment. The medication effect assessment system 100A according to the second embodiment has an image processing device 1A that functions as a server, and a terminal device 8 that is used by a subject and functions as a client. The image processing device 1A and the terminal device 8 communicate data via a network 99. Hereinafter, the same components as those in the first embodiment will be appropriately designated by the same reference numerals, and their description will be omitted.

[0080] The terminal device 8 is a terminal used by a user who will be the subject, and has input, display, communication, and imaging functions, and functions as the input device 2, output device 3, and measurement device 5 including camera 51 shown in Fig. 1. The terminal device 8 may be, for example, a personal computer, a tablet terminal such as a smartphone, or a PDA (Personal Digital Assistant). The terminal device 8 transmits the facial image of the subject output by the camera 51 to the image processing device 1A via the network 99.

[0081] The image processing device 1A has the same hardware configuration as the image processing device 1 shown in Fig. 2, and the processor 11 of the image processing device 1A has the functional blocks shown in Fig. 4 described in the first embodiment. The image processing device 1A receives camera images from the terminal device 8 via the network 99 and executes a process for assessing the medication effect of the subject. In addition, the image processing device 1A transmits an output signal for outputting the processing results to the terminal device 8 via the network 99 based on a display request from the terminal device 8.

[0082] In this way, the image processing device 1A in the second embodiment performs processing related to determining the medication effectiveness of the subject who is the user of the terminal device 8, and can present information regarding the determination results of the medication effectiveness to the subject via the terminal device 8 in an appropriate manner.

[0083] 9 is a block diagram of an image processing device 1X according to a third embodiment. The image processing device 1X mainly includes a medication information acquisition unit 14X, an imaging timing determination unit 15X, and a pupil information generation unit 17X. The image processing device 1X may be configured by a plurality of devices.

[0084] The medication information acquiring means 14X acquires medication information related to the medication administered to the subject. The medication information acquiring means 14X can be, for example, the medication information acquiring unit 14 in the first or second embodiment.

[0085] The imaging timing determination means 15X determines the timing of imaging the subject by the imaging means based on the medication information. The imaging timing determination means 15X can be, for example, the imaging timing determination unit 15 in the first or second embodiment.

[0086] The pupil information generating means 17X generates pupil information relating to the pupil of the subject based on the image generated by the imaging means at the timing of imaging. The pupil information generating means 17X can be, for example, the pupil information generating unit 17 in the first or second embodiment.

[0087] 10 is an example of a flowchart executed by the image processing device 1X in the third embodiment. First, the medication information acquisition means 14X acquires medication information related to the medication administered to the subject (step S21). The imaging timing determination means 15X determines the timing of imaging the subject by the imaging means based on the medication information (step S22). The pupil information generation means 17X generates pupil information related to the pupil of the subject based on the image generated by the imaging means at the imaging timing (step S23).

[0088] According to the third embodiment, the image processing device 1X can suitably acquire pupil information relating to the pupils that change with the use of medicine by the subject.

[0089] In the above-described embodiments, the program can be stored using various types of non-transitory computer-readable media and supplied to a computer processor or the like. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic storage media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical storage media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory)). The program may also be supplied to a computer by various types of transitory computer-readable media. Examples of transitory computer-readable media include electrical signals, optical signals, and electromagnetic waves. The transitory computer-readable media can be supplied to a computer via wired communication paths such as electric wires and optical fibers, or via wireless communication paths.

[0090] In addition, some or all of the above embodiments may be described as, but are not limited to, the following supplementary notes.

[0091] [Supplementary Note 1] An image processing device comprising: medication information acquisition means for acquiring medication information related to medication administered to a subject; imaging timing determination means for determining when to capture an image of the subject by an imaging means based on the medication information; and pupil information generation means for generating pupil information related to the pupils of the subject based on an image generated by the imaging means at the imaging timing. [Supplementary Note 2] The image processing device according to Supplementary Note 1, further comprising medication effect assessment means for making a determination regarding the effect of the medication based on the pupil information. [Supplementary Note 3] The image processing device according to Supplementary Note 2, wherein the pupil information generation means generates the pupil information indicating at least the size of the pupil, and the medication effect assessment means makes a determination regarding the effect based on a change in the pupil size. [Supplementary Note 4] The image processing device according to Supplementary Note 1, wherein the imaging timing determination means at least determines, based on the medication information, a timing at which the effect of the medication is estimated to peak as the imaging timing. [Supplementary Note 5] The image processing device according to Supplementary Note 1, wherein the imaging timing determination means at least determines, based on the medication information, a timing at which it is estimated that the effect of the medication will begin to take effect, as the imaging timing. [Supplementary Note 6] The image processing device according to Supplementary Note 1, wherein the imaging timing determination means at least determines, based on the medication information, a timing at which it is estimated that the effect of the medication will decrease, as the imaging timing. [Supplementary Note 7] The image processing device according to Supplementary Note 1, further comprising a notification means for notifying the subject that it is the timing to image when the imaging timing arrives. [Supplementary Note 8] The image processing device according to Supplementary Note 7, wherein the notification means outputs guidance information regarding the distance between the subject and the imaging means, based on a measurement result of the iris of the subject using the image. [Supplementary Note 9] The image processing device according to Supplementary Note 2, further comprising a notification means for notifying a third party other than the subject of the determination result, when the determination result by the medication effect determination means indicates that the effect is not within an appropriate range a predetermined number of times or more consecutively.[Supplementary Note 10] An image processing method in which a computer acquires medication information regarding medication administered to a subject, determines a timing for photographing the subject by an imaging means based on the medication information, and generates pupil information regarding the pupils of the subject based on an image generated by the imaging means at the photographing timing. [Supplementary Note 11] A storage medium having stored therein a program that causes a computer to execute processes of acquiring medication information regarding medication administered to a subject, determining a timing for photographing the subject by an imaging means based on the medication information, and generating pupil information regarding the pupils of the subject based on an image generated by the imaging means at the photographing timing.

[0092] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above embodiments. Various modifications within the scope of the present invention that would be understood by those skilled in the art can be made to the configuration and details of the present invention. In other words, the present invention naturally includes various modifications and alterations that would be possible for those skilled in the art based on the entire disclosure, including the claims, and the technical ideas. Furthermore, the disclosures of the above-cited patent and non-patent documents are incorporated herein by reference.

[0093] REFERENCE SIGNS LIST 1, 1A, 1X Image processing device 2 Input device 3 Output device 4 Storage device 5 Measuring device 8 Terminal device 11 Processor 12 Memory 13 Interface 51 Camera 90 Data bus 99 Network 100, 100A Medication effect assessment system

Claims

1. a medication information acquisition means for acquiring medication information regarding medications used by a subject; an imaging timing determination means for determining an imaging timing of the subject by an imaging means based on the medication information; a pupil information generating means for generating pupil information relating to the pupil of the subject based on the image generated by the imaging means at the imaging timing; An image processing device having:

2. The image processing device according to claim 1 , further comprising a medication effect determining means for determining the effect of the medication based on the pupil information.

3. the pupil information generating means generates the pupil information indicating at least the size of the pupil; The image processing device according to claim 2 , wherein the medication effect determining means determines the effect based on a change in the size of the pupil.

4. The image processing device according to claim 1 , wherein the imaging timing determination means determines, based on the medication information, at least a timing when the effect of the medication is estimated to be at its peak, as the imaging timing.

5. The image processing device according to claim 1 , wherein the imaging timing determination means determines, based on the medication information, at least a timing when the medication is estimated to start to take effect, as the imaging timing.

6. The image processing device according to claim 1 , wherein the imaging timing determination means determines, as the imaging timing, at least a timing when the effect of the medicine is estimated to decrease based on the medication information.

7. The image processing apparatus according to claim 1 , further comprising a notification unit that, when the timing for photographing arrives, notifies the subject that it is the timing for photographing.

8. The image processing device according to claim 7 , wherein the notification means outputs guidance information regarding a distance between the subject and the imaging means based on a measurement result of the iris of the subject using the image.

9. The computer Obtaining medication information regarding medications used by the subject; determining a timing for photographing the subject by an imaging means based on the medication information; generating pupil information relating to the pupil of the subject based on the image generated by the imaging means at the imaging timing; Image processing methods.

10. Obtaining medication information regarding medications used by the subject; determining a timing for photographing the subject by an imaging means based on the medication information; A program that causes a computer to execute a process of generating pupil information regarding the pupil of the subject based on the image generated by the imaging means at the imaging timing.