Monitoring system

Abstract

An apparatus for indicating a health status of a user, comprising: a camera (118) configured to capture a reference image of a user's face and a new image of the user's face; a processor (102) configured to: determine facial features of the user in the reference image and the new image; determine differences in the facial features determined from the reference image and the status image; generate an alert when differences in the facial features between the reference image and the new image are determined; and store the record in memory (104). A facial recognition method and a computer program including machine-readable instructions for indicating a health status of a user are also provided.

Description

Technical Field

[0001] The present invention relates to a monitoring system for patients undergoing drug treatment, particularly a face monitoring system.

Background Art

[0002] For example, there are various diseases that require regular treatment by injecting drugs. Such injections are performed by using an injection device applied by a healthcare provider or the patient themselves. As an example, type 1 and type 2 diabetes are treated by the patient themselves, for example, by injecting insulin doses once or several times a day. Similarly, in the pre-diabetes stage, overweight or obese patients can be given injection treatment for chronic weight management.

[0003] Unfortunately, during the process of drug treatment, patients may experience side effects that can be caused by either the drug itself or the condition being treated. These side effects include, for example, weight gain, a higher susceptibility to other diseases, or an overall deterioration of the general health condition. In some cases, these side effects may themselves indicate a change in the characteristics or features of the patient's face.

[0004] Therefore, when patients require regular or long-term treatment, there is a need to provide a system that can monitor the overall health condition of the patient according to that treatment. In particular, it is necessary to determine any changes in the patient's health according to the treatment regimen and encourage good patient habits in faithfully following the treatment regimen. This is important because changes in the patient's response to treatment may require adjustments to the nature of the treatment, such as the dosing regimen, or the introduction of additional treatment.

[0005] Aspects of the present invention have been devised with the above in mind.

Summary of the Invention

Means for Solving the Problems

[0006] According to one aspect of the present invention, a computer program is provided which, when executed by a processor, causes the processor to control a camera to capture a reference image of the user's face and a new image of the user's face; to determine the user's facial features in the reference image and the new image; to determine the difference in facial features determined from the reference image and the new image; to generate a record of the facial features; to generate a warning when a difference in facial features between the reference image and the new image is determined; and to store the record in memory.

[0007] This is advantageous because it provides a means of monitoring and tracking the patient's overall health status in accordance with treatment. In addition, it is possible to identify any changes in the patient's condition in response to treatment and / or the onset of complications. Thus, the computer program provides an early warning system for changes in the patient's health and compliance with treatment regimens.

[0008] Facial characteristics may relate to at least one of the following: eyes, skin, hair, and facial impression.

[0009] The processor may also determine the facial impression based on the distance measured between fixed face points and variable face points.

[0010] Fixed facial points may include at least one of the nasal bridge and the outer edge of the nostrils. Variable facial points may include at least one of the outer edge of the eyelid and the corner of the mouth.

[0011] The processor can determine at least one of the following: the color and clarity of the dermis of the eye.

[0012] The processor may determine at least one of the following: skin color, skin tone, and skin moisture.

[0013] The processor can determine at least one of the following: hair distribution and hair volume.

[0014] The processor may control the display to show a warning, which may include a user questionnaire.

[0015] The processor may control the communication unit to send the questionnaire to an external device when it is completed.

[0016] The processor may generate an input window requesting the capture of a reference image or a new image before capturing the reference image and / or a new image, and may control the display to show the input window.

[0017] According to another aspect of the present invention, a smartphone application including a computer program according to the present invention is provided.

[0018] According to another aspect of the present invention, a device for indicating a user's health status is provided, comprising: a camera configured to capture a reference image of the user's face and a new image of the user's face; a processor configured to determine the user's facial features in the reference image and the new image; determine the difference in facial features determined from the reference image and the state image; generate a warning when a difference in facial features between the reference image and the new image is determined; and store the record in memory.

[0019] The device may be a portable device.

[0020] According to another aspect of the present invention, a face recognition method for indicating a user's health status is provided, which includes: capturing a reference image of the user's face and a new image of the user's face; determining facial features in the reference image and the new image; determining the difference in facial features determined from the reference image and the new image; generating a record of facial features; generating a warning when a difference in facial features between the reference image and the new image is determined; and storing the record in memory.

[0021] Herein, embodiments of the present invention will be described only as examples, with reference to the attached drawings. [Brief explanation of the drawing]

[0022] [Figure 1] It is a schematic diagram showing the internal components of a portable device. [Figure 2] It is a flowchart showing the operation method of a monitoring system during an initial setup process. [Figure 3] It is a flowchart showing the operation method of a monitoring system during subsequent use (for example, daily). [Figure 4] It is a diagram showing fixed facial points and variable facial points identified by an evaluation tool used during face analysis. [Figure 5] It is a diagram showing skin target areas identified by an evaluation tool used during face analysis.

MODE FOR CARRYING OUT THE INVENTION

[0023] In the following disclosure, a computer program having machine-readable instructions that cause a processor to start various operations when executed by the processor is described.

[0024] In the following exemplary embodiments, the computer program is implemented on a portable device, and the computer program is in the form of a monitoring application.

[0025] In summary, a computer program implemented on a portable device provides a monitoring system or monitoring function for monitoring changes in the facial impression of a patient during drug treatment. When executed, the monitoring application provides, for example, an indication of the overall health status of the patient according to the drug treatment. The monitoring application is provided to assist various clinical studies. These include, for example, clinical studies related to diabetes, chronic weight management, cardiovascular conditions, rheumatism or psoriasis.

[0026] According to the following exemplary embodiment, the portable device is a mobile phone, such as a smartphone. Advantageously, the monitoring application is a separate application. The monitoring application is either provided within the portable device at the time of manufacture or downloaded to the portable device by the user from, for example, an application marketplace or application store.

[0027] Figure 1 is a schematic diagram of some of the internal components of the portable device 100. The portable device 100 includes a processor 102. The processor 102 may be any type of integrated circuit. The processor 102 controls the operation of the other hardware components of the portable device 100. The processor 102 and the other hardware components are connected via a system bus (not shown). Each hardware component is connected to the system bus either directly or via an interface.

[0028] The portable device 100 includes a display 112 (e.g., LCD, TFT (thin-film transistor), OLED (organic light-emitting diode), e-paper). The display may be a touch-sensitive display having a display section 113 and a tactile interface section 114. The portable device 100 also includes a communication interface 116, such as a Bluetooth® interface. The portable device 100 also includes a camera 118. Any suitable camera may be used, and the camera 118 may further include a front lens and / or a rear lens. The portable device 100 may also include a radar sensor (not shown). The radar sensor may be capable of performing face recognition by detecting and recording changes in facial features, such as facial contours or dimensions. The radar sensor may thereby contribute to the operation of the camera 118 when performing face recognition. The portable device 100 also houses a battery 120 for powering the portable device 100 by a power supply 119.

[0029] The processor 102 is configured to send and receive signals to and from other components in order to control the operation of those components. For example, the processor 102 controls the display of content on the display 112 and receives signals as a result of user input from the haptic interface 114. The display 112 may be any type of resistive or capacitive touchscreen. Alternatively, the display 112 may not be a touchscreen. For example, the display 112 may be a liquid crystal display (LCD).

[0030] The portable device 100 includes memory 104, i.e., working memory or volatile memory such as random access memory (RAM), and non-volatile memory. The processor 102 can access RAM to process data and can control the storage of data in memory 104. RAM can be any type of RAM, e.g., static RAM (SRAM), dynamic RAM (DRAM), or flash memory. Non-volatile memory stores data files and associated metadata, as well as the operating system 108 and the monitoring application 110. Memory 104 can be any type of non-volatile memory, such as read-only memory (ROM), flash memory, and magnetic drive memory.

[0031] The processor 102 operates under the control of the operating system 108. The operating system 108 may include hardware such as the display 112 and the communication interface 116, as well as code related to the basic operation of the portable device 100. In addition to the monitoring application 110, the operating system 108 may also trigger the activation of other software modules stored in memory 104.

[0032] Other standard or optional components of the portable device 100, such as transceivers, are omitted.

[0033] Figures 2 and 3 are flowcharts illustrating how the monitoring application 110 operates when machine-readable instructions are executed by the processor 102 of the portable device 100. The flowcharts show how the monitoring application 110 and the portable device 100 interact and operate to provide a monitoring system. The steps are executed by the processor 102 of the portable device 100 under the control of the monitoring application 110 stored in memory 104.

[0034] Figure 2 details the initial operation or setup process of the monitoring application 110. In Figure 2, operation method 200 begins when the monitoring application 110 is launched for the first time. In step 201, the initial reference questionnaire is started to collect information about the patient's overall health status.

[0035] The questionnaire may include questions that assess, for example, a patient's physical function, pain, or overall health. The questionnaire may include questions associated with or related to one or more specific clinical studies. For example, a rheumatoid arthritis patient might be asked to answer questions according to the RAPID3 checklist. As another example, questions for a diabetic patient might focus on monitoring typical side effects to improve drug titration or to determine the correct dosage and timing of injections. Further examples might focus on monitoring secondary conditions commonly arising from a patient's primary condition, such as heart disease associated with diabetes or psoriatic arthritis associated with rheumatoid arthritis. However, the above examples are not intended to be limiting, and any suitable questions regarding the patient's overall health status may be included.

[0036] In step 202, the camera 118 of the portable device 100 is enabled. For example, a request is displayed asking for permission for the surveillance application 110 to access and enable the camera 118 so that at least one image of the user is captured by the camera 118. The camera 118 may face forward, for example, so that an image is taken while the user is looking at the display 112 of the portable device 100. In step 203, the camera 118 captures a reference image of the user's face and head and stores the reference image. The reference image provides a starting point for comparison to any future images captured by the camera 118 (Figure 3). The reference image is stored, for example, in the memory 104 of the portable device 100.

[0037] In step 204, a facial analysis of a reference image is performed. This may include, for example, determining the user's facial features (traits). The determined facial features are then compared to pre-stored information detailing predefined facial features that indicate a disease, disorder, or drug side effect. This comparison may identify one or more facial indicators. Facial indicators represent facial features that indicate a disease, disorder, drug side effect, or an overall change in the user's health condition.

[0038] If no face indicator is identified in step 205, no further action is required, and the monitoring application 110 is closed.

[0039] If a facial indicator is identified in step 205, a health status questionnaire (user questionnaire) is initiated in step 206. In other words, an alert is generated in the form of a health status questionnaire. The health status questionnaire is considered a secondary questionnaire to a reference questionnaire that focuses on changes in the user's facial features. The health status questionnaire may request further details about the patient's health. For example, the health status questionnaire may request further information related to a condition indicated by one or more facial indicators identified in the facial analysis. In other words, the health status questionnaire is structured to attempt to uncover further information about the patient's health linked to changes in conditions indicated by the facial analysis.

[0040] In step 207, the completed health status questionnaire is stored. The completed questionnaire is stored in the portable device 100, for example, in the memory 104 of the portable device 100. In step 207, the completed questionnaire is sent (output) to an external device. An example of an external device is a server. The server may be a server accessed by healthcare professionals. Once the questionnaire is sent, no further action is required, and the monitoring application 110 is closed.

[0041] Figure 3 details the daily operation of the monitoring application 110 after the initial setup is complete (Figure 2). In Figure 3, operation method 300 begins when the monitoring application 110 is started. This can be any time according to the user. In step 301, the monitoring application 110 captures and stores images of the user's face and head. The captured images represent new images related to the reference images captured during the initial operation of the monitoring application 110 (Figure 2, step 203).

[0042] In step 302, an image comparison is performed to determine if there are any changes in the user's facial features by comparing the new image to a reference image. This comparison involves performing a face analysis of the new image to determine if there are any differences between the facial features determined from the new image and those determined from the reference image. A record of the facial features determined from the new image is generated, and the results of the record are compared to the results of the face analysis performed on the reference image (Figure 2, step 204). This may include analyzing the user's facial features in comparison to pre-stored information detailing predefined facial features that may indicate, for example, a disease, disorder, or drug side effect.

[0043] In step 303, the comparison results are stored (comparison record). The results are stored in the form of a health record. The record is stored in the portable device 100, for example, in the memory 104 of the portable device 100.

[0044] In step 304, if no changes are detected in the user's facial features in the new image, and therefore one or more facial indicators indicating a disease are not identified in the user's facial features, no further action is required, and the monitoring application 110 is closed.

[0045] If, in step 304, a change in one or more features of the user's face is detected, thereby identifying one or more facial indicators indicating a disease, disorder, drug side effect, or other general change in the user's health, then in step 305, a health status questionnaire (user questionnaire) is initiated. In other words, when a difference in facial features between a reference image and a new image is determined, an alert is generated in the form of a health status questionnaire. The health status questionnaire may request further details about the patient's health. For example, the health status questionnaire may request further information related to the condition indicated by one or more facial indicators identified in the facial analysis. In other words, the health status questionnaire is configured to attempt to uncover further information about the patient linked to the condition indicated by the facial analysis.

[0046] In step 306, the completed health status questionnaire is stored. The completed questionnaire is stored in the portable device 100, for example, in the memory 104 of the portable device 100. In step 307, the completed questionnaire is sent (output) to an external device. An example of an external device is a server. The server may be accessible by healthcare professionals. Once the questionnaire is sent, no further action is required, and the monitoring application 110 is closed.

[0047] Here, the face analysis performed by the monitoring application 110 in step 204 of Figure 2 and step 302 of Figure 3 will be described in more detail. Face analysis can be any appropriate form of face analysis for determining the features or characteristics of a user's face. For example, face analysis may include face recognition performed by a software algorithm that evaluates image data. Radar sensor data may also be evaluated in combination with image data.

[0048] A user's facial features or characteristics include, for example, the user's eyes, skin, hair, the placement of features, and the overall facial impression. A facial indicator is, for example, a facial feature that indicates a disease, disorder, or drug side effect that is either in the pre-symptomatic or well-established stage.

[0049] Figures 4 and 5 show images of the target region of the user's face used during face analysis.

[0050] In Figure 4, exemplary fixed facial points 1 and variable facial points 2 are identified on the user's face. Fixed facial points 1 represent areas of the user's face that are unlikely to change. These areas include, for example, the point on the inner side of the eye near the nose where the tear duct is located, the bridge of the nose, or the outer edge of the nostrils. In contrast, variable facial points 2 represent areas of the user's face that may change in response to the user's facial expressions (e.g., emotional expressions such as smiling when happy or frowning when unhappy). These areas include, for example, the outer edge of the eye or eyelid, or the corners of the mouth.

[0051] In face analysis, the evaluation tool is configured to measure the distance between a fixed face point 1 and a variable face point 2. The distance measured between fixed face point 1 and variable face point 2 can provide an indication of the user's overall facial impression. The evaluation tool measures various distances, such as the distance between the eyes across the bridge of the nose, the height of the eyes from the upper eyelid to the lower eyelid, the distance from the nostrils to the outer edge of the eyes, or the distance between the nostrils and the outer edge of the lips. Facial indicators associated with fixed face point 1 and variable face point 2 include drooping mouth, cheeks, or eyes.

[0052] The evaluation tool may also measure eye features. Eye features include, for example, the color and / or clarity of the dermis of the eye, or the color and / or shape of the eyelids. Facial indicators associated with the eyes include, for example, dry, red, and / or discolored eyes, or red and / or swollen eyelids.

[0053] The evaluation tool may also measure skin characteristics. The target skin region is shown in Figure 5. The evaluation tool detects changes in the skin of the target region. Skin characteristics include, for example, color, tone, or the presence of moisture (e.g., sweating). Facial indicators associated with the skin include, for example, redness or excessive moisture.

[0054] The evaluation tool may also measure hair characteristics. For example, the evaluation tool may detect changes in hair distribution and volume. Facial indicators associated with hair may include, for example, hair loss, receding hairline, or excessive facial hair growth.

[0055] Facial features identified during facial analysis are compared to pre-memorized information detailing predefined facial features. These features, which may indicate disease, disorder, or drug side effects, are classified as facial indicators.

[0056] In addition, during the course of drug treatment, several images and health status questionnaires are taken and stored. When the number of stored images and / or questionnaires exceeds a predetermined value, the monitoring system may generate a new baseline image or determine new facial features to be evaluated during facial analysis. For example, the new baseline image may represent an ideal baseline image that shows the user's actual (baseline) appearance without any emotional influences such as being happy, tired, or angry. This allows the new baseline image to provide a more effective diagnostic assessment.

[0057] Next, various changes and modifications to the above embodiments will be described.

[0058] While this disclosure is described with reference to diabetes and chronic weight management, this is not intended to be limiting, and the teachings herein are equally well extended to other diseases or health conditions.

[0059] This disclosure is described in the context of a computer program implemented on a portable device 100, but this is not intended to limit the computer program to other suitable devices. For example, the device may be implemented on other portable devices 100 such as a PDA, any type of tablet computer, or a medical device such as a blood glucose measuring device. Alternatively, the computer program may be implemented on another suitable device such as a PC.

[0060] Although the health status questionnaire is described as being started in step 206, the initial health status questionnaire can also be started after camera 118 is enabled in step 202 (Figure 2). In addition, the reference image (step 203) and / or the completed health status questionnaire and / or discovery questionnaire (steps 206, 306) are transmitted to and stored on an external device (Figures 2 and 3). If necessary, input windows requesting the capture of a reference image and / or a new image appear in steps 203 and 301, respectively, before the reference image and / or new image are captured.

[0061] The evaluation tool can measure one or any combination of one or more of the listed facial features.

[0062] The facial indicators listed are illustrative and do not represent an exhaustive list; other features may also represent facial indicators.

Claims

1. It is a computer program, When executed by the processor, to the processor (102): The camera (118) is controlled to capture a reference image of the user's face and a new image of the user's face; Determine the facial features of the user in the reference image and the new image; Determine the difference in facial features between the reference image and the new image; Generate a record of facial characteristics; A warning is generated when a difference in facial features between the reference image and the new image is detected; Includes a machine-readable instruction to store the record in memory (104), The processor controls the display to show a warning, which includes a user questionnaire for when a difference in facial features between a reference image and a new image is detected. The user questionnaire requests further details about the patient's health, including additional information related to the condition indicated by the facial characteristics identified in the facial analysis. The aforementioned computer program.

2. The computer program according to claim 1, wherein the facial features relate to at least one of the eyes, skin, hair, and facial impression.

3. The computer program according to claim 2, wherein the processor determines the impression of a face based on the distance measured between a fixed facial point and a variable facial point.

4. The computer program according to claim 3, wherein the fixed facial point includes at least one of the bridge of the nose and the outer edge of the nostrils.

5. The computer program according to claim 3 or 4, wherein the variable facial point includes at least one of the outer edge of the eyelid and the corner of the mouth.

6. A computer program according to any one of claims 1 to 5, wherein the processor determines at least one facial characteristic among eye color and clarity.

7. A computer program according to any one of claims 1 to 6, wherein the processor determines at least one facial characteristic among skin color, skin tone, and skin moisture.

8. A computer program according to any one of claims 1 to 7, wherein the processor determines at least one facial characteristic, which is hair distribution and hair volume.

9. The computer program according to any one of claims 1 to 8, wherein the processor controls a communication unit to transmit a questionnaire to an external device when the questionnaire is completed.

10. The computer program according to any one of claims 1 to 9, wherein the processor generates an input window requesting the capture of a reference image or a new image before capturing the reference image and / or a new image, and controls the display to show the input window.

11. A smartphone application comprising a computer program according to any one of claims 1 to 10.

12. A device for indicating the user's health status: A camera (118) configured to capture a reference image of the user's face and a new image of the user's face; Determine the facial features of the user in the reference image and the new image; Determine the difference in facial features determined from the reference image and the new image; Generate a record of facial characteristics; A warning is generated when a difference in facial features between the reference image and the new image is detected; A processor (102) configured to store records in memory (104) and Includes, The processor controls the display to show a warning, which includes a user questionnaire when a difference in facial features between a reference image and a new image is determined, and the user questionnaire requests further details about the patient's health, including further information related to the condition indicated by the facial features identified in the facial analysis. The aforementioned device.

13. The apparatus according to claim 12, which is a portable device.

14. A method for operating a facial recognition device to indicate the user's health status, The device is activated: This involves capturing a reference image of the user's face and a new image of the user's face; Determining facial features in a reference image and a new image; To determine the difference in facial characteristics determined from a reference image and a new image; To generate a record of facial characteristics; To generate a warning when a difference in facial features between a reference image and a new image is detected; Control the display to show a warning, including a user questionnaire, when a difference in facial features between a reference image and a new image is detected. This includes storing the record in memory (104), The user questionnaire here requests further details about the patient's health, including additional information related to the condition indicated by the facial characteristics identified in the facial analysis. The aforementioned method.

Images