program

Abstract

A moving image acquisition unit (111) acquires moving image data indicating a moving image of the user's face captured by the user using a camera (16). The acquired moving image data is accumulated in a moving image DB (121) of a memory (12). A pulse measurement unit (112) reads the moving image data from the moving image DB (121), and measures vital values including pulse on the basis of the moving image data. A ratio calculation unit (113) calculates the ratio of sympathetic nerves and parasympathetic nerves on the basis of the pulse included in the measured vital values. An activity calculation unit (114) calculates the activity of the autonomic nerves overall on the basis of the pulse included in the measured vital value. A plotting unit (115) plots the state of the user's nervous system indicated by LF / HF and TP on a plane in which LF / HF and TP are the two axes.

Description

program

[0001] The present invention relates to a technology for outputting the state of a user's nervous system based on an image.

[0002] Patent Document 1 discloses an image processing method for stress monitoring that acquires facial image data using an RGB camera, creates hemoglobin image data from the facial image data using pigment component separation, creates heart rate data based on the hemoglobin image data, creates heart rate interval data based on the heart rate data, and creates heart rate variability spectrogram data by performing frequency conversion on the heart rate interval data.

[0003] JP 2017-29318 A

[0004] The technology described in Patent Document 1 can determine whether a subject is in a resting state or a stressed state based on an image of the subject's face. However, a person's physical and mental state may not be expressed by a single quantity, such as the intensity of stress. In other words, the technology described in Patent Document 1 only determines whether the subject considers themselves to be in a resting state or a stressed state, and is unable to identify various states.

[0005] One of the objects of the present invention is to provide a program that allows a person to be measured to easily check their own detailed condition in a variety of ways using a smartphone or the like.

[0006] In one aspect, the present invention provides a program for causing a computer to execute the steps of measuring a user's pulse based on an image of the user's body, calculating the ratio between sympathetic and parasympathetic nerves from the measured pulse, calculating the activity level of the entire autonomic nervous system from the measured pulse, and plotting the calculated ratio and the calculated activity level on a plane having the ratio between sympathetic and parasympathetic nerves and the activity level of the entire autonomic nervous system as two axes.

[0007] According to the present invention, the subject can check his / her own detailed condition in a variety of ways.

[0008] 1 is a diagram showing an example of the configuration of the measurement device 1. A diagram showing an example of a video DB 121. A diagram for explaining an example of the concept of video data. A diagram showing an example of a vital sign DB 122. A diagram showing an example of a physical condition DB 123. A diagram showing an example of a plane partitioned based on the physical condition DB 123. A diagram showing an example of a recommendation DB 124. A diagram showing an example of a medical interview DB 125. A diagram showing an example of a diagnostic image DB 126. A diagram showing an example of a first functional configuration of the measurement device 1. A diagram showing an example of a second functional configuration of the measurement device 1. A diagram showing an example of a third functional configuration of the measurement device 1. A diagram showing an example of the operation flow for the first function of the measurement device 1. A diagram showing an example of a display region R0. A diagram showing an example of a display when taking a selfie video. A diagram showing an example of a screen displayed when measuring pulse, etc. A diagram showing example plots of LF / HF and TP. A diagram showing an example of the operation flow for the second function of the measurement device 1. A diagram showing an example of the operation flow for the third function of the measurement device 1. A diagram showing an example of the overall configuration of a measurement system 9 according to a modified example. FIG. 10 is a diagram showing an example of the configuration of a server device 2 according to a modified example.

[0009] <Embodiments> <Configuration of Measuring Device> Fig. 1 is a diagram showing an example of the configuration of a measuring device 1. The measuring device 1 is a device that measures the physical condition of a user based on an image of the user's face (hereinafter also referred to as a "face image"). The measuring device 1 shown in Fig. 1 is a mobile terminal, such as a smartphone. Furthermore, "physical condition" here refers mainly to the state or condition of the mind and body resulting from disturbances in the autonomic nervous system.

[0010] 1, the measuring device 1 includes a processor 11, a memory 12, a communication unit 13, an operation unit 14, a display unit 15, and a camera 16. These are connected by a bus. The measuring device 1 may also include an observation unit 17, which is indicated by a dashed line in FIG. 1.

[0011] The processor 11 reads and executes programs stored in the memory 12 to control each part of the measurement device 1. The processor 11 is, for example, a CPU (Central Processing Unit).

[0012] The communication unit 13 is a communication circuit that communicatively connects the measuring device 1 to other devices via a wired or wireless connection. For example, the communication unit 13 may have a circuit that complies with wireless communication system standards such as IMT-2000, IMT-Advanced, or IMT-2020. Furthermore, for example, the communication unit 13 may have a circuit that complies with wireless LAN standards such as IEEE 802.11.

[0013] Furthermore, for example, the communication unit 13 may include a module that realizes near field communication (NFC). Examples of NFC standards include ISO / IEC 18092 (NFCIP-1), ISO / IEC 14443, ISO / IEC 15693, and IEEE 802.15.

[0014] The operation unit 14 includes operation buttons, a touch panel, and other operators for issuing various instructions, and receives operations and sends signals corresponding to the operation content to the processor 11. Examples of such operations include pressing a button or making a gesture on the touch panel.

[0015] The operation unit 14 may have a microphone for collecting voice. In this case, the processor 11 may perform voice recognition processing on the voice data representing the user's voice collected by the microphone, and accept the recognition result as the user's operation.

[0016] The display unit 15 has a display screen such as a liquid crystal display, and displays images under the control of the processor 11. A transparent touch panel of the operation unit 14 may be disposed on top of the display screen.

[0017] The display unit 15 has a display screen that displays images and the like that are visually perceived by the user, but in addition to this, it may also have, for example, a speaker that emits sound.

[0018] The camera 16 is an imaging unit, such as a digital still camera, that captures an image of the surroundings of the measuring device 1. The camera 16 includes an optical system such as a lens, as well as an imaging element such as a CMOS (Complementary Metal Oxide Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0019] The camera 16 shown in Fig. 1 has a common configuration with, for example, the "RGB camera" disclosed in Patent Document 1. That is, the camera 16 is a device that acquires image data by arranging a plurality of detection elements, each of which is sensitive to the light intensity in a wavelength range corresponding to each of the colors R (red), G (green), and B (blue), as a single pixel unit. The wavelength ranges corresponding to each color are preferably, for example, 580 nm to 680 nm for R (red), 500 nm to 630 nm for G (green), and 410 nm to 530 nm for B (blue).

[0020] The camera 16 may have a polarizing plate as part of its optical system. The polarizing plate is placed in the direction in which the camera 16 photographs the object, allowing the camera 16 to remove reflected light from the surface of the object.

[0021] The camera 16 may also detect light in a wavelength range other than the above-mentioned range. For example, the camera 16 may be a five-band camera in which two sensitivities, cyan (C) and orange (O), are added to a normal RGB camera.

[0022] 1 is a sensor or the like that observes physical quantities in the environment surrounding the user of the measurement device 1. This observation unit 17 is, for example, a barometer, a hygrometer, a thermometer, etc. Note that the measurement device 1 does not necessarily have to have this observation unit 17.

[0023] The memory 12 is a storage means for storing an operating system, various programs, data, etc. that are loaded into the processor 11. The memory 12 includes a RAM (Random Access Memory) and a ROM (Read Only Memory).

[0024] The memory 12 may include a solid state drive, a hard disk drive, etc. The memory 12 shown in FIG. 1 stores a video DB 121, a vital sign DB 122, a physical condition DB 123, a recommendation DB 124, a medical interview DB 125, and a diagnostic image DB 126.

[0025] <Configuration of Video DB> Fig. 2 is a diagram showing an example of the video DB 121. The video DB 121 is a database that stores data on videos (video data) of users' faces. The video DB 121 shown in Fig. 2 has a table structure that stores values ​​for each associated item. The video DB 121 shown in Fig. 2 has the following items: video ID, start date and time, length, and video data.

[0026] In this video DB 121, the value of the video data is the video data itself. The value of the video ID is identification information for uniquely identifying the corresponding video data. The value of the start date and time is information on the date and time when filming of the corresponding video data started (i.e., the start date and time). The value of the length is a numerical value indicating the length of the corresponding video data. The unit of this length is, for example, a time unit such as minutes or seconds.

[0027] FIG. 3 is a diagram illustrating an example of the concept of video data. Video data is composed of multiple still images captured at regular intervals. One frame of these multiple still images is called a frame. The number of still images captured per unit time is called the frame rate. As shown in FIG. 3, video data is composed of multiple ordered frames, each of which is associated with image data indicating the still image captured at each instant. Note that video data may be compressed as long as the image data for each frame can be extracted and acquired.

[0028] <Configuration of Vital Sign DB> FIG. 4 is a diagram showing an example of the vital sign DB 122. The vital sign DB 122 is a database that stores vital signs (hereinafter also referred to as vitals) that indicate the user's vital signs and physical condition measured based on video data. The vital sign DB 122 shown in FIG. 4 has a table structure that stores values ​​for each associated item. The vital sign DB 122 shown in FIG. 4 has the following items: video ID, pulse, heart rate variability, respiratory rate, LF, HF, LF / HF, and TP. Of these items, the measured vital sign values ​​are five items: pulse, heart rate variability, respiratory rate, LF, and HF. Furthermore, of these items, the index values ​​calculated by the measurement device 1 from the vital sign values ​​are two items: LF / HF and TP.

[0029] The video ID is identification information that identifies the video data that is the subject of measurement, and is common information with that stored in the video DB 121. From the video data identified by this video ID, the corresponding five vital signs are measured.

[0030] The pulse value is the number of times the blood vessels in the user's face beat per minute. The pulse is equivalent to the number of times the heart beats per minute, and is therefore synonymous with the heart rate.

[0031] Heart rate variability (HRV) is a numerical value that represents the fluctuation observed in the time interval between heartbeats. Heart rate variability reflects information related to biological control such as the autonomic nervous system, and its characteristics are known to change with age and the presence or absence of illness.

[0032] The respiration rate is the number of breaths per minute measured from an image of the user's face.

[0033] The LF and HF values ​​are both stored as numerical values ​​calculated as a result of power spectrum analysis of heartbeat interval fluctuations. For example, the LF value is stored as a value (integral value) obtained by summing the intensities of components in the low frequency region, and the HF value is stored as a value (integral value) obtained by summing the intensities of components in the high frequency region. It is said that the low frequency components correspond to blood pressure fluctuations, and the high frequency components correspond to respiratory fluctuations.

[0034] The measurement device 1 then calculates index values ​​used to identify the user's physical and mental state from the measured vital values, and stores the calculated index values ​​in the vital DB 122. In the example shown in FIG. 4, the vital DB 122 stores two calculated index values. LF / HF is a numerical value obtained by dividing LF by HF, and is an index value representing the "ratio of sympathetic nerves to parasympathetic nerves." TP is an index value representing the "level of activity of the entire autonomic nerves." TP is calculated, for example, using the following formula (1):

[0035] [Equation 1] TP = LF / HF + HF (1) In other words, TP is expressed as the sum of the value of "LF / HF", which is controlled by the sympathetic and parasympathetic nerves, and the value of "HF", which is controlled by the parasympathetic nerve.

[0036] <Configuration of the Physical Condition DB> FIG. 5 is a diagram showing an example of the physical condition DB 123. The physical condition DB 123 is a database that associates the above-mentioned ratio between the sympathetic and parasympathetic nerves and the overall activity level of the autonomic nerves with the type of physical condition (state of the nervous system) determined by these. Physical conditions are not determined by a single index value such as the ratio or activity level, but are classified into various types based on their combinations. The physical condition DB 123 stores the following items in association with each other: LF / HF, TP, physical condition ID, and physical condition name. The LF / HF and TP in the physical condition DB 123 each store a range of index values: "ratio between the sympathetic and parasympathetic nerves" and "overall activity level of the autonomic nerves." By combining these index value ranges, the types of physical conditions are divided on a plane with these two index values ​​as two axes.

[0037] The physical condition ID is identification information for identifying each physical condition associated with each set of LF / HF and TP ranges. The physical condition name is associated with the physical condition ID and is a character string that expresses the physical condition identified by the physical condition ID using human sensibilities. Examples of physical condition names include "tired," "sluggish," and "relaxed." Note that the physical condition name corresponding to a set of LF / HF and TP ranges may be a character string that changes into multiple expressions depending on the conditions. For example, the physical condition name corresponding to a set of ranges may be defined as "irritable" if the user is in a bad mood, and as "concentration mode" if the user is in a good mood.

[0038] FIG. 6 is a diagram showing an example of a plane partitioned based on the physical condition DB 123. The dashed vertical and horizontal lines in FIG. 6 represent reference values ​​for LF / HF and TP, respectively. The ranges of LF / HF and TP stored in the physical condition DB 123 may be determined based on these reference values. The reference values ​​may be representative values, such as the average, median, or mode, of the LF / HF and TP index values ​​calculated from vital signs measured from a certain number of people or more. The reference values ​​may also be representative values ​​of these index values ​​obtained by measuring the vital signs of a user who owns the measurement device 1 over a certain period of time.

[0039] <Configuration of recommendation DB> Fig. 7 is a diagram showing an example of the recommendation DB 124. The recommendation DB 124 is a database that stores various information to be recommended according to the user's physical condition. The recommendation DB 124 stores items such as a physical condition ID, a meal, nutrients, exercise, and herbal tea in association with each other. The physical condition ID is identification information for the identified physical condition of the user. The user's physical condition is identified by which section, defined in the physical condition DB 123, a set of index values ​​calculated from the measured pulse of the user belongs to.

[0040] Each item of food, nutrients, exercise, and herbal tea in the recommendation DB 124 is information to be recommended to a user whose physical condition is identified by the corresponding physical condition ID. This information is composed of text information, photos, videos, audio, etc. that explain the food, nutrients, exercise, and herbal tea that are appropriate for the user's physical condition at that time.

[0041] <Configuration of medical interview DB> Fig. 8 is a diagram showing an example of the medical interview DB 125. The medical interview DB 125 is a database that stores medical interview items to be administered to a specified user according to the user's physical condition. The medical interview DB 125 shown in Fig. 8 includes a physical condition ID list 1251 and a medical interview item list 1252.

[0042] The physical condition ID list 1251 is a list of physical condition IDs. The medical interview item list 1252 is a list associated with each physical condition ID listed in the physical condition ID list 1251, and lists medical interview items to be administered to a user with a physical condition identified by the physical condition ID. Suitable medical interview items include, for example, the Patient Health Questionnaire-9 (PHQ-9), the EuroQol-5 Dimension (EQ-5D), and the Quick Inventory of Depressive Symptomatology (QIDS-J).

[0043] <Configuration of Diagnostic Image DB> Fig. 9 is a diagram showing an example of the diagnostic image DB 126. The diagnostic image DB 126 is a database that stores diagnostic images to be presented to a user when a set of the user's index values ​​(i.e., LF / HF and TP) calculated multiple times satisfies a predetermined condition. Here, a diagnostic image is an image that is presented to the user to observe the user's reaction, which appears as changes in the user's facial expression, voice, etc. The diagnostic image DB 126 includes a condition ID list 1261 and a diagnostic image list 1262.

[0044] The condition ID list 1261 is a list of identification information for identifying each condition that is determined to be satisfied by the history of multiple consecutive index values, such as when the physical condition name "Exhausted" is identified five times in a row.

[0045] The diagnostic image list 1262 is a list associated with each condition ID listed in the condition ID list 1261, and lists diagnostic images to be presented to the user to observe their reaction when the user's index value history satisfies the condition identified by that condition ID. The multiple diagnostic images listed in this diagnostic image list 1262 are presented to the user in that order by the measuring device 1. By presenting the multiple diagnostic images in order and observing the user's reaction to viewing them, the measuring device 1 can, for example, identify the user's tendency toward dementia.

[0046] Furthermore, if the user's reaction observed when one of the diagnostic images is presented satisfies certain criteria, the diagnostic image is identified as an image that will have a certain effect on the user, and can be used to prevent the progression of dementia, etc.

[0047] <Functional Configuration of Measuring Device> A program according to the present invention is read into and executed by the processor 11 of the measuring device 1. By executing this program, the processor 11 realizes three functions. - First function: A function of plotting the state of the user's nervous system on a plane and outputting various information according to the area to which the plot belongs. - Second function: A function of asking questions to the user according to the evaluation results of the state of the user's nervous system, accepting responses, and outputting diagnostic results. - Third function: A function of presenting diagnostic images to the user according to the history of the state of the user's nervous system, judging the user's reaction, and identifying an image from the presented diagnostic images that will have a predetermined effect. The functional configuration of the processor 11 that realizes these three functions is described below.

[0048] <First Functional Configuration> Fig. 10 is a diagram showing an example of a first functional configuration of the measurement device 1. The processor 11 of the measurement device 1 reads and executes programs stored in the memory 12, thereby functioning as a video acquisition unit 111, a pulse measurement unit 112, a ratio calculation unit 113, an activity calculation unit 114, a plot unit 115, a region identification unit 116, and an information output unit 117 shown in Fig. 10 .

[0049] The video acquisition unit 111 acquires video data representing a video of a user's own face captured by the camera 16. The video data acquired by the video acquisition unit 111 is stored in the video DB 121 of the memory 12.

[0050] The pulse measurement unit 112 reads video data from the video DB 121 and measures vital signs including the pulse based on the video data. The vital signs measured by the pulse measurement unit 112 are stored in the vital DB 122 together with a video ID that identifies the video data, as shown in FIG. 4 .

[0051] The ratio calculation unit 113 calculates the ratio of the sympathetic nerves to the parasympathetic nerves (i.e., LF / HF) based on the pulse rate included in the measured vital signs. The calculated ratio is stored in the vital signs DB 122.

[0052] The activity level calculation unit 114 calculates the activity level of the entire autonomic nerves (i.e., TP) based on the pulse rate included in the measured vital sign value. The calculated activity level is stored in the vital sign DB 122.

[0053] The plotting unit 115 reads out a pair of LF / HF and TP from the vital DB 122, and reads out sections of physical condition IDs and physical condition names from the physical condition DB 123. Then, the plotting unit 115 plots the state of the user's nervous system indicated by the LF / HF and TP on a plane with these as the two axes.

[0054] The region specifying unit 116 specifies the region to which the state of the user's nervous system belongs from among the divided regions. This region indicates the user's physical condition.

[0055] The information output unit 117 extracts information corresponding to the area indicating the user's physical condition, which is specified by the area specifying unit 116 , from the recommendation DB 124 , and outputs the information to the display unit 15 .

[0056] <Second Functional Configuration> Fig. 11 is a diagram showing an example of a second functional configuration of the measurement device 1. The processor 11 of the measurement device 1 reads and executes programs stored in the memory 12, thereby functioning as a condition evaluation unit 1181, a medical questionnaire presentation unit 1182, a response acceptance unit 1183, and a diagnosis result output unit 1184 shown in Fig. 11 .

[0057] The state evaluation unit 1181 reads out a set of LF / HF and TP from the vital DB 122, and evaluates the state of the nervous system of the user by referring to the section of the physical condition DB 123. Here, the state of the nervous system is the latest physical condition.

[0058] The medical interview presentation unit 1182 extracts, from the medical interview DB 125, a medical interview item list 1252 corresponding to the state of the user's nervous system evaluated by the state evaluation unit 1181. Then, the medical interview presentation unit 1182 causes the display unit 15 to sequentially display (present) the medical interview items listed in the extracted medical interview item list 1252.

[0059] The answer receiving unit 1183 receives a user's operation from the operation unit 14 and receives the user's answer to the above-mentioned medical question.

[0060] The diagnosis result output unit 1184 outputs the result of the diagnosis made based on the user's answer accepted by the answer accepting unit 1183 by displaying it on the display unit 15. This diagnosis may be made by the processor 11. When the processor 11 makes the diagnosis, the memory 12 may store in advance a combination of the user's vital sign information, its history, trends, and the user's answers to the medical interviews, as well as the corresponding diagnosis results. The memory 12 may also store a machine learning model related to the above diagnosis that can be used by the processor 11.

[0061] This diagnosis may also be performed by a doctor. In this case, the diagnosis result output unit 1184 transmits information on the user's vital signs and information on the answers to the medical interview to a terminal device used by the doctor via the communication unit 13. Then, the diagnosis result output unit 1184 receives information indicating the diagnosis result input by the doctor from the terminal device via the communication unit 13.

[0062] <Third Functional Configuration> Fig. 12 is a diagram showing an example of a third functional configuration of measuring device 1. Processor 11 of measuring device 1 reads and executes programs stored in memory 12, thereby functioning as state evaluation unit 1181, image presentation unit 1191, response determination unit 1192, image specification unit 1193, and specified image output unit 1194 shown in Fig. 12 .

[0063] The state evaluation unit 1181 reads out the pairs of LF / HF and TP from the vital DB 122 and evaluates the state of the user's nervous system by referring to the sections of the physical condition DB 123. Furthermore, in this third functional configuration, the state evaluation unit 1181 determines whether the history of the user's LF / HF and TP pairs calculated multiple times satisfies a predetermined condition. Here, the state of the nervous system is the result of determining whether the history of the physical condition indicated by the above pairs satisfies the predetermined condition.

[0064] When the state evaluation unit 1181 determines that the history satisfies a predetermined condition, the image presentation unit 1191 extracts a diagnostic image list 1262 corresponding to the predetermined condition from the diagnostic image DB 126. Then, the image presentation unit 1191 causes the display unit 15 to sequentially display (present) the diagnostic images included in the extracted diagnostic image list 1262.

[0065] The response determination unit 1192 acquires video of the user's face as it looks at the diagnostic images, captured by the camera 16, while the display unit 15 sequentially displays the diagnostic images to the user under the control of the image presentation unit 1191. The response determination unit 1192 then identifies the user's response from changes in facial contours, changes in blood flow, etc., and determines whether the response satisfies predetermined determination conditions.

[0066] When the reaction determination section 1192 determines that the reaction of the user satisfies a predetermined determination condition, the image specification section 1193 specifies the diagnostic image that the user has been looking at.

[0067] The specific image output unit 1194 outputs the diagnostic image specified by the image specifying unit 1193 by displaying it on the display unit 15 as a specific image.

[0068] <Operation of Measuring Device> <Operation of First Function> Figure 13 is a diagram showing an example of the flow of operations relating to the first function of measuring device 1. Processor 11 of measuring device 1 reads and executes a program according to the present invention from memory 12. When this program is started, it displays a screen prompting the user to take a picture of their own face, a so-called selfie.

[0069] Fig. 14 is a diagram showing an example of a display area R0. As shown in Fig. 14, a display unit 15 and a camera 16 are provided on the front of the measurement device 1. As described above, the display unit 15 has a display screen such as a liquid crystal display. The upper part of this display screen is provided with a general information display area for displaying radio wave reception sensitivity, remaining battery power, time, etc., and the lower part is provided with a general operation area for accepting operations on the operating system. The display area R0 is provided between these areas.

[0070] 15 is a diagram showing an example of a display when shooting a selfie video. When the processor 11 prompts the user to take a selfie, a cancel icon area R1, an explanation area R2, a face shooting area R3, and a measurement start button area R4 are displayed in the display area R0, as shown in FIG.

[0071] The cancel icon area R1 is an area for accepting an instruction to cancel selfie video shooting.

[0072] The explanation area R2 is an area for displaying text explaining the procedure for taking a selfie video. In the example shown in Fig. 15, the explanation area R2 displays the following explanation: "Hold the smartphone directly in front of your face, align your face with the circle, and press the 'Start measurement' button."

[0073] The face shooting area R3 is an area that indicates the range in which the user's own face should fit within the image area that simultaneously displays the selfie video being shot.

[0074] The measurement start button area R4 is an area that displays a button for starting selfie video shooting.

[0075] Returning to the description of the operation shown in FIG. 13 , when the user presses the measurement start button in response to the above-described screen to start taking a selfie, the processor 11 acquires video provided by the camera 16 (step S101). The processor 11 then measures the user's pulse from the acquired video (step S102). Note that in this embodiment, the user starts a selfie to capture a picture of their own face, but the subject of the photo is not limited to the face and may be any image of the user's own body. In other words, step S102 is an example of a step in which the user's pulse is measured based on a video of the user's body.

[0076] When the pulse rate is measured, the processor 11 calculates LF / HF (the ratio of sympathetic nerve activity to parasympathetic nerve activity) from the pulse rate (step S103).

[0077] The processor 11 also calculates TP (the overall activity of the autonomic nerves) from the measured pulse (step S104).

[0078] Therefore, step S103 is an example of a step of calculating the ratio of sympathetic nerve activity to parasympathetic nerve activity from the measured pulse. Also, step S104 is an example of a step of calculating the activity level of the entire autonomic nerves from the measured pulse. Note that the order in which steps S103 and S104 are executed may be reversed or may not be predetermined due to the execution of parallel processing.

[0079] 16 is a diagram showing an example of a screen displayed when measuring pulse rate, etc. When the user finishes taking a selfie video, the processor 11 measures the pulse rate based on the captured video, as described above, and calculates the LF / HF and TP based on the pulse rate. At this time, as shown in FIG. 16 , the processor 11 may control the display unit 15 to display a graph showing the change in LF / HF or pulse rate over time in the display area R0.

[0080] Returning to the explanation of the operation shown in Fig. 13, once LF / HF and TP have been calculated, the processor 11 controls the display unit 15 to plot these values ​​in the display area R0 on a plane having the LF / HF and TP values ​​as its respective axes (step S105). In other words, step S105 is an example of a step of plotting the calculated ratio and the calculated activity level on a plane having the ratio of the sympathetic nerves to the parasympathetic nerves and the activity level of the entire autonomic nerves as its two axes.

[0081] 17 is a diagram showing an example of a plot of LF / HF and TP. When step S105 described above is executed, a graph with LF / HF on the vertical axis and TP on the horizontal axis is displayed in the display area R0 of the display unit 15, as shown in FIG. 17. Then, for example, a plot P1 showing LF / HF and TP is drawn on this graph, as shown in FIG.

[0082] For example, the user looks at plot P1 on this graph and recognizes that his or her physical condition is identified by the physical condition ID "N8." Note that the graph may display the physical condition name associated with the physical condition ID in the physical condition DB 123 instead of or in addition to this physical condition ID.

[0083] Returning to the description of the operation shown in Fig. 13, once LF / HF and TP have been plotted, the processor 11 identifies an area containing coordinates of the plot indicating these index values ​​(step S106). In other words, step S106 is an example of a step of identifying an area on a plane containing coordinates indicating the ratio and activity level plotted on the plane having two axes representing the ratio of the sympathetic nerves to the parasympathetic nerves and the activity level of the entire autonomic nerves.

[0084] Then, the processor 11 refers to the recommendation DB 124, extracts recommendation information such as food recommendation information corresponding to the physical condition ID of the specified area, and outputs the recommendation information corresponding to the user's physical condition (step S107). In other words, step S107 is an example of a step of outputting information corresponding to the specified area.

[0085] <Operation of Second Function> Figure 18 is a diagram showing an example of the flow of operation related to the second function of the measuring device 1. The processor 11 of the measuring device 1 reads the latest calculated LF / HF and TP from the vital DB 122 of the memory 12. Then, based on these read index values, the state of the user's nervous system is evaluated (step S201). In other words, step S201 is an example of a step of evaluating the state of the user's nervous system based on the calculated ratio and activity level.

[0086] Next, the processor 11 extracts the medical interview items corresponding to the evaluated condition from the medical interview DB 125, and displays them in order on the display unit 15 to present the medical interview (step S202).

[0087] Next, the processor 11 accepts the user's response to the medical questionnaire by acquiring the response from the user who has viewed the presented medical questionnaire via the operation unit 14 (step S203). In other words, steps S202 and S203 are examples of steps in which a medical questionnaire corresponding to the evaluated condition is presented and the user's response to the medical questionnaire is accepted.

[0088] Then, the processor 11 diagnoses the user's physical and mental condition based on the evaluated condition and the received answer from the user, and outputs the diagnosis result by displaying it on the display unit 15 (step S204). In other words, step S204 is an example of a step of diagnosing the user based on the evaluated condition and the received answer, and outputting the diagnosis result.

[0089] With this second function, the measuring device 1 roughly grasps the state of the autonomic nervous system before the interview, and only the interview items corresponding to that state are presented, thereby making the interview and diagnosis more efficient and reducing the time spent on them.

[0090] 19 is a diagram showing an example of the flow of operations related to the third function of the measuring device 1. The processor 11 of the measuring device 1 refers to the vital DB 122 and extracts the index values ​​calculated multiple times, i.e., pairs of ratio and activity level (step S301).

[0091] Then, the processor 11 determines whether the extracted ratio-activity combination satisfies predetermined conditions (hereinafter referred to as presentation conditions) for presenting a diagnostic image (step S302).

[0092] When it is determined that the ratio-activity pair does not satisfy the presentation condition (step S302; NO), the processor 11 ends the processing.

[0093] On the other hand, if it is determined that the ratio / activity level pair satisfies the presentation condition (step S302; YES), the processor 11 causes the display unit 15 to sequentially display and present the diagnostic images associated with the satisfied presentation condition (step S303). In other words, step S303 is an example of a step in which multiple diagnostic images are presented to the user when the ratio / activity level pair calculated from the user's pulse measured at least multiple times satisfies a predetermined condition.

[0094] When step S303 is executed to present a plurality of diagnostic images in a predetermined order, the processor 11 analyzes the user's reaction based on video captured by the camera 16 of the user viewing the diagnostic images. The processor 11 then determines whether the user's reaction exceeds, for example, a certain level (step S304). In other words, step S304 is an example of a step for determining whether the user's reaction to viewing the presented diagnostic images satisfies a predetermined determination condition.

[0095] If the user's reaction satisfies the predetermined judgment condition, the processor 11 identifies the diagnostic image that the user was looking at at that time (step S305). That is, step S305 is an example of a step of identifying the diagnostic image that the user was looking at when it is judged that the user's reaction to the presented diagnostic image satisfies the judgment condition.

[0096] A diagnostic image that satisfies the criteria is an image that, when presented to the user, elicits a certain level of user response and is therefore expected to have a certain effect when displayed to the user. The criteria are determined, for example, based on characteristics such as LF / HF, TP, pulse rate, facial expression, blood flow, and complexion that indicate a good user response and a high level of happiness when viewing the diagnostic image. The processor 11 then determines whether the conditions for displaying the identified diagnostic image (i.e., the specific image) (hereinafter referred to as the display conditions) are met (step S306). Examples of the display conditions include, for example, not operating the measuring device 1 for a certain period of time, or being asleep.

[0097] If it is determined that the display condition is not satisfied (step S306; NO), the processor 11 ends the process.

[0098] On the other hand, when it is determined that the display condition is satisfied (step S306; YES), the processor 11 outputs the identified diagnostic image (i.e., the identified image) by displaying it on the display unit 15 (step S307). That is, step S307 is an example of a step of outputting the identified diagnostic image.

[0099] In this case, a specific image is output when the display conditions are met, so step S307 is an example of a step in which a specified diagnostic image is output when the specified display conditions are met.

[0100] With this third function, the measuring device 1 can present diagnostic images to a user who is suspected of having dementia because, for example, the state of the autonomic nervous system satisfies certain conditions, and extracts diagnostic images that have a good response as specific images and displays these images in a focused manner, which can be expected to have a therapeutic and ameliorative effect on dementia.

[0101] Through the above-described operations, this program allows the user (subject) to easily check their own detailed condition in a variety of ways. By using this program, the user can check their own physical and mental condition in a two-dimensional area, enabling, for example, early detection of abnormalities and stress checks. Furthermore, this program can be used, for example, for monitoring the elderly, managing driver fatigue, providing health promotion information, supporting exercise habits to prevent adult diseases, online medical care, clinical trials, and the like.

[0102] <Modifications> The above is a description of the embodiment, but the content of this embodiment can be modified as follows: In addition, the following modifications may be combined with each other.

[0103] In the above-described embodiment, the measuring device 1 has the processor 11 configured as a CPU, but the control means for controlling the measuring device 1 may have other configurations.

[0104] That is, the measuring device 1 may have various types of processors as the processor 11, such as a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), a programmable logic device, etc., in addition to a CPU.

[0105] The operations of the processors in the above-described embodiments may not only be performed by a single processor, but may also be performed by a plurality of processors located at physically separate locations working together.

[0106] Furthermore, the order of the operations of the processor is not limited to the order described in the above embodiment, and may be changed as appropriate.

[0107] In the above-described embodiment, the index values ​​of the "ratio of sympathetic and parasympathetic nerves" and the "level of activity of the entire autonomic nerves" were calculated from the measured pulse rate itself, but these index values ​​may also be calculated based on the pulse rate after a predetermined processing process has been performed.

[0108] For example, the processor 11 of the measuring device 1 may generate data indicating the same phase and opposite phase from the time series data of the measured pulse, and generate data (referred to as concatenated data) by alternating these data.

[0109] The processor 11 may then calculate the two index values ​​described above from the generated linked data. That is, the program executed by the processor 11 in this modified example is an example of a program for causing a computer to execute the steps of generating linked data by alternately linking data indicating the same phase and opposite phase of the measured pulse, and calculating the ratio and activity level from the generated linked data. According to the program shown in this modified example, by generating linked data by alternately linking the same phase and opposite phase data and calculating the index value, the measurement time can be shortened without reducing the accuracy of the index calculation.

[0110] In the above-described embodiment, the measurement device 1 is a single mobile terminal, but some of the functions may be realized by an external device.

[0111] 20 is a diagram showing an example of the overall configuration of a measurement system 9 according to a modified example. The measurement system 9 shown in FIG.

[0112] The measurement device 1 is a mobile terminal, such as a smartphone, and the server device 2 is a computer connected to one or more measurement devices 1 so as to be able to communicate with them.

[0113] The communication line 3 is a line that communicatively connects the measurement device 1 and the server device 2. The communication line 3 may be, for example, a LAN (Local Area Network), a WAN (Wide Area Network), the Internet, or a combination of these.

[0114] The number of measurement devices 1 and the number of server devices 2 in the measurement system 9 are not limited to those shown in Fig. 20. For example, the server device 2 may be configured as a cluster system in which multiple devices share functions.

[0115] Fig. 21 is a diagram showing an example of the configuration of a server device 2 according to a modified example. As shown in Fig. 21, the server device 2 includes a processor 21, a memory 22, and a communication unit 23. These are connected by a bus.

[0116] The processor 21 reads and executes a program stored in the memory 22 to control each part of the measurement device 1. The processor 21 is, for example, a CPU.

[0117] The communication unit 23 is a communication circuit that connects the server device 2 to other devices via the communication line 3 in a wired or wireless manner so that communication is possible.

[0118] The memory 22 is a storage means for storing an operating system, various programs, data, etc. that are read into the processor 21. The memory 22 includes a RAM and a ROM.

[0119] The memory 22 may include a solid state drive, a hard disk drive, etc. The memory 22 also stores a video DB 221, a vital sign DB 222, a physical condition DB 223, a recommendation DB 224, a medical interview DB 225, and a diagnostic image DB 226. These are databases common to the video DB 121, the vital sign DB 122, the physical condition DB 123, the recommendation DB 124, the medical interview DB 125, and the diagnostic image DB 126 stored in the memory 12 of the measuring device 1 shown in the embodiment.

[0120] The processor 21 of the server device 2 may implement any of the functional configurations implemented by the processor 11 in the embodiments in place of the processor 11. For example, the processor 21 may function as a video acquisition unit that acquires video from the measurement device 1. In this case, the processor 11 of the measurement device 1 may transmit the video to the server device 2 via the communication unit 13, the communication line 3, and the communication unit 23. The processor 21 may also function as a pulse measurement unit that measures the user's pulse based on the acquired video, a ratio calculation unit that calculates the above-mentioned ratio based on the measured pulse, and an activity calculation unit that calculates the activity level.

[0121] In the above-described embodiment, the processor 11 of the measuring device 1 realizes the first function, the second function, and the third function by reading and executing the program according to the present invention, but it is not necessary to realize all of these functions. Also, the processor 11 may realize only part of each function, rather than all of them.

[0122] For example, the processor 11 may not control the display unit 15 or the like to plot the region to which the state of the user's nervous system belongs on a plane. In this case, the processor 11 may, for example, identify the region to which the state of the user's nervous system belongs and output various pieces of information corresponding to the identified region.

[0123] The processor 11 may also identify the state of the user's nervous system using not only the LF and HF values ​​but also the VLF value, which is a very low frequency region, and express the state as, for example, a point on a three-dimensional space indicated by three axes. The processor 11 may also express the state of the user's nervous system in a tabular format. The processor 11 may also express the state of the user's nervous system auditorily, tactilely, or the like, in addition to or instead of a visual expression. That is, the program according to the present invention may be implemented as long as it allows the subject to confirm their own detailed state in various ways. In short, the method according to the present invention may include the steps of measuring the user's pulse based on an image of the user's body, calculating the ratio of the sympathetic nervous system to the parasympathetic nervous system from the measured pulse, calculating the overall activity of the autonomic nervous system from the measured pulse, and plotting the calculated ratio and the calculated activity on a plane.

[0124] The program for executing each of the above steps may be provided in a state stored in a computer-readable recording medium such as a magnetic recording medium such as a magnetic tape or a magnetic disk, an optical recording medium such as an optical disk, a magneto-optical recording medium, a semiconductor memory, etc. This program may also be downloaded via a communication line such as the Internet.

[0125] 1...measuring device, 11...processor, 111...video acquisition unit, 112...pulse measurement unit, 113...ratio calculation unit, 114...activity calculation unit, 115...plotting unit, 116...area identification unit, 117...information output unit, 1181...condition evaluation unit, 1182...questionnaire presentation unit, 1183...answer reception unit, 1184...diagnosis result output unit, 1191...image presentation unit, 1192...response determination unit, 1193...image identification unit, 1194...specific image output unit, 12...memory, 121...video DB, 122...vital DB, 123...physical condition DB, 124...recommendation DB, 125...questionnaire DB, 1251...physical condition ID list, 12 52...list of medical interview items, 126...diagnostic image DB, 1261...list of condition IDs, 1262...list of diagnostic images, 13...communication unit, 14...operation unit, 15...display unit, 16...camera, 17...observation unit, 2...server device, 21...processor, 22...memory, 221...video DB, 222...vital DB, 223...physical condition DB, 224...recommendation DB, 225...medical interview DB, 226...diagnostic image DB, 23...communication unit, 3...communication line, 9...measurement system, P1...plot, R0...display area, R1...cancel icon area, R2...explanation area, R3...face capture area, R4...measurement start button area.

Claims

1. A program for causing a computer to execute the steps of: measuring a user's pulse based on an image of the user's body; calculating the ratio of sympathetic to parasympathetic nerves from the measured pulse; calculating the activity level of the entire autonomic nerves from the measured pulse; and plotting the calculated ratio and the calculated activity level on a plane with the ratio of sympathetic to parasympathetic nerves and the activity level of the entire autonomic nerves as the two axes.

2. The program according to claim 1, which causes the computer to further execute the steps of: identifying an area on the plane that includes coordinates indicating the ratio and the activity plotted on the plane; and outputting information corresponding to the identified area.

3. The program described in claim 1, which causes the computer to further execute the steps of: evaluating the state of the user's nervous system based on the calculated ratio and activity level; presenting a medical questionnaire corresponding to the evaluated state and accepting the user's responses to the medical questionnaire; and diagnosing the user based on the state and the responses and outputting the diagnostic results.

4. The program described in claim 3, which causes the computer to further execute the steps of: presenting a plurality of diagnostic images to the user when a set of the ratio and the activity level calculated from the user's pulse measured at least multiple times satisfies a predetermined condition; determining whether the user's reaction to the presented diagnostic images satisfies a predetermined judgment condition; identifying the diagnostic image that the user was looking at when it is determined that the reaction satisfies the judgment condition; and outputting the identified diagnostic image.

5. The program according to claim 4, further causing the computer to execute the step of outputting the identified diagnostic image when a predetermined display condition is satisfied.

6. The program according to claim 1, further causing the computer to execute the steps of: generating concatenated data by alternately concatenating data indicating the same phase and opposite phase of the measured pulse; and calculating the ratio and the activity level from the generated concatenated data.

Images