program

Abstract

To provide a program that allows a measurement object person to easily check his or her state in detail using a smartphone or the like by various expressions.SOLUTION: A moving image acquisition unit 111 acquires moving image data showing a moving image that a user has taken of his or her face by a camera 16. The acquired moving image data is accumulated in a moving image DB 121 of a memory 12. A pulse measuring unit 112 reads out the moving image data from the moving image DB 121, and measures vital values including a pulse on the basis of the moving image data. A ratio calculation unit 113 calculates a ratio of a sympathetic nerve to a parasympathetic nerve on the basis of the pulse included in the measured vital values. An activity calculation unit 114 calculates the activity of the entire autonomous nerve on the basis of the pulse included in the measured vital values. A plotting part 115 plots a state of the user's nervous system indicated by LF / HF and TP on a plane with the LF / HF and the TP as two axes.SELECTED DRAWING: Figure 10

Description

[Technical Field]

[0001] The present invention relates to a program for outputting the state of a user's nervous system based on a moving image. [Background technology]

[0002] Patent Document 1 discloses an image processing method for stress monitoring that acquires face image data using an RGB camera, creates hemoglobin image data from the face image data using pigment component separation, creates heart rate data based on the hemoglobin image data, creates heart rate interval data based on the heart rate data, and creates heart rate variability spectrogram data by performing frequency conversion on the heart rate interval data. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2017-29318 Summary of the Invention [Problem to be solved by the invention]

[0004] The technology described in Patent Document 1 can determine whether a person being measured is in a resting state or a stressed state based on an image of the person's face. However, a person's physical and mental state may not be expressed by a single quantity, such as the intensity of stress. In other words, the technology described in Patent Document 1 can only determine whether the person being measured is in a resting state or a stressed state, and cannot identify various states.

[0005] One of the objects of the present invention is to provide a program that allows a person to be measured to easily check their own detailed condition in a variety of ways using a smartphone or the like. [Means for solving the problem]

[0006] In one aspect, the present invention provides a program for causing a computer to execute the steps of measuring a user's pulse based on a video of the user's body, calculating the ratio between the sympathetic and parasympathetic nerves from the measured pulse, calculating the activity level of the entire autonomic nerves from the measured pulse, and plotting the calculated ratio and the calculated activity level on a plane having the ratio between the sympathetic and parasympathetic nerves and the activity level of the entire autonomic nerves as two axes. [Effects of the Invention]

[0007] According to the present invention, the subject can check his / her own detailed condition in a variety of ways. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of a measurement device 1. [Figure 2] FIG. 10 is a diagram showing an example of a video DB 121. [Figure 3] FIG. 1 is a diagram for explaining an example of the concept of video data. [Figure 4] FIG. 10 is a diagram showing an example of a vital sign DB 122. [Figure 5] FIG. 10 is a diagram showing an example of a physical condition DB 123. [Figure 6] FIG. 10 is a diagram showing an example of a plane partitioned based on the physical condition DB 123. [Figure 7] FIG. 10 is a diagram showing an example of a recommendation DB 124. [Figure 8] FIG. 10 is a diagram showing an example of a medical interview DB 125. [Figure 9] FIG. 2 is a diagram showing an example of a diagnostic image DB 126. [Figure 10] FIG. 2 is a diagram showing an example of a first functional configuration of the measurement device 1. [Figure 11] FIG. 4 is a diagram showing an example of a second functional configuration of the measurement device 1. [Figure 12] FIG. 10 is a diagram showing an example of a third functional configuration of the measurement device 1. [Figure 13] FIG. 4 is a diagram showing an example of the flow of operations relating to a first function of the measurement device 1. [Figure 14] FIG. 4 is a diagram showing an example of a display area R0. [Figure 15] FIG. 10 is a diagram showing an example of a display when shooting a selfie video. [Figure 16] FIG. 10 is a diagram showing an example of a screen displayed when measuring pulse rate, etc. [Figure 17] FIG. 1 shows an example of a plot of LF / HF and TP. [Figure 18] FIG. 10 is a diagram showing an example of the flow of operations relating to the second function of the measurement device 1. [Figure 19] FIG. 10 is a diagram showing an example of the flow of operations relating to the third function of the measurement device 1. [Figure 20] FIG. 10 is a diagram showing an example of the overall configuration of a measurement system 9 according to a modified example. [Figure 21] FIG. 10 is a diagram showing an example of the configuration of a server device 2 according to a modified example. DETAILED DESCRIPTION OF THE INVENTION

[0009] <Embodiment> <Configuration of measurement device> FIG. 1 is a diagram showing an example of the configuration of a measuring device 1. The measuring device 1 is a device that measures the physical condition of a user based on an image of the user's face (hereinafter also referred to as a "face image"). The measuring device 1 shown in FIG. 1 is a mobile terminal, such as a smartphone. Furthermore, the "physical condition" referred to here mainly refers to the state and condition of the mind and body resulting from disturbances in the autonomic nervous system.

[0010] As shown in Fig. 1, the measuring device 1 has a processor 11, a memory 12, a communication unit 13, an operation unit 14, a display unit 15, and a camera 16. These are connected by a bus. The measuring device 1 may also have an observation unit 17, which is indicated by a dashed line in Fig. 1.

[0011] The processor 11 reads and executes a program stored in the memory 12 to control each part of the measurement device 1. The processor 11 is, for example, a CPU (Central Processing Unit).

[0012] The communication unit 13 is a communication circuit that communicatively connects the measuring device 1 to other devices via a wired or wireless connection. For example, the communication unit 13 may have a circuit that complies with a wireless communication system standard such as IMT-2000, IMT-Advanced, or IMT-2020. Alternatively, for example, the communication unit 13 may have a circuit that complies with a wireless LAN standard such as IEEE802.11.

[0013] Furthermore, for example, the communication unit 13 may include a module that realizes near field communication (NFC). Examples of NFC standards include ISO / IEC18092 (NFCIP-1), ISO / IEC14443, ISO / IEC15693, and IEEE802.15.

[0014] The operation unit 14 is equipped with operation buttons, a touch panel, and other operators for issuing various instructions, and receives an operation and sends a signal corresponding to the operation content to the processor 11. This operation is, for example, pressing a button, making a gesture on the touch panel, or the like.

[0015] The operation unit 14 may have a microphone for collecting voice. In this case, the processor 11 may perform voice recognition processing on the voice data representing the user's voice collected by the microphone, and accept the recognition result as the user's operation.

[0016] The display unit 15 has a display screen such as a liquid crystal display, and displays images under the control of the processor 11. A transparent touch panel of the operation unit 14 may be placed on top of the display screen.

[0017] The display unit 15 has a display screen that displays images and the like that are visually perceived by the user, but may also have, in addition to this, for example, a speaker that emits sound.

[0018] The camera 16 is an imaging means, such as a digital still camera, that captures an image of the surroundings of the measuring device 1. The camera 16 includes an optical system such as a lens, as well as an imaging element such as a CMOS (Complementary Metal Oxide Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0019] The camera 16 shown in Fig. 1 has a common configuration with, for example, the "RGB camera" of Patent Document 1 mentioned above. That is, this camera 16 is a device that acquires image data by arranging a plurality of detection elements, each of which has sensitivity to light intensity in a wavelength range corresponding to each of the colors R (red), G (green), and B (blue), as one pixel unit. The wavelength ranges corresponding to each color are preferably, for example, 580 nm to 680 nm for R (red), 500 nm to 630 nm for G (green), and 410 nm to 530 nm for B (blue).

[0020] The camera 16 may have a polarizing plate as part of its optical system. This polarizing plate is placed in the direction in which the camera 16 photographs the object. This allows the camera 16 to remove reflected components from the surface of the object.

[0021] Furthermore, the camera 16 may detect light in a wavelength range other than the above-mentioned ranges. For example, the camera 16 may be a five-band camera that adds two sensitivities, cyan (C) and orange (O), to a normal RGB camera.

[0022] 1 is a sensor or the like that observes physical quantities in the environment surrounding the user of the measurement device 1. This observation unit 17 is, for example, a barometer, a hygrometer, a thermometer, etc. Note that the measurement device 1 does not necessarily have to have this observation unit 17.

[0023] The memory 12 is a storage means for storing an operating system, various programs, data, etc. that are loaded into the processor 11. The memory 12 includes a RAM (Random Access Memory) and a ROM (Read Only Memory).

[0024] The memory 12 may include a solid state drive, a hard disk drive, etc. The memory 12 shown in FIG.

[0025] <Video DB configuration> Fig. 2 is a diagram showing an example of the video DB 121. The video DB 121 is a database that stores data of videos (also called video data) that capture a user's face. The video DB 121 shown in Fig. 2 has a table structure that stores values ​​for each item that is associated with each other. The video DB 121 shown in Fig. 2 has the following items: video ID, start date and time, length, and video data.

[0026] In this video DB 121, the value of the video data is the video data itself. The value of the video ID is identification information for uniquely identifying the corresponding video data. The value of the start date and time is information on the date and time when filming of the corresponding video data started (i.e., the start date and time). The value of the length is a numerical value indicating the length of the corresponding video data. The unit of this length is, for example, a time unit such as minutes or seconds.

[0027] FIG. 3 is a diagram illustrating an example of the concept of video data. Video data is made up of multiple still images captured at regular intervals. One frame of these multiple still images is called a frame. The number of still images captured per unit time is called the frame rate. As shown in FIG. 3, video data is made up of multiple ordered frames, each of which is associated with image data indicating the still image captured at each instant. Note that video data may be compressed as long as the image data for each frame can be extracted and acquired.

[0028] <Vital DB configuration> FIG. 4 is a diagram showing an example of the vital DB 122. The vital DB 122 is a database that stores vital signs (hereinafter also referred to as vitals) that indicate the user's vital signs and physical condition measured based on video data. The vital DB 122 shown in FIG. 4 has a table structure that stores values ​​for each associated item. The vital DB 122 shown in FIG. 4 has the following items: video ID, pulse, heart rate variability, respiratory rate, LF, HF, LF / HF, and TP. Of these items, the measured vital values ​​are five items: pulse, heart rate variability, respiratory rate, LF, and HF. Furthermore, of these items, the index values ​​calculated by the measurement device 1 from the vital values ​​are two items: LF / HF and TP.

[0029] The moving image ID is identification information that identifies the moving image data that was the subject of measurement, and is the same information as that stored in the moving image DB 121. From the moving image data identified by this moving image ID, the corresponding five vital signs are measured.

[0030] The pulse value is the number of times the blood vessels in the user's face beat per minute. The pulse is equivalent to the number of times the heart beats per minute, so it is synonymous with the heart rate.

[0031] Heart rate variability (HRV) is a numerical value that represents the fluctuations observed in the time intervals between heartbeats. Heart rate variability reflects information related to biological control, such as the autonomic nervous system, and its characteristics are known to change with age and the presence or absence of illness.

[0032] Respiratory rate is the number of breaths per minute measured from the user's facial image.

[0033] The LF and HF values ​​are both stored as numerical values ​​calculated as a result of power spectrum analysis of heartbeat interval fluctuations. For example, the LF value is stored as the sum (integral value) of the intensities of low-frequency components, and the HF value is stored as the sum (integral value) of the intensities of high-frequency components. It is said that the low-frequency components correspond to blood pressure fluctuations, and the high-frequency components correspond to respiratory fluctuations.

[0034] The measurement device 1 then calculates index values ​​used to identify the user's physical and mental state from the measured vital values, and stores the calculated index values ​​in the vital DB 122. In the example shown in FIG. 4, the vital DB 122 stores two calculated index values. LF / HF is a numerical value obtained by dividing LF by HF, and is an index value representing the "ratio of sympathetic nerves to parasympathetic nerves." TP is an index value representing the "level of activity of the entire autonomic nerves." TP is calculated, for example, using the following formula (1):

[0035] [Number 1] TP=LF / HF+HF ……(1) That is, TP is expressed as the sum of the value of "LF / HF", which is controlled by the sympathetic and parasympathetic nerves, and the value of "HF", which is controlled by the parasympathetic nerves.

[0036] <Configuration of health database> FIG. 5 is a diagram showing an example of the physical condition DB 123. The physical condition DB 123 is a database that associates the above-mentioned ratio between the sympathetic and parasympathetic nerves and the activity level of the entire autonomic nerves with the type of physical condition (state of the nervous system) determined by them. A physical condition is not determined by only one index value each of the ratio, activity level, etc., but is classified into various types by combinations of these. The physical condition DB 123 stores items such as LF / HF, TP, physical condition ID, and physical condition name in association with each other. The LF / HF and TP in the physical condition DB 123 store ranges of index values, namely, "ratio between the sympathetic and parasympathetic nerves" and "activity level of the entire autonomic nerves," respectively. By combining the ranges of these index values, the types of physical condition are divided on a plane with these two index values ​​as two axes.

[0037] The physical condition ID is identification information for identifying each physical condition associated with each set of LF / HF and TP ranges. The physical condition name is associated with a physical condition ID and is a character string that expresses the physical condition identified by the physical condition ID using human sensibilities. Examples of physical condition names include "tired," "sluggish," and "relaxed." Note that the physical condition name corresponding to a set of LF / HF and TP ranges may be a character string that changes into multiple expressions depending on the conditions. For example, the physical condition name corresponding to a set of ranges may be defined as "irritable" if the user is in a bad mood, and as "concentration mode" if the user is in a good mood.

[0038] FIG. 6 is a diagram showing an example of a plane partitioned based on the physical condition DB 123. The dashed vertical and horizontal lines in FIG. 6 represent reference values ​​for LF / HF and TP, respectively. The ranges of LF / HF and TP stored in the physical condition DB 123 may be determined based on these reference values. The reference values ​​may be representative values ​​such as the average, median, or mode of the LF / HF and TP index values ​​calculated from vital signs measured from a certain number of people or more. The reference values ​​may also be representative values ​​of these index values ​​obtained by measuring the vital signs of a user who owns the measurement device 1 over a certain period of time.

[0039] <Recommendation DB configuration> FIG. 7 is a diagram showing an example of the recommendation DB 124. The recommendation DB 124 is a database that stores various information to be recommended according to the user's physical condition. The recommendation DB 124 stores items such as a physical condition ID, food, nutrients, exercise, and herbal tea in association with each other. The physical condition ID is identification information for the identified physical condition of the user. The user's physical condition is identified by which section, defined in the physical condition DB 123, a set of index values ​​calculated from the measured pulse of the user belongs to.

[0040] Each item of food, nutrients, exercise, and herbal tea in the recommendation DB 124 is information to be recommended to a user whose physical condition is identified by the corresponding physical condition ID. This information is composed of text information, photos, videos, audio, etc. that explain the food, nutrients, exercise, and herbal tea that are appropriate for the user's physical condition at that time.

[0041] <Configuration of medical interview database> Fig. 8 is a diagram showing an example of medical interview DB 125. Medical interview DB 125 is a database that stores medical interview items to be administered to a specified user according to the user's physical condition. Medical interview DB 125 shown in Fig. 8 includes a physical condition ID list 1251 and a medical interview item list 1252.

[0042] The physical condition ID list 1251 is a list of physical condition IDs. The medical interview item list 1252 is a list associated with each physical condition ID listed in the physical condition ID list 1251, and lists medical interview items to be administered to a user with a physical condition identified by the physical condition ID. Suitable medical interview items include, for example, the Patient Health Questionnaire-9 (PHQ-9), the EuroQol-5 Dimension (EQ-5D), and the Quick Inventory of Depressive Symptomatology (QIDS-J).

[0043] <Configuration of diagnostic image database> 9 is a diagram showing an example of the diagnostic image DB 126. The diagnostic image DB 126 is a database that stores diagnostic images to be presented to a user when a set of the user's index values ​​(i.e., LF / HF and TP) calculated multiple times satisfies a predetermined condition. Here, a diagnostic image is an image that is presented to the user to observe the user's reaction, which appears as changes in the user's facial expression, voice, etc. The diagnostic image DB 126 includes a condition ID list 1261 and a diagnostic image list 1262.

[0044] The condition ID list 1261 is a list of identification information for identifying each condition that is determined whether or not a history of consecutive index values ​​satisfies the condition, such as when the physical condition name "Exhausted" is identified five times in a row.

[0045] The diagnostic image list 1262 is a list associated with each condition ID listed in the condition ID list 1261, and lists diagnostic images to be presented to the user to observe their reaction when the user's index value history satisfies the condition identified by that condition ID. The multiple diagnostic images listed in this diagnostic image list 1262 are presented to the user in that order by the measurement device 1. By presenting the multiple diagnostic images in order and observing the user's reaction to viewing them, the measurement device 1 can, for example, identify the user's tendency toward dementia.

[0046] Furthermore, if the user's reaction observed when one of the diagnostic images is presented satisfies certain criteria, the diagnostic image is identified as an image that will have a certain effect on the user, and can be used to prevent the progression of dementia, etc.

[0047] <Functional configuration of the measuring device> The program according to the present invention is read and executed by the processor 11 of the measurement device 1. By executing this program, the processor 11 realizes three functions. · First function: Plots the state of the user's nervous system on a plane and outputs various information according to the area to which the plot belongs. · Second function: A function to ask the user questions based on the evaluation results of the user's nervous system condition, receive their answers, and output a diagnosis. -Third function: A function to present diagnostic images to the user based on the history of the user's nervous system condition, judge the user's reaction, and identify from the presented diagnostic images the image that will have a predetermined effect. The functional configuration of the processor 11 that realizes these three functions will be described below.

[0048] <First functional configuration> Fig. 10 is a diagram showing an example of a first functional configuration of the measurement device 1. The processor 11 of the measurement device 1 reads and executes a program stored in the memory 12, thereby functioning as a video acquisition unit 111, a pulse measurement unit 112, a ratio calculation unit 113, an activity calculation unit 114, a plot unit 115, a region identification unit 116, and an information output unit 117 shown in Fig. 10.

[0049] The video acquisition unit 111 acquires video data showing a video of a user's own face captured by the camera 16. The video data acquired by the video acquisition unit 111 is stored in the video DB 121 of the memory 12.

[0050] The pulse measurement unit 112 reads video data from the video DB 121 and measures vital signs including the pulse based on the video data. The vital signs measured by the pulse measurement unit 112 are stored in the vital DB 122 together with a video ID that identifies the video data, as shown in FIG.

[0051] The ratio calculation unit 113 calculates the ratio of the sympathetic nerve to the parasympathetic nerve (i.e., LF / HF) based on the pulse rate included in the measured vital sign value. The calculated ratio is stored in the vital sign DB 122.

[0052] The activity level calculation unit 114 calculates the activity level of the entire autonomic nerves (that is, TP) based on the pulse rate included in the measured vital sign value. The calculated activity level is stored in the vital sign DB 122.

[0053] The plotting unit 115 reads out a pair of LF / HF and TP from the vital DB 122, and reads out sections of the physical condition ID and physical condition name from the physical condition DB 123. Then, the plotting unit 115 plots the state of the user's nervous system indicated by the LF / HF and TP on a plane with these as the two axes.

[0054] The region specifying unit 116 specifies the region to which the state of the user's nervous system belongs from among the divided regions. This region indicates the user's physical condition.

[0055] The information output unit 117 extracts information corresponding to the area indicating the user's physical condition, which has been specified by the area specifying unit 116, from the recommendation DB 124, and outputs the information to the display unit 15.

[0056] <Second functional configuration> Fig. 11 is a diagram showing an example of a second functional configuration of the measurement device 1. The processor 11 of the measurement device 1 reads and executes a program stored in the memory 12, thereby functioning as a condition evaluation unit 1181, a medical questionnaire presentation unit 1182, a response reception unit 1183, and a diagnosis result output unit 1184 shown in Fig. 11.

[0057] The state evaluation unit 1181 reads out the pair of LF / HF and TP from the vital DB 122, and evaluates the state of the nervous system of the user by referring to the section of the physical condition DB 123. Here, the state of the nervous system is the latest physical condition.

[0058] The medical interview presentation unit 1182 extracts a medical interview item list 1252 corresponding to the state of the user's nervous system evaluated by the state evaluation unit 1181 from the medical interview DB 125. Then, the medical interview presentation unit 1182 causes the display unit 15 to sequentially display (present) the medical interview items listed in the extracted medical interview item list 1252.

[0059] The answer receiving unit 1183 receives a user's operation from the operation unit 14 and receives the user's answer to the above-mentioned medical question.

[0060] The diagnosis result output unit 1184 outputs the result of the diagnosis made based on the user's answer accepted by the answer accepting unit 1183 by displaying it on the display unit 15. This diagnosis may be made by the processor 11. When the processor 11 makes the diagnosis, the memory 12 may store in advance a combination of the user's vital sign information, its history, trends, and the user's answer to the medical interview, as well as the corresponding diagnosis result. The memory 12 may also store a machine learning model related to the above diagnosis that the processor 11 can use.

[0061] This diagnosis may also be performed by a doctor. In this case, the diagnosis result output unit 1184 transmits information on the user's vital signs and information on the answers to the medical interview to a terminal device used by the doctor via the communication unit 13. Then, the diagnosis result output unit 1184 receives information indicating the diagnosis result input by the doctor from the terminal device via the communication unit 13.

[0062] <Third functional configuration> Fig. 12 is a diagram showing an example of a third functional configuration of measuring device 1. Processor 11 of measuring device 1 reads and executes a program stored in memory 12, thereby functioning as state evaluation unit 1181, image presentation unit 1191, reaction determination unit 1192, image specification unit 1193, and specified image output unit 1194 shown in Fig. 12.

[0063] The state evaluation unit 1181 reads out the pair of LF / HF and TP from the vital DB 122 and evaluates the state of the user's nervous system by referring to the section of the physical condition DB 123. Furthermore, in this third functional configuration, the state evaluation unit 1181 determines whether or not the history of the pair of the user's LF / HF and TP calculated multiple times satisfies a predetermined condition. Here, the state of the nervous system is the result of determining whether or not the history of the physical condition indicated by the pair satisfies a predetermined condition.

[0064] When the state evaluation unit 1181 determines that the history satisfies a predetermined condition, the image presentation unit 1191 extracts a diagnostic image list 1262 corresponding to the predetermined condition from the diagnostic image DB 126. Then, the image presentation unit 1191 causes the display unit 15 to sequentially display (present) the diagnostic images included in the extracted diagnostic image list 1262.

[0065] While the display unit 15 sequentially displays diagnostic images to the user under the control of the image presentation unit 1191, the response determination unit 1192 acquires video of the user's face as he or she looks at the diagnostic images, captured by the camera 16. The response determination unit 1192 then identifies the user's response from changes in facial contours, changes in blood flow, etc., and determines whether or not the response satisfies predetermined determination conditions.

[0066] When the reaction determination section 1192 determines that the reaction of the user satisfies a predetermined determination condition, the image specification section 1193 specifies the diagnostic image that the user has been looking at.

[0067] The specific image output unit 1194 outputs the diagnostic image specified by the image specifying unit 1193 by displaying it on the display unit 15 as a specific image.

[0068] <Operation of the measuring device> <Operation of the first function> 13 is a diagram showing an example of the flow of operation relating to the first function of measuring device 1. Processor 11 of measuring device 1 reads and executes a program according to the present invention from memory 12. When this program is started, it displays a screen prompting the user to take a picture of their own face, a so-called selfie.

[0069] Fig. 14 is a diagram showing an example of display area R0. As shown in Fig. 14, a display unit 15 and a camera 16 are provided on the front of the measuring device 1. As described above, the display unit 15 has a display screen such as a liquid crystal display. The upper part of this display screen is provided with a general information display area for displaying radio wave reception sensitivity, remaining battery power, time, etc., and the lower part is provided with a general operation area for accepting operations on the operating system. Display area R0 is provided between these areas.

[0070] 15 is a diagram showing an example of a display when shooting a selfie video. When processor 11 prompts the user to take a selfie, a cancel icon area R1, an explanation area R2, a face shooting area R3, and a measurement start button area R4 are displayed in display area R0, as shown in FIG.

[0071] The cancel icon area R1 is an area for accepting an instruction to cancel selfie video shooting.

[0072] Explanation area R2 is an area for displaying text explaining the steps for taking a selfie video. In the example shown in Figure 15, explanation area R2 displays the following explanation: "Hold the smartphone directly in front of your face, align your face with the circle, and press the 'Start measurement' button."

[0073] The face shooting area R3 is an area that indicates the range in which the user's own face should fit within the image area that simultaneously displays the selfie video being shot.

[0074] The measurement start button area R4 is an area that displays a button for starting selfie video shooting.

[0075] Returning to the explanation of the operation shown in Fig. 13, when the user presses the measurement start button in response to the above-mentioned screen to start taking a selfie, processor 11 acquires video supplied from camera 16 (step S101). Processor 11 then measures the user's pulse from the acquired video (step S102). Note that in this embodiment, the user starts taking a selfie and captures a picture of their own face, but the subject of the photo is not limited to the face, and may be any part of the user's own body. In other words, step S102 is an example of a step in which the user's pulse is measured based on a video of the user's body.

[0076] When the pulse rate is measured, the processor 11 calculates LF / HF (the ratio of sympathetic nerve activity to parasympathetic nerve activity) from the pulse rate (step S103).

[0077] Furthermore, the processor 11 calculates TP (the activity level of the entire autonomic nerves) from the measured pulse (step S104).

[0078] Therefore, step S103 is an example of a step of calculating the ratio of sympathetic nerves to parasympathetic nerves from the measured pulse. Also, step S104 is an example of a step of calculating the activity of the entire autonomic nerves from the measured pulse. Note that the order in which steps S103 and S104 are executed may be reversed from the above, or may not be predetermined due to the execution of parallel processes.

[0079] 16 is a diagram showing an example of a screen displayed when measuring pulse rate, etc. When the user finishes taking a selfie video, processor 11 measures the pulse rate based on the captured video, as described above, and calculates LF / HF and TP based on the pulse rate. At this time, as shown in FIG. 16, processor 11 may control display unit 15 to display a graph showing the change in LF / HF or pulse rate over time in display area R0.

[0080] Returning to the explanation of the operation shown in Fig. 13, once LF / HF and TP have been calculated, the processor 11 controls the display unit 15 to plot them in the display area R0 on a plane having the LF / HF and TP as its respective axes (step S105). That is, step S105 is an example of a step of plotting the calculated ratio and the calculated activity level on a plane having the ratio of the sympathetic nerve to the parasympathetic nerve and the activity level of the entire autonomic nerve as its two axes.

[0081] Fig. 17 is a diagram showing an example of a plot of LF / HF and TP. When the above-mentioned step S105 is executed, a graph with LF / HF on the vertical axis and TP on the horizontal axis is displayed in the display area R0 of the display unit 15, as shown in Fig. 17. Then, for example, a plot P1 showing LF / HF and TP is drawn on this graph, as shown in Fig. 17.

[0082] For example, the user looks at plot P1 on this graph and recognizes that his or her physical condition is identified by a physical condition ID of "N8." Note that the graph may display a physical condition name associated with the physical condition ID in the physical condition DB 123 instead of or in addition to this physical condition ID.

[0083] Returning to the explanation of the operation shown in Fig. 13, once LF / HF and TP have been plotted, processor 11 identifies an area containing coordinates of the plot indicating these index values ​​(step S106). That is, step S106 is an example of a step of identifying an area on a plane including coordinates indicating the ratio and activity level plotted on the plane having the ratio of the sympathetic nerve to the parasympathetic nerve and the activity level of the entire autonomic nerve as its two axes.

[0084] Then, processor 11 refers to recommendation DB 124, extracts recommendation information such as food according to the physical condition ID of the identified area, and outputs the recommendation information according to the user's physical condition (step S107). That is, step S107 is an example of a step of outputting information corresponding to the identified area.

[0085] <Operation of the second function> 18 is a diagram showing an example of the operational flow of the second function of the measurement device 1. The processor 11 of the measurement device 1 reads the latest calculated LF / HF and TP from the vital DB 122 in the memory 12. Then, based on these read index values, the state of the user's nervous system is evaluated (step S201). In other words, step S201 is an example of a step of evaluating the state of the user's nervous system based on the calculated ratio and activity level.

[0086] Next, processor 11 extracts from medical interview DB 125 medical interview items corresponding to the evaluated condition, and displays them in order on display unit 15 to present the medical interview (step S202).

[0087] Next, the processor 11 accepts the user's response to the medical question by acquiring the response from the user who has viewed the presented medical question via the operation unit 14 (step S203). In other words, steps S202 and S203 are examples of steps in which a medical question corresponding to the evaluated state is presented and the user's response to the medical question is accepted.

[0088] Then, processor 11 diagnoses the user's physical and mental condition based on the evaluated condition and the received answer from the user, and outputs the diagnosis result by displaying it on display unit 15 (step S204). That is, step S204 is an example of a step of diagnosing the user based on the evaluated condition and the received answer, and outputting the diagnosis result.

[0089] With this second function, the measuring device 1 roughly grasps the state of the autonomic nervous system before the interview, and only the interview items corresponding to that state are presented, thereby making the interview and diagnosis more efficient and reducing the time spent on them.

[0090] <Operation of the third function> 19 is a diagram showing an example of the operational flow of the third function of the measurement device 1. The processor 11 of the measurement device 1 refers to the vital DB 122 and extracts the calculated index values, i.e., pairs of ratio and activity level (step S301).

[0091] Then, the processor 11 determines whether the extracted ratio-activity pair satisfies a predetermined condition for presenting a diagnostic image (hereinafter referred to as a presentation condition) (step S302).

[0092] When it is determined that the ratio-activity pair does not satisfy the presentation condition (step S302; NO), processor 11 ends the process.

[0093] On the other hand, if it is determined that the ratio-activity ratio pair satisfies the presentation condition (step S302; YES), processor 11 causes display unit 15 to sequentially display and present diagnostic images associated with the satisfied presentation condition (step S303). In other words, step S303 is an example of a step in which multiple diagnostic images are presented to the user when the ratio-activity ratio pairs calculated from the user's pulses measured at least multiple times satisfy a predetermined condition.

[0094] When a plurality of diagnostic images are presented in a predetermined order by executing step S303, processor 11 analyzes the user's reaction based on video captured by camera 16 of the user viewing these diagnostic images. Then, processor 11 determines whether the user's reaction exceeds, for example, a certain level (step S304). In other words, step S304 is an example of a step for determining whether the user's reaction upon viewing the presented diagnostic images satisfies a predetermined determination condition.

[0095] If the user's reaction satisfies the predetermined judgment condition, the processor 11 identifies the diagnostic image that the user was looking at at that time (step S305). That is, step S305 is an example of a step of identifying the diagnostic image that the user was looking at when it is judged that the user's reaction to the presented diagnostic image satisfies the judgment condition.

[0096] A diagnostic image that satisfies the determination condition is an image that, when presented, elicits a certain level of user reaction and is therefore expected to have a certain effect when displayed to the user. The determination condition is determined, for example, based on characteristics such as LF / HF, TP, pulse rate, facial expression, blood flow, and complexion that indicate a good user reaction and a high level of happiness when viewing the diagnostic image. The processor 11 then determines whether the conditions (hereinafter referred to as display conditions) for displaying the identified diagnostic image (i.e., the specific image) are met (step S306). Here, the display condition may be, for example, that the measuring device 1 has not been operated for a certain period of time, that it is bedtime, or the like.

[0097] If it is determined that the display condition is not satisfied (step S306; NO), the processor 11 ends the process.

[0098] On the other hand, when it is determined that the display condition is satisfied (step S306; YES), the processor 11 outputs the identified diagnostic image (i.e., the identified image) by displaying it on the display unit 15 (step S307). That is, step S307 is an example of a step of outputting the identified diagnostic image.

[0099] In this case, the specific image is output when the display conditions are satisfied, so step S307 is an example of a step of outputting the specified diagnostic image when the predetermined display conditions are satisfied.

[0100] With this third function, the measuring device 1 can present diagnostic images to a user who is suspected of having dementia because, for example, the state of the autonomic nervous system satisfies certain conditions, and extracts diagnostic images that have a good response as specific images and displays these images in a focused manner, which is expected to have a therapeutic and ameliorative effect on dementia.

[0101] Through the above-described operations, this program allows the user (subject) to easily check their own detailed condition in a variety of ways. By using this program, the user can check their own physical and mental condition in a two-dimensional area, which allows for, for example, early detection of abnormalities and stress checks. Furthermore, this program can be used for, for example, monitoring the elderly, managing driver fatigue, providing health promotion information, supporting exercise habits to prevent adult diseases, online medical care, clinical trials, etc.

[0102] <Modification> The above is a description of the embodiment, but the contents of this embodiment can be modified as follows. In addition, the following modifications can be combined with each other.

[0103] In the above-described embodiment, the measurement device 1 has the processor 11 configured as a CPU, but the control means for controlling the measurement device 1 may have other configurations.

[0104] That is, the measuring device 1 may have various types of processors as the processor 11, such as a GPU (Graphics Processing Unit), ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array), programmable logic device, etc., in addition to a CPU.

[0105] The operations of the processors in the above-described embodiments may not only be performed by a single processor, but may also be performed by a plurality of processors located at physically separate locations working together.

[0106] Furthermore, the order of the operations of the processor is not limited to the order described in the above embodiment, and may be changed as appropriate.

[0107] In the above-described embodiment, the index values ​​of the "ratio of sympathetic and parasympathetic nerves" and the "level of activity of the entire autonomic nerves" were calculated from the measured pulse rate itself, but these index values ​​may also be calculated based on the pulse rate after a predetermined processing process has been performed.

[0108] For example, the processor 11 of the measurement device 1 may generate data indicating the same phase and opposite phase from the time-series data of the measured pulse, and generate data (referred to as concatenated data) by alternately concatenating these data.

[0109] Then, processor 11 may calculate the two index values ​​described above from the generated linked data. That is, the program executed by processor 11 in this modified example is an example of a program for causing a computer to execute the steps of generating linked data by alternately linking data indicating the same phase and opposite phase of the measured pulse, and calculating the ratio and activity level from the generated linked data. According to the program shown in this modified example, by generating linked data by alternately linking data of the same phase and opposite phase and calculating the index value, it is possible to shorten the measurement time without reducing the accuracy of the index calculation.

[0110] In the above-described embodiment, the measurement device 1 is a single mobile terminal, but some of the functions may be realized by an external device.

[0111] 20 is a diagram showing an example of the overall configuration of a measurement system 9 according to a modified example. The measurement system 9 shown in FIG.

[0112] The measurement device 1 is a mobile terminal, such as a smartphone, and the server device 2 is a computer connected to one or more measurement devices 1 so as to be able to communicate with them.

[0113] The communication line 3 is a line that communicatively connects the measurement device 1 and the server device 2. The communication line 3 may be, for example, a LAN (Local Area Network), a WAN (Wide Area Network), the Internet, or a combination of these.

[0114] The number of measurement devices 1 and the number of server devices 2 in the measurement system 9 are not limited to those shown in Fig. 20. For example, the server device 2 may be configured as a cluster system in which multiple devices share functions.

[0115] Fig. 21 is a diagram showing an example of the configuration of a server device 2 according to a modified example. As shown in Fig. 21, the server device 2 includes a processor 21, a memory 22, and a communication unit 23. These are connected by a bus.

[0116] The processor 21 reads and executes a program stored in the memory 22 to control each part of the measurement device 1. The processor 21 is, for example, a CPU.

[0117] The communication unit 23 is a communication circuit that connects the server device 2 to other devices via the communication line 3 in a wired or wireless manner so that communication is possible.

[0118] The memory 22 is a storage means for storing an operating system, various programs, data, etc. that are read into the processor 21. The memory 22 includes a RAM and a ROM.

[0119] The memory 22 may include a solid state drive, a hard disk drive, etc. The memory 22 also stores a video DB 221, a vital sign DB 222, a physical condition DB 223, a recommendation DB 224, a medical interview DB 225, and a diagnostic image DB 226. These are databases common to the video DB 121, the vital sign DB 122, the physical condition DB 123, the recommendation DB 124, the medical interview DB 125, and the diagnostic image DB 126 stored in the memory 12 of the measuring device 1 shown in the embodiment.

[0120] Processor 21 of server device 2 may implement any of the functional configurations implemented by processor 11 in the embodiments in place of processor 11. For example, processor 21 may function as a video acquisition unit that acquires video from measuring device 1. In this case, processor 11 of measuring device 1 may transmit the video to server device 2 via communication unit 13, communication line 3, and communication unit 23. Processor 21 may also function as a pulse measurement unit that measures the user's pulse based on the acquired video, a ratio calculation unit that calculates the above-mentioned ratio based on the measured pulse, and an activity calculation unit that calculates the activity level.

[0121] In the above-described embodiment, the processor 11 of the measuring device 1 realizes the first function, the second function, and the third function by reading and executing the program according to the present invention, but it is not necessary to realize all of these functions. Also, the processor 11 may realize only part of each function, rather than all of them.

[0122] For example, the processor 11 may not control the display unit 15 or the like to plot the region to which the state of the user's nervous system belongs on a plane. In this case, the processor 11 may, for example, identify the region to which the state of the user's nervous system belongs and output various information corresponding to the identified region.

[0123] Furthermore, the processor 11 may identify the state of the user's nervous system using not only the LF and HF values ​​but also the VLF value, which is in the very low frequency range, and express it, for example, as a point in a three-dimensional space indicated by three axes. The processor 11 may also express the state of the user's nervous system in a tabular format. The processor 11 may also express the state of the user's nervous system auditorily, tactilely, or the like, in addition to or instead of a visual expression. In other words, the program according to the present invention may be sufficient if it allows the subject to confirm his or her own detailed state using a variety of expressions.

[0124] The program for executing each of the above steps may be provided in a state stored in a computer-readable recording medium such as a magnetic recording medium such as a magnetic tape or a magnetic disk, an optical recording medium such as an optical disk, a magneto-optical recording medium, a semiconductor memory, etc. This program may also be downloaded via a communication line such as the Internet. [Explanation of symbols]

[0125] 1...measuring device, 11...processor, 111...video acquisition unit, 112...pulse measurement unit, 113...ratio calculation unit, 114...activity calculation unit, 115...plotting unit, 116...area identification unit, 117...information output unit, 1181...condition evaluation unit, 1182...questionnaire presentation unit, 1183...answer reception unit, 1184...diagnosis result output unit, 1191...image presentation unit, 1192...response determination unit, 1193...image identification unit, 1194...specific image output unit, 12...memory, 121...video DB, 122...vital DB, 123...physical condition DB, 124...recommendation DB, 125...questionnaire DB, 1251...physical condition ID list, 12 52...list of medical interview items, 126...diagnostic image DB, 1261...list of condition IDs, 1262...list of diagnostic images, 13...communication unit, 14...operation unit, 15...display unit, 16...camera, 17...observation unit, 2...server device, 21...processor, 22...memory, 221...video DB, 222...vital DB, 223...physical condition DB, 224...recommendation DB, 225...medical interview DB, 226...diagnostic image DB, 23...communication unit, 3...communication line, 9...measurement system, P1...plot, R0...display area, R1...cancel icon area, R2...explanation area, R3...face capture area, R4...measurement start button area.

Claims

1. On the computer, measuring the pulse of the user based on a video of the user's body; calculating a ratio of sympathetic nerves to parasympathetic nerves from the measured pulse rate; calculating the overall activity of the autonomic nerves from the measured pulse rate; Plotting the calculated ratio and the calculated activity level on a plane having two axes representing the ratio of the sympathetic nerves to the parasympathetic nerves and the activity level of the entire autonomic nerves; A program to execute.

2. The computer, identifying an area on the plane including coordinates indicating the ratio and the activity level plotted on the plane; outputting information corresponding to the identified region; The program according to claim 1, further comprising:

3. The computer, evaluating the state of the user's nervous system based on the calculated ratio and the activity level; presenting a medical questionnaire corresponding to the evaluated condition and receiving a response from the user to the medical questionnaire; diagnosing the user based on the state and the answer and outputting a diagnosis result; The program according to claim 1, further comprising:

4. The computer, presenting a plurality of diagnostic images to the user when a set of the ratio and the activity level calculated from the user's pulse measured at least a plurality of times satisfies a predetermined condition; a step of determining whether or not a reaction of the user viewing the presented diagnostic image satisfies a predetermined determination condition; identifying the diagnostic image that the user was viewing when it was determined that the reaction satisfied the determination condition; outputting the identified diagnostic image; The program according to claim 3, further comprising:

5. The computer, outputting the identified diagnostic image when a predetermined display condition is satisfied; The program according to claim 4, further comprising:

6. The computer, generating concatenated data by alternately concatenating data indicating the same phase and opposite phase of the measured pulse; calculating the ratio and the activity from the generated linked data; The program according to claim 1, further comprising:

Images