Information processing device, information processing method and program

Abstract

To provide an information processing device, an information processing method and a program capable of more easily determining an autonomic nerve state.SOLUTION: An information processing device is provided with a control part. The control part acquires a continuous photographic image in two areas of a first area in which skin blood flow in a human body is easily influenced by autonomic nerves and a second area in which the skin blood flow is hardly influenced by the autonomic nerves. The control part respectively calculates amplitudes of signals corresponding to pulse waves in the two areas. The control part acquires a first estimation result about the autonomic nerve state by a relative value of the amplitudes in the two areas.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] Conventionally, there is a technology for judging the state of the autonomic nerves of the human body, particularly the stress level. When the degree of tension is high, the sympathetic nerves are dominant. It is known that the degree of tension can be judged from the fluctuation state of the pulse interval. In addition, Patent Document 1 discloses a technology for capturing an image of the face with far-infrared rays and near-infrared rays, and estimating the presence or absence of stress based on the change in temperature distribution according to tension. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-198531 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, conventional techniques have the drawback of being time-consuming, requiring long measurement times and equipment such as infrared sensors with multiple wavelengths to detect the state of the autonomic nerves.

[0005] An object of the present disclosure is to provide an information processing device, an information processing method, and a program that can more easily determine an autonomic nerve state. [Means for solving the problem]

[0006] In order to achieve the above object, the present disclosure provides: A method for detecting the skin blood flow of a human body, comprising: acquiring successive photographic images of two regions, a first region in which the skin blood flow is easily influenced by the autonomic nervous system, and a second region in which the skin blood flow is not easily influenced by the autonomic nervous system; Calculating the amplitude of a signal corresponding to the pulse wave in each of the two regions; a control unit for obtaining a first estimation result regarding a state of the autonomic nerves based on the relative values ​​of the amplitudes in the two regions; The information processing device includes: Effect of the Invention

[0007] According to the present disclosure, there is an advantage that the autonomic nerve state can be determined more simply. [Brief description of the drawings]

[0008] [Figure 1] 1 is a diagram illustrating a configuration of a state estimation system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a block diagram showing the functional configuration of the electronic device and the server device. [Diagram 3] FIG. 11 is a diagram illustrating a flow of an operation for estimating an autonomic nerve state. [Figure 4] FIG. 1 is a diagram illustrating fluctuations in pulse rate. [Diagram 5] The experimental procedure for measurement in the second method is shown. [Figure 6] The experimental results of measurements using the second method before and after loading are shown. [Figure 7] 5 is a flowchart showing a control procedure of a state estimation control process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment will be described with reference to the drawings. FIG. 1 is a diagram showing a configuration of a state estimation system 100 according to the present embodiment. The state estimation system 100 includes an electronic device 10 and a server device 50.

[0010] The electronic device 10 is an electronic device that can be easily used by a user and is capable of capturing moving images or continuous images. The electronic device 10 is, for example, a smartphone.

[0011] The server device 50 is an information processing device of this embodiment, and receives image data of a moving image or a continuous image from the electronic device 10 to estimate the state of the autonomic nerves. The server device 50 only needs to have a computing power and memory capacity capable of processing the received images, and may be an ordinary computer (PC; Personal Computer).

[0012] The electronic device 10 and the server device 50 communicate with each other at a communication speed that allows communication of the image data. The communication does not have to be performed directly between the electronic device 10 and the server device 50. That is, the communication may be performed via a wireless LAN router, a wired LAN cable, the Internet, or the like. For example, the electronic device 10 and the server device 50 can communicate with each other via a wireless LAN (Local Area Network). Alternatively, the electronic device 10 and the server device 50 may be able to communicate with each other directly using a high-speed communication standard (HS mode) of Bluetooth (registered trademark), or the like.

[0013] Note that one server device 50 may be able to acquire images captured by a plurality of electronic devices 10 and estimate the state of the autonomic nerves of each of them.

[0014] FIG. 2 is a block diagram showing the functional configuration of the electronic device 10 and the server device 50. As shown in FIG. As shown in FIG. 2(a), the electronic device 10 includes a CPU 11 (Central Processing Unit), a RAM 12 (Random Access Memory), a storage unit 13, an imaging unit 14, a communication unit 15, a display unit 16, an operation reception unit 17, and the like.

[0015] The CPU 11 is a processor that performs arithmetic processing and controls the overall operation of the electronic device 10. The CPU 11 may have a single processor. Alternatively, the CPU 11 may have multiple processors that perform arithmetic processing in parallel, or multiple processors that perform arithmetic processing independently depending on the application. The RAM 12 provides a working memory space for the CPU 11 and stores temporary data.

[0016] The storage unit 13 is a non-volatile memory that stores control programs, setting data, etc. The non-volatile memory may be, for example, a flash memory.

[0017] The photographing unit 14 is a camera having an optical system for irradiating light, an exposure unit for detecting the incident light at each pixel position, and an image output unit for generating and outputting image data based on the detected light amount distribution. The photographing unit 14 is capable of photographing moving images. As described later, the moving images have a frame rate required for accurately detecting changes according to the pulse rate. The photographing unit 14 is capable of photographing moving images at, for example, about 30 fps.

[0018] As described above, the communication unit 15 communicates image data with the server device 50. The communication unit 15 includes, for example, a network card for a wireless LAN and an antenna.

[0019] The display unit 16 displays various information on the display screen under the control of the CPU 11. The display screen is, for example, a liquid crystal display or an organic EL (Electro-Luminescent) display, but is not limited to these.

[0020] The operation reception unit 17 receives input operations from the outside. The operation reception unit 17 has, for example, a touch panel located overlapping the display screen and a push button switch located on the side of the electronic device 10. The operation reception unit 17 may also have other switches, for example, a slide switch.

[0021] As shown in FIG. 2(b), the server device 50 includes a CPU (Central Processing Unit) 51 (control unit), a RAM (Random Access Memory) 52, a storage unit 53, a communication unit 55, a display unit 56, an operation reception unit 57, and the like.

[0022] The CPU 51 is a processor that performs arithmetic processing and controls the overall operation of the server device 50. The CPU 51 may have a single processor. Alternatively, the CPU 51 may have multiple processors that perform arithmetic processing in parallel, or multiple processors that perform arithmetic processing independently depending on the application. The RAM 52 provides a working memory space for the CPU 51 and stores temporary data.

[0023] The storage unit 53 is a non-volatile memory that stores a control program, setting data, and the like. The non-volatile memory may be, for example, a hard disk drive (HDD) and / or a flash memory. The storage unit 53 stores a program 531 related to autonomic nerve state estimation, history information 532 that summarizes history data of the estimated autonomic nerve state, and the like. The storage unit 53 is not limited to a built-in auxiliary storage device. It may be an auxiliary storage device external to the server device 50, or a drive located on a network. Alternatively, a cloud server or the like may be used as the storage unit 53. Note that the program 531 may be stored in a built-in auxiliary storage device, and the history information 532 may be stored externally, such as a network drive or a cloud server.

[0024] As described above, the communication unit 55 communicates image data and transmits and receives estimation results with the electronic device 10. The communication unit 55 has, for example, a network card for communication via LAN with a router to which the electronic device 10 is connected via wireless LAN.

[0025] The display unit 56 displays various information on the display screen under the control of the CPU 51. The display screen is, for example, a liquid crystal display or an organic EL (Electro-Luminescent) display, but is not limited to these.

[0026] The operation reception unit 57 receives input operations from the outside. The operation reception unit 57 has, for example, a keyboard, a pointing device, and a push button switch. The pointing device is, for example, a mouse. The operation reception unit 57 may have other switches, for example, a slide switch. The display unit 56 and the operation reception unit 57 may be peripheral devices externally attached to the server device 50. It is also possible to acquire image data and the like via a portable medium such as a USB memory instead of the communication unit 55. That is, the server device 50 as a computer of this embodiment includes at least a CPU 51, a RAM 52, and a storage unit 53.

[0027] Next, the operation of estimating the state of the autonomic nerves in state estimation system 100 will be described. The state estimation system 100 of this embodiment uses two different methods for detecting whether the sympathetic nerves or the parasympathetic nerves are dominant from a visible light image of the face.

[0028] FIG. 3 is a diagram illustrating the flow of an operation for estimating the state of the autonomic nerves. In the state estimation system 100, the electronic device 10, which is each terminal, captures a video of the face (head) of the user or the person to be estimated (P1). The face is detected from the captured data (P2). The detection may be performed using a well-known image recognition algorithm. As an image recognition algorithm for recognizing a face, for example, the Viola-Jones method using Adaboost for machine learning can be mentioned.

[0029] When a face is detected, feature points are defined to indicate the boundaries of each component of the face, such as the area around the eyes, the area under the nose and the bridge of the nose, the areas around the upper and lower lips, the eyebrows, the jaw line, and the like, in an appropriate number.

[0030] Among these feature points, a particular point is used as a reference to define the region of interest in this embodiment (P3). The region of interest is the forehead and nose. The forehead region may be defined between the feature points indicating the eyebrows and the boundary between the upper end of the face outline, i.e., the lower end of the bangs. The nose region may be defined by points below the nose and the upper end point of the bridge of the nose. The image size of the nose region depends on the size of the nose in the captured image. The image size of the forehead region does not necessarily have to be the same as the image size of the nose region. In general, the larger the image size, the easier it is to reduce the effects of noise. Therefore, the forehead region may be defined as large within a range that allows it to be sufficiently separated from other parts of the face.

[0031] The electronic device 10 cuts out the defined attention area from the face image of each frame and sends it to the server device 50 as an attention area image (continuously captured images). At this time, the image format of the moving image or continuous still images may be changed from that at the time of capture. Note that in this embodiment, the image resolution after the attention area is cut out is not necessarily required to be high resolution. Therefore, the attention area image may have a lower image resolution than that at the time of capture.

[0032] As described below, the nose region is a region (first region) in the human body where the skin blood flow is strongly (easily) affected by the autonomic nerves. The forehead region is a region (second region) where the skin blood flow is less affected by the autonomic nerves. By cropping out the region-of-interest images of these two regions, it becomes easier to compare the features of the two regions.

[0033] The above-mentioned processes P2 to P3 do not necessarily have to be executed by the electronic device 10. The electronic device 10 may simply transmit the captured image to the server device 50. It may be possible to select which performs the processes P2 to P3. For example, it may be set individually depending on the processing capacity of the CPU 11 of the electronic device 10 and the communication speed between the electronic device 10 and the server device 50. Alternatively, it may be directly switched depending on the setting by the user's input operation to the operation reception unit 17.

[0034] The server device 50 extracts the pulse wave from the image of the area of ​​interest (P4). In the human body, blood vessels expand and contract periodically in response to the pulse. Hemoglobin in the blood absorbs light in specific wavelength bands, for example, the green wavelength band and infrared light well, so the components of these wavelength bands in the light reflected from the face periodically increase and decrease in response to the pulse. The pulse wave is extracted by detecting the periodic fluctuation of this component, the green component in this case, from the captured image. The pulse rate can be calculated based on the number of times per minute, the average period over a period of about the most recent 10-15 seconds.

[0035] FIG. 4 is a diagram illustrating fluctuations in pulse rate.

[0036] Even when one is at rest, the pulse intervals are not constant but fluctuate. In Fig. 4(a), T1 to T6 are the intervals between each pulse. These intervals are not equal to each other and change irregularly. When this change in pulse intervals, i.e., the fluctuation, is analyzed by frequency analysis, it is known that peaks appear in the frequency range of 0.04-0.15 Hz (LF band, second frequency band) and in the frequency range of 0.15-0.40 Hz (HF band, first frequency band), as shown in Fig. 4(b).

[0037] A method is known for determining the autonomic nervous state by a value (relative value) that specifies the relative relationship between the power value of the LF band (second fluctuation intensity) and the power value of the HF band (first fluctuation intensity), for example, the ratio "LF / HF". When a typical human is in a stressed or tense state, the balance of the autonomic nervous system is dominated by the sympathetic nervous system, and the value of LF / HF increases. Conversely, when in a relaxed state, the balance of the autonomic nervous system is dominated by the parasympathetic nervous system, and the value of LF / HF decreases. Therefore, by taking the ratio between the power value of the HF band in the fluctuation of the pulse interval and the power value of the LF band, it is possible to obtain the state in which the sympathetic nervous system or the parasympathetic nervous system is dominant in the autonomic nervous system, that is, the degree of stress (second estimation result). Note that the magnitude of the fluctuation (fluctuation intensity) represented by the power value of each frequency band depends on the characteristics (age, sex, etc.) and constitution of the individual, and therefore the numerical values ​​are not necessarily simply comparable between different individuals. Also, even for the same person, the numerical values ​​may vary slightly depending on fatigue, so caution is required.

[0038] The first method of estimating the state of the autonomic nerves, particularly the degree of stress, based on such fluctuations in pulse intervals has been known for a long time and is reliable. In this case, an image of the third region is used to estimate the state of the autonomic nerves. For the image of the third region, an image of only one of the first region and the second region may be used, or an image of both the first region and the second region may be used in combination. Alternatively, an image of another region not including the first region and the second region may be used to estimate the state of the autonomic nerves by the first method, or an image of the entire face, for example, including either the first region or the second region or both may be used. The frequency fluctuation of the detected pulse wave is identified (P5), and the state of the autonomic nerves is estimated based on the power ratio of the two frequency bands, the LF band and the HF band (P6). However, in this first method, it is necessary to stably obtain a maximum of 0.04 Hz, that is, a fluctuation with a period of about 25 seconds, multiple times. Therefore, in order to obtain an estimation result with a certain degree of accuracy, stable image capture and analysis of the image for several minutes to five minutes are required, which takes time and effort from the user.

[0039] In contrast, in the state estimation system 100 of this embodiment, in addition to the above-mentioned first method, a second method capable of easily determining the state of the autonomic nerves in a short time is used in combination. Returning to FIG. 3, of the two attention areas, the nose area is an area with many arteriovenous anastomotic blood vessels (AVA blood vessels). In an area with many AVA blood vessels, the skin blood flow is easily affected by the autonomic nerve state, and in a state of tension, the blood vessels close and the blood flow decreases. As a result, it is assumed that the fluctuation amplitude of the light amount in the green wavelength band according to the pulse becomes smaller. On the other hand, in the forehead area, such influence by the autonomic nerves is small. Therefore, there is no large change in the amplitude of the fluctuation of the light amount in the green wavelength band.

[0040] Due to such differences between regions, the ratio of the amplitude in the nose region to the amplitude in the forehead region for the pulse wave extracted from the image shows fluctuations according to the autonomic nerve state. That is, this amplitude ratio gives a result (first estimation result) indicating the stress level (autonomic nerve state). As in the case of the pulse interval fluctuation described above, the fluctuation in the ratio of pulse wave amplitudes between different parts also depends on the individual's characteristics and state such as physical fatigue. In addition, the absolute value of the amount of light depends on the shooting environment. Therefore, it is difficult to use the amount of reflected light in each region alone to estimate the state of the autonomic nerve. On the other hand, as described above, this index is obtained from the ratio of the amplitude of the luminance time series signal (signal) obtained from the fluctuation of the reflected light in the first region and the reflected light in the second region in the same image. Therefore, this index is not easily affected by the shooting environment. In addition, this second method can obtain a result after multiple pulses, that is, about 5-10 seconds.

[0041] FIG. 5 shows the experimental procedure for measurement in the second method. Before the measurement, the subject is kept in a resting state for 20 minutes. After that, the subject's face is photographed for 10 seconds, and the intensity amplitude of green wavelength band light in each area of ​​interest is measured. 20 seconds after the photograph, the subject is given a stressful load for 2 minutes. After that, the subject is photographed again for 10 seconds.

[0042] In experiments, loads that can directly and immediately cause stress include those that cause physical pain and those that cause psychological discomfort. Psychological discomfort includes those that cause senses other than touch, such as noise, vibration, unpleasant images, and smells. In human experiments, immersing the hands in cold water was used as a less intense load. Stress also depends largely on the mental state of the subject. Therefore, depending on the initial state of the subject, there may be cases where stress does not increase.

[0043] Figure 6 shows the experimental results of measurements using the second method before and after loading. As shown in Figure 6(b), when comparing before and after the application of a cooling load, it was confirmed that the pulse amplitude in the nose area, taken as the pulse wave amplitude in the forehead area w as the standard, tended to decrease. This graph was obtained from the average value of a sample number N=13. The error bars in the graph represent the standard deviation of the measured values ​​of the 13 samples in both the positive and negative directions.

[0044] 3, in the state estimation system 100 of this embodiment, the amplitude of the change in light intensity corresponding to the pulse is identified for the forehead region and the nose region (P7). The state of the autonomic nerves is estimated by the second method based on the ratio of the amplitude in the nose region to the amplitude in the forehead region (P8).

[0045] As described above, the longer the shooting (measurement) time, the higher the accuracy of the conventional first method is compared to the second method. Therefore, in this embodiment, the estimation results of the state of the autonomic nerves obtained by the two measurement methods are used in combination according to the shooting time, and in particular, a weighted average estimation result (third estimation result) is obtained. The weighting may be, for example, a simple linear change with respect to the shooting time, but is not particularly limited to this. For example, the weighting for the result of the first method is zero when the shooting time is 30 seconds or less, increases linearly between 30 seconds and 5 minutes (300 seconds), and becomes 100% when the shooting time is 5 minutes or more. The weighting for the result of the second method is anti-correlated with the weighting for the result of the first method. That is, the shorter the shooting time, the higher the weighting of the first estimation result, and the longer the shooting time, the lower the weighting. When an image captured for 5 minutes or more is stably obtained, the weighting of the first estimation result is 0%.

[0046] Here, as described above, it is not the simple duration of the captured image that is important for the evaluation, but the stability. For this purpose, a fluctuating factor of the amount of light in the captured image is detected separately. The fluctuating factor includes, for example, blurring of the field of view. When the electronic device 10 is held by hand, the shooting direction is likely to be blurred. The fluctuating factor may also include the movement of the subject, for example, a change in the angle of the face. Furthermore, the fluctuating factor may include fluctuations in external light, for example, the movement of clouds outdoors, or the movement of a reflector that reflects incident light or an obstruction that passes in front of a window. When the fluctuating factor is detected, the use of the captured image is terminated at the timing immediately before the detection, and the time up to that timing is set as the duration of detection (P11).

[0047] When the clipping of the attention area from the photographed image is performed by the server device 50, the server device 50 detects and evaluates the above-mentioned fluctuation factors. When the clipping of the attention area from the photographed image is performed by the electronic device 10, information on the stability of the attention area image to be transmitted may be attached to the attention area image. For example, information on the change in the range to be clipped from the photographed image, the change in the positional relationship of the feature points used for clipping, etc. may be quantified and attached to the attention area image. When the photographed image or the attention area image is no longer stable due to the above-mentioned fluctuation factors in the middle of the photographed image or the attention area image, the acquisition of the photographed image and the estimation of the autonomic nerve state may be terminated at that point. In this case, the time until the termination is the duration of the photographed image that is stably obtained. Note that when the obtained duration is long, for example, 30 seconds or more, the entire image data for the duration may not be used for the estimation of the autonomic nerve state by the second method. In other words, the time width used for the estimation of the autonomic nerve state by the first method and the estimation of the autonomic nerve state by the second method may be different as long as the reference time overlaps.

[0048] In this way, when the duration of the captured images used for estimation is determined, the weighting is determined based on the duration. In response to this, the state of the autonomic nerves obtained by the processes of P6 and P8 is weighted and averaged (P9). The estimation result of the state of the autonomic nerves obtained in this way does not have to be output as it is. The estimation result obtained may be further classified into several stages, for example, about 3 to 5 stages of stress level, and a symbol, code, or sign according to the stress level may be output as the estimation result (P10).

[0049] FIG. 7 is a flowchart showing a control procedure of the state estimation control process executed by the CPU 51 of the server device 50. The state estimation control process, which is an information processing method of this embodiment, is an example of processing when a captured image is received as is by the server device 50. When a captured image is acquired together with a request for an estimation result of an autonomic nerve state, the CPU 51 reads out and executes this state estimation control process from the program 531.

[0050] The CPU 51 acquires image data received by the communication unit 55 (S1). The CPU 51 detects a face from each image data (S2). The CPU 51 sets feature points in the detected face, sets a region of interest based on the feature points, and extracts an image of the set region of interest (S3; acquisition means).

[0051] The CPU 51 identifies the fluctuation in the luminance value of the green wavelength band in the image of each region of interest (S4).The CPU 51 determines the detection time based on the degree of blurring of the image of each region, etc. (S5).

[0052] The CPU 51 calculates the range of the identified fluctuation for each of the target regions (S6; calculation means). The CPU 51 calculates a first estimation result based on the ratio of the ranges of the fluctuation for each of the calculated regions (S7; estimation means).

[0053] The CPU 51 calculates the frequency spectrum of the identified fluctuation, and calculates the spectral intensity in each of the HF band and the LF band (S8). The CPU 51 obtains a second estimation result indicating the state of the autonomic nerves based on the ratio of these spectral intensities (S9).

[0054] The CPU 51 acquires a weighting coefficient based on the determined detection time, and calculates a weighted average value of the first estimation result and the second estimation result based on the acquired weighting coefficient to determine a third estimation result (S10).

[0055] The CPU 51 determines a stress index to be output according to the third estimation result, and outputs it as the estimation result (S11). The estimation result is output to the electronic device 10 that has requested the result. The CPU 51 stores the estimation result in the history information 532 together with the estimation time and the identification information of the electronic device 10 that has requested the result. The stored estimation result does not have to be limited to the final one according to the third estimation result. The history information 532 may store the first to third estimation results together. The CPU 51 may refer to the history information 532 and output a comparison result between the autonomic nerve state at the most recent timing and the autonomic nerve state at the previous timing (before the most recent timing), or a tendency of change such as maintenance, deterioration, or improvement of the autonomic nerve state over a longer period. The CPU 51 ends the state estimation control process.

[0056] As described above, the processes S1 to S3 may be executed by the CPU 11 in the electronic device 10, and then the server device 50 may obtain the processed data of S3. The CPU 51 of the server device 50 executes the processes S4 and thereafter based on the obtained data.

[0057] As described above, the server device 50 (part of which may be the electronic device 10) which is an information processing device of this embodiment includes a CPU 51 (CPU 11 in the case of the electronic device 10; the same applies below). The CPU 51 acquires continuous captured images (attention area images) of two areas of the human body, a first area in which the skin blood flow is easily influenced by the autonomic nerves, and a second area in which the skin blood flow is not easily influenced by the autonomic nerves. Being easily influenced by the autonomic nerves means that there are many AVA blood vessels. The CPU 51 calculates the amplitude of the signal corresponding to the pulse wave in the two areas, here, the amplitude of the luminance fluctuation of the green component. The CPU 51 obtains a first estimation result of the state of the autonomic nerves from the relative value, for example, the ratio, of the amplitudes in the two areas. In this way, the second method of this embodiment is simple because the estimation result is obtained from the captured image. In particular, since the amplitude of the blood flow fluctuation is obtained in pulse units, multiple measurement results can be obtained even in a few to 10 seconds. Therefore, the second method of this embodiment can easily obtain a result in a short time, and can reduce the inconvenience to the user. In particular, in this second method, the influence of external fluctuations can be reduced by short-time detection, and the conditions required for imaging are relaxed. That is, it is not necessary to fix the imaging unit in a place where the incident light does not fluctuate for a long time. Also, the person to be imaged does not need to remain still for a long time. This reduces the effort required to prepare the measurement environment. Furthermore, this second method can reduce the psychological impact caused by preparing the imaging environment and the person maintaining a posture for a long time.

[0058] Furthermore, the CPU 51 may calculate the fluctuation intensity of the period (pulse period) of the fluctuation of the reflected light intensity (signal) of the green wavelength band corresponding to the pulse wave (pulse) with respect to the frequency based on the continuous captured images of the third region of the human body. The CPU 51 obtains a second estimation result of the state of the autonomic nerves from the relative value (ratio) of the first fluctuation intensity in the first frequency band (HF band) and the second fluctuation intensity in the second frequency band (LF band). The CPU 51 obtains a third estimation result of the state of the autonomic nerves by using the first estimation result and the second estimation result in combination. By using the first estimation result, which is easily obtained in a short period of time, in combination with the second estimation result, which is conventionally used and has high reliability but is difficult to obtain accuracy in a short period of time, complementary results can be obtained. Therefore, the server device 50 can suppress a decrease in the accuracy of the estimation result obtained in a short period of time.

[0059] Furthermore, the CPU 51 may obtain a third estimation result by taking a weighted average of the first estimation result and the second estimation result. By appropriately weighting the estimation results according to the reliability of each result, it is possible to reduce the degree of inaccuracy of the result and improve the reliability.

[0060] Furthermore, the weighted average may be determined such that the shorter the shooting time of the captured image, the heavier the weight of the first estimation result. Since the accuracy of the second estimation result depends on the shooting time, the first estimation result may be prioritized when the shooting time is short, and the second estimation result may be prioritized when the shooting time is long. By determining the weights in this manner, the server device 50 can improve the accuracy of the estimation result to be output.

[0061] The shooting time may be determined based on the duration over which the captured image is stably obtained. In particular, if the image is blurred or disturbance is introduced during long-term shooting, the noise in the captured image increases and the accuracy of the estimation decreases. Therefore, by selecting a portion of the input captured image that is stably captured and using it for estimation, the degradation of the estimation result due to noise can be suppressed.

[0062] The third region may include at least one of the first region and the second region. That is, it is not necessary to make the range of the captured image different between the first method and the second method. By analyzing the same range with different methods to obtain respective results, the server device 50 can reduce factors that cause different results. Also, the labor required for image processing due to an increase in the number of captured locations is not increased.

[0063] Also, the CPU 51 may cut out the captured images of the two regions from the captured image of the face. There are several regions in the human body that have many AVA blood vessels, but if the face is originally exposed, it is easy to capture them. Also, since the first region and the second region are captured side by side, it is easy to set the capture range.

[0064] The two areas may be the forehead and nose. Both are particularly in the front of the face, so the photographer can take the image while facing the image capture unit. This makes it easier to obtain an image with an appropriate amount of light without being affected by unintended external light.

[0065] Also, the CPU 51 may compare the first estimation result obtained at the most recent timing with the first estimation result obtained before this result to obtain the change in the state of the autonomic nerves. By performing a simple measurement in a short time, the amount of work does not increase even when performing multiple measurements, and the measurement is unlikely to become troublesome midway. In particular, since it is easy to obtain a relatively high accuracy when measuring the same person, it is easy to obtain the comparison results of multiple measurements.

[0066] The information processing method of this embodiment includes the following processes: (1) Obtaining continuous images of two regions of the human body: a first region where skin blood flow is easily affected by the autonomic nerves, and a second region where skin blood flow is not easily affected by the autonomic nerves. (2) Calculating the amplitudes of signals corresponding to the pulse waves in the two regions, particularly the luminance time-series signal of the green component. (3) Obtaining a first estimation result of the state of the autonomic nerves from the ratio of the amplitudes in the two regions. According to this information processing method, since an estimation result can be obtained from an image captured in a short time, measurement is easy, and therefore, this information processing method can reduce the inconvenience of the user having to stand still for a long time and having to set up the environment to obtain a stable image for a long time.

[0067] Moreover, by installing and executing the program 531 of this embodiment on a computer, the state of the autonomic nerves can be easily estimated from images that have been captured more easily than before.

[0068] The present invention is not limited to the above-described embodiment, but may be modified in various ways. For example, in the above embodiment, the server device 50 is described as analyzing the state of the autonomic nerves and obtaining the estimation result. However, the analysis and obtaining the estimation result may be performed by the electronic device 10. In other words, all the processes from taking a face image to estimating the state may be completed within the electronic device 10.

[0069] In the above embodiment, the first method, which is time-consuming but has been used and is reliable, and the second method, which can be performed simply and quickly, are used in combination, but this is not limited to the above. In cases where it is clear that the shooting time is limited to a short period of time from the beginning, only the second method may be used from the beginning.

[0070] In the above embodiment, the nose is taken as an example of an area with many AVA blood vessels, but the present invention is not limited to this. For example, ears can also be cited as an area with many AVA blood vessels. The subject of the image does not have to be a face. Even in the case of a face, the image does not have to be the entire face as long as the area of ​​interest (forehead and nose) is included. An image detection algorithm different from the above may be used to identify the area of ​​interest according to these.

[0071] In addition, the above process does not have to be a real-time process. When it is not necessary to immediately inform the user of the estimation result, images at one or more appropriate estimation times (ranges) may be extracted in order from the obtained long-term shooting data, and the autonomic nerve state may be estimated for each of them.

[0072] In the above, a weighted average according to the duration of the results obtained by the two methods is used, but this is not limited to this. The degree of stability during the application period of the first method and the application period of the second method may be quantitatively evaluated and taken into account in the weighting.

[0073] Furthermore, weighting may not be used at all, and a uniform arithmetic average may be used.

[0074] Moreover, although a change in the green component of the captured image is detected in the above, a change in infrared light may be detected. In the above embodiment, the index in the second method is calculated from the ratio of the amplitudes of the luminance time-series signals, but this is not limited to this. Any value other than the ratio may be used as long as it indicates the relative relationship between the amplitudes in the two regions (relative value). Furthermore, the frequency ranges of the HF and LF bands may deviate from those in the above embodiment.

[0075] Furthermore, the stability of an image may be evaluated using not only the content of the captured image but also the measurement results of other sensor devices such as an acceleration sensor, tilt sensor, and light quantity sensor of the electronic device 10.

[0076] In the above description, the image is captured by the image capturing unit 14 included in the electronic device 10, but the present invention is not limited to this. Image capturing may be performed by connecting a camera to the electronic device 10 or the server device 50 as a peripheral device.

[0077] In the above description, the storage unit 53 including a non-volatile memory such as an HDD or a flash memory is taken as an example of a computer-readable medium for storing the program 531 related to the state estimation control of the present disclosure, but is not limited thereto. As other computer-readable media, other non-volatile memories such as MRAM and portable recording media such as CD-ROMs and DVD disks can be applied. In addition, a carrier wave is also applied to the present invention as a medium for providing data of the program according to the present invention via a communication line. In addition, the specific configurations, contents and procedures of the processing operations, etc. shown in the above embodiments can be modified as appropriate without departing from the spirit of the present invention. The scope of the present invention includes the scope of the invention described in the claims and its equivalents. [Explanation of symbols]

[0078] 10 Electronic equipment 11 CPU 12 RAM 13 Storage section 14. Photography Department 15 Communications Department 16 Display section 17 Operation reception section 50 Server equipment 51 CPU 52 RAM 53 Storage section 531 Programs 532 History Information 55 Communications Department 56 Display section 57 Operation reception section 100 State Estimation System

Claims

1. Obtaining successive photographed images of two regions of a human body, the first region having skin blood flow that is easily influenced by the autonomic nerves, and the second region having skin blood flow that is not easily influenced by the autonomic nerves; Calculating the amplitude of a signal corresponding to a pulse wave in each of the two regions; a control unit for obtaining a first estimation result regarding a state of the autonomic nerves based on the relative values ​​of the amplitudes in the two regions; An information processing device comprising:

2. The control unit is Calculating a fluctuation intensity of a period of the change in the pulse interval of the signal with respect to frequency based on successive photographed images of the third region of the human body; obtaining a second estimation result regarding the state of the autonomic nerve based on a ratio between a first fluctuation intensity in a first frequency band and a second fluctuation intensity in a second frequency band; The information processing apparatus according to claim 1 , further comprising: a third estimation result for the state of the autonomic nerves obtained by using the first estimation result and the second estimation result in combination.

3. The information processing apparatus according to claim 2 , wherein the control unit obtains the third estimation result by taking a weighted average of the first estimation result and the second estimation result.

4. The information processing apparatus according to claim 3 , wherein the weighted average is determined such that the shorter the capture time of the captured image, the greater the weight of the first estimation result.

5. The information processing apparatus according to claim 4 , wherein the photographing time is determined based on a duration during which the photographed images are being stably obtained.

6. The information processing apparatus according to claim 2 , wherein the third area includes at least one of the first area and the second area.

7. The information processing apparatus according to claim 1 , wherein the control unit cuts out the captured images of the two regions from a captured image of a face.

8. The information processing device according to claim 1 , wherein the two areas are a forehead and a nose.

9. The information processing device according to claim 1 , wherein the control unit compares the obtained first estimation result at the most recent timing with the first estimation result obtained before the most recent timing to obtain a change in the state of the autonomic nerves.

10. Obtaining successive photographed images of two regions of a human body, the first region having skin blood flow that is easily influenced by the autonomic nerves, and the second region having skin blood flow that is not easily influenced by the autonomic nerves; Calculating the amplitude of a signal corresponding to a pulse wave in each of the two regions; obtaining a first estimation result of an autonomic nerve state based on the relative values ​​of the amplitudes in the two regions; Information processing methods.

11. Computer, an acquisition means for acquiring successive photographed images of two regions of a human body, the first region having a skin blood flow that is easily influenced by the autonomic nerves, and the second region having a skin blood flow that is not easily influenced by the autonomic nerves; a calculation means for calculating amplitudes of signals corresponding to the pulse waves in the two regions; an estimation means for obtaining a first estimation result regarding an autonomic nerve state based on the relative values ​​of the amplitudes in the two regions; A program that functions as a

Images