Pulse wave estimation device, condition estimation device, and pulse wave estimation method

Abstract

The present invention comprises: a captured image acquisition unit (11) that acquires a captured image capturing a human; a skin region detection unit (12) that uses the captured image to detect a skin region of the human; a measurement region setting unit (13) that, in a region corresponding to the skin region in the captured image, sets a measurement region for extracting a pulse wave source signal indicating a luminance change; a pulse wave source signal extraction unit (14) that extracts the pulse wave source signal on the basis of the luminance change in the measurement region in the captured image; a simulation signal generation unit (15) that, on the basis of a measurement region position change and a distribution model of ambient light in a captured area, generates a simulation signal simulating the luminance change in the measurement region occurring due to the measurement region position change in ambient light; and a pulse wave estimation unit (16) that estimates the pulse wave of the human on the basis of the pulse wave source signal and the simulation signal.

Description

[Technical field]

[0001] The present disclosure relates to a pulse wave estimation device, a condition estimation device, and a pulse wave estimation method. [Background technology]

[0002] A technology is known that estimates a person's pulse wave from minute changes in luminance of the skin surface without contact by estimating the pulse wave based on the luminance signal of an area (hereinafter referred to as "skin area") including the person's skin in an image captured by an imaging device. In this case, when a person (hereinafter referred to as "subject") whose pulse wave is to be estimated moves, the position of the subject's skin area in the captured image changes, and the estimation accuracy of the pulse wave is affected by the change in the position of the skin area and decreases. Therefore, a technology is known in the past that removes a signal based on the change in the position of the skin area from the luminance signal of the subject's skin area as noise. For example, Patent Document 1 discloses a pulse detection device that removes a specific frequency component obtained by frequency analysis of position information of a face area in an image capture area from an image signal from the face area as noise. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-68720 A Summary of the Invention [Problem to be solved by the invention]

[0004] In a scene where the distribution of ambient light in the imaging range is non-uniform, the pulse wave estimated from minute changes in luminance of the subject's skin surface in the captured image is affected by not only changes in the position of the skin area but also by the ambient light. Conventional techniques such as those disclosed in Patent Document 1 do not take this into consideration, and therefore have a problem of low accuracy in estimating the subject's pulse wave.

[0005] The present disclosure has been made to solve the above-mentioned problems, and aims to provide a pulse wave estimation device that can estimate a subject's pulse wave even in a scene where the distribution of ambient light in the imaging range is uneven. [Means for solving the problem]

[0006] A pulse wave estimation device according to the present disclosure includes an image acquisition unit that acquires an image of a person, a skin area detection unit that detects a skin area of ​​the person from the image, a measurement area setting unit that sets a measurement area in an area on the image corresponding to the skin area for extracting a pulse wave original signal indicating a luminance change, a pulse wave original signal extraction unit that extracts a pulse wave original signal based on the luminance change in the measurement area on the image, a simulation signal generation unit that generates a simulation signal simulating a luminance change in the measurement area caused by a position change in the measurement area under ambient light based on a position change in the measurement area and a distribution model of ambient light in the imaging range, and a pulse wave estimation unit that estimates the person's pulse wave based on the pulse wave original signal and the simulation signal. The simulation signal generating unit generates a simulation signal based on a plurality of different ambient light distribution models corresponding to the situation in which the ambient light is irradiated and a position change of the measurement area. It is something. Effect of the Invention

[0007] According to the present disclosure, the pulse wave of a subject can be estimated even in a scene in which the distribution of ambient light in an imaging range is non-uniform. [Brief description of the drawings]

[0008] [Figure 1] 1 is a diagram showing an example of the configuration of a state estimation device including a pulse wave estimation device according to a first embodiment. [Diagram 2] 1 is a diagram illustrating an example of the configuration of a pulse wave estimation device according to a first embodiment. [Diagram 3] 3A, 3B, and 3C are diagrams for explaining an example of a method for setting a measurement region by a measurement region setting unit in the pulse wave estimation device according to the first embodiment. [Figure 4] FIG. 2 is a diagram showing an outline of an example of an environmental light model in a captured image when a two-dimensional Gaussian distribution is used for the environmental light model in the first embodiment. [Diagram 5]4 is a flowchart illustrating the operation of the pulse wave estimation device according to the first embodiment. [Figure 6] 4 is a flowchart for explaining an operation of the state estimating device according to the first embodiment. [Figure 7] 7A and 7B are diagrams for explaining the effect of ambient light on a signal used to estimate a subject's pulse wave based on an captured image, which occurs when the distribution of ambient light is non-uniform. FIG. 7A shows an example of an captured image captured in a scene where the distribution of ambient light is uniform, and FIG. 7B shows an example of an captured image captured in a scene where the distribution of ambient light is non-uniform. [Figure 8] 5 is a diagram for explaining an example of a method for generating a simulation signal based on a plurality of environmental light models by a simulation signal generating unit in the first embodiment. FIG. [Figure 9] 1A is a diagram showing an example of a circumscribing rectangle of a measurement area and its center coordinates, and an image of a distance between a vertex coordinate of one corner of the measurement area and the corresponding center coordinate in the first embodiment. FIG. [Figure 10] FIG. 4 is a diagram showing an example of a first test signal and a second test signal in which the difference in scale is made uniform by multiplying the first test signal and the second test signal by a normalization coefficient in the test signal generating unit in the first embodiment. [Figure 11] 11A and 11B are diagrams illustrating an example of the hardware configuration of the pulse wave estimation device and the state estimation device according to the first embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Embodiment 1 FIG. 1 is a diagram showing an example of the configuration of a condition estimating device 2 including a pulse wave estimating device 1 according to the first embodiment.

[0010] Pulse wave estimation device 1 estimates the pulse wave of a person based on a captured image of the person. A person whose pulse wave is estimated by pulse wave estimation device 1 is referred to as a "subject." Pulse wave estimation device 1 captures a series of frames Im(k) of images at least of a range where a skin area including the subject's skin should be present (hereinafter referred to as "skin present range") at a predetermined frame rate Fr. Here, k indicates a frame number assigned to each frame. For example, the frame provided next to frame Im(k) is frame Im(k+1). In the first embodiment, the skin region is a region corresponding to the subject's face. Note that this is merely an example, and the skin region may be a region other than the subject's face. For example, the skin region may be a region corresponding to a part belonging to the face, such as the subject's eyes, eyebrows, nose, mouth, forehead, cheeks, or chin. The skin region may also be a region corresponding to a body part other than the face, such as the subject's head, shoulders, hands, neck, or feet. The skin region may be a plurality of regions.

[0011] Pulse wave estimation device 1 then estimates the subject's pulse wave from a series of frames Im(k-Tp+1) to Im(k) for each specific number of frames Tp, and outputs pulse wave estimation result P(t) which is information indicating the estimated pulse wave (hereinafter referred to as "pulse wave information"). Specifically, pulse wave estimation device 1 estimates the subject's pulse wave by removing a signal based on a change in position of the subject's skin region, which becomes noise, from a signal based on a change in luminance of the subject's skin region in the series of frames Im(k-Tp+1) to Im(k). In doing so, pulse wave estimation device 1 generates the signal to be removed, taking into account the distribution of ambient light in the imaging range.

[0012] Here, t indicates an output number assigned to each specific number of frames Tp. For example, the pulse wave estimation result given at the timing following pulse wave estimation result P(t) is pulse wave estimation result P(t+1). Frame number k and output number t are integers equal to or greater than 1. Frame number Tp is an integer equal to or greater than 2.

[0013] The signal indicating the luminance change of the subject's skin region used when estimating the subject's pulse wave includes, for example, a signal based on the subject's movement in addition to a signal based on the pulse wave. The signal based on the subject's movement becomes noise that reduces the estimation accuracy of the subject's pulse wave. In addition, the signal indicating the luminance change of the subject's skin region is also affected by the distribution of environmental light. Therefore, the pulse wave estimation device 1 estimates the subject's pulse wave after removing the signal based on the subject's movement, in other words, the position change of the subject's skin region on the frame as a signal indicating noise, taking into account the environmental light. In the first embodiment, the "subject's movement" that causes a luminance change on the captured image that is considered to be noise when estimating the subject's pulse wave refers to any of the following: movement accompanied by movement of the subject's head position itself, movement accompanied by a change in face direction, movement of facial expression (smiling, moving of the mouth during conversation, etc.), and movement due to a detection error in the region for extracting a signal indicating a luminance change (called a measurement region, details of which will be described later).

[0014] The number of subjects included in the captured image may be one or more. For ease of explanation, the following embodiment 1 will be described assuming that the number of subjects included in the captured image is one.

[0015] State estimation device 2 acquires pulse wave estimation result P(t) from pulse wave estimation device 1 and estimates the state of the subject. In the first embodiment, as an example, the state of the subject is assumed to be whether or not the subject is awake. State estimation device 2 estimates the level of wakefulness indicating the degree to which the subject is awake, based on pulse wave estimation result P(t). The state estimation device 2 outputs state information Z(t) regarding the estimated state of the subject to the output device 3. For example, when the awakening level of the subject is decreased, the state estimation device 2 outputs state information Z(t) indicating that the awakening level of the subject is decreased to the output device 3.

[0016] The output device 3 outputs information based on the state information Z(t) output from the state estimation device 2. The output device 3 is, for example, an audio output device. For example, when the state estimation device 2 outputs state information Z(t) indicating that the subject's alertness is decreasing, the output device 3 outputs an alarm sound to warn of the decrease in alertness.

[0017] In the first embodiment, as an example, pulse wave estimation device 1, state estimation device 2, and output device 3 are mounted on a vehicle (not shown), and the subject is the driver of the vehicle. That is, pulse wave estimation device 1 estimates the pulse wave of the vehicle driver. Also, state estimation device 2 estimates the state of the driver. Then, output device 3 outputs a warning sound to the driver based on the state of the driver estimated by state estimation device 2. An example configuration of the state estimating device 2 shown in FIG. 1 will be described in detail later, but first, an example configuration of the pulse wave estimating device 1 will be described in detail.

[0018] FIG. 2 is a diagram showing an example of the configuration of the pulse wave estimation device 1 according to the first embodiment. As shown in FIG. 2, pulse wave estimation device 1 includes an image acquisition unit 11, a skin area detection unit 12, a measurement area setting unit 13, a pulse wave original signal extraction unit 14, a simulation signal generation unit 15, a pulse wave estimation unit 16, and an output unit 17.

[0019] The captured image acquisition unit 11 acquires a captured image of a subject. In detail, the captured image acquisition unit 11 acquires a captured image of a driver of the vehicle by an imaging device (not shown) mounted on the vehicle. The imaging device is installed so as to be able to capture an image of the driver's skin area. The captured image acquisition unit 11 outputs the acquired captured image to the skin area detection unit 12. Here, the imaging device in the first embodiment will be described. The imaging device includes an imaging section (not shown) and an illumination section (not shown). The illumination section is, for example, configured with an LED (Light Emitting Diode). The illumination section irradiates light onto the imaging range of the imaging section. The imaging section images the imaging range irradiated with light by the illumination section emitting light. The imaging device may include one illumination section or multiple illumination sections. In the first embodiment, as an example, two illumination sections are provided in the imaging device at positions adjacent to the imaging section so as to sandwich the imaging section. The illumination light from the illumination section is most strongly irradiated near the center of the imaging range of the imaging section. As a result, in the captured image captured by the imaging section, the brightness of the central section is highest, and the brightness decreases from the central section to the edge of the captured image. In the first embodiment, the "center" is not limited to being strictly "center" and includes approximately the center.

[0020] The skin area detection unit 12 detects the skin area of ​​the subject from a frame Im(k) included in the captured image acquired by the captured image acquisition unit 11. The skin area detection unit 12 may detect the skin area using a known means. For example, the skin area detection unit 12 may detect the skin area using a cascade type face detector using Haar-like features. The skin area detection unit 12 generates skin area information S(k) indicating the detected skin area. The skin region information S(k) may include information indicating whether a skin region has been detected and information indicating the position and size of the detected skin region on the captured image. In the first embodiment, the skin region is represented by a rectangular region on the captured image, and the skin region information S(k) includes information indicating the position and size of the rectangular region on the captured image. Specifically, when the skin region corresponds to the subject's face, the skin region information S(k) indicates, for example, whether the subject's face is detected, the center coordinates Fc(Fcx, Fcy) of the rectangle surrounding the subject's face on the captured image, and the width Fcw and height Fch of the rectangle. The presence or absence of detection of the subject's face is represented, for example, by "1" if it is detected and "0" if it is not detected. The center coordinates of the rectangle surrounding the face are expressed in the coordinate system of frame Im(k). Note that the upper left corner of frame Im(k) is the origin, the rightward direction of frame Im(k) is the positive direction of the x-axis, and the downward direction of frame Im(k) is the positive direction of the y-axis. The skin region detection unit 12 outputs the generated skin region information S(k) to the measurement region setting unit 13.

[0021] The measurement region setting unit 13 sets a plurality of measurement regions for extracting a pulse wave original signal indicating a luminance change in an image region on the frame Im(k) corresponding to the skin region indicated by the skin region information S(k) based on the frame Im(k) of the captured image acquired by the captured image acquiring unit 11 and the skin region information S(k) output by the skin region detecting unit 12. Note that the measurement region setting unit 13 may acquire the captured image acquired by the captured image acquiring unit 11 via the skin region detecting unit 12. When the measurement area setting unit 13 sets a plurality of measurement areas, it generates measurement area information R(k) indicating the set plurality of measurement areas. The measurement area information R(k) includes information indicating the position and size of Rn (positive integer) measurement areas on the captured image. Each measurement area is defined as measurement area ri(k) (i=1, 2, . . . , Rn). In the first embodiment, the measurement area ri(k) is a quadrangle, and the position and size of the measurement area ri(k) are the coordinate values ​​of the four vertices of the quadrangle on the captured image.

[0022] 3A, 3B, and 3C are diagrams for explaining an example of a method for setting a measurement region by measurement region setting section 13 in pulse wave estimation device 1 according to embodiment 1. FIG. An example of a method in which the measurement area setting unit 13 sets a plurality of measurement areas will be described with reference to FIG. First, as shown in Fig. 3A and Fig. 3B, the measurement region setting unit 13 detects Ln (a positive integer) landmarks of facial parts such as the corners and corners of the eyes, the nose, and the mouth in the skin region sr indicated by the skin region information S(k). In Fig. 3A and Fig. 3B, the landmarks are indicated by circles. The measurement region setting unit 13 sets a vector storing the coordinate values ​​of the detected landmarks as L(k). The measurement region setting unit 13 may detect face parts using a known method such as a model called Constrained Local Model (CLM).

[0023] Next, the measurement area setting unit 13 sets the vertex coordinates of the quadrangle of the measurement area ri(k) based on the detected landmark. For example, the measurement area setting unit 13 sets the vertex coordinates of a quadrangle as shown in Fig. 3C, and sets Rn measurement areas ri(k).

[0024] To explain an example in which the measurement area setting unit 13 sets the measurement area ri(k) in a portion of the skin area sr corresponding to the cheek, the measurement area setting unit 13 selects a landmark LA1 on the contour of the face and a landmark LA2 on the nose. The measurement area setting unit 13 first selects the landmark LA2 on the nose, and then selects the landmark LA1 on the contour of the face that is closest to the landmark LA2 on the nose. Then, the measurement area setting unit 13 sets auxiliary landmarks a1, a2, and a3 so as to divide the line segment between the landmark LA1 and the landmark LA2 into four equal parts. Similarly, the measurement area setting unit 13 selects a landmark LB1 on the face outline and a landmark LB2 on the nose. The measurement area setting unit 13 also sets auxiliary landmarks b1, b2, and b3 so as to divide the line segment between the landmarks LB1 and LB2 into four equal parts. Note that the landmarks LB1 and LB2 may be selected, for example, from the landmarks on the face outline or the nose adjacent to the landmarks LA1 and LA2, respectively. The measurement area setting unit 13 sets a quadrilateral area surrounded by the auxiliary landmarks a1, b1, b2, and a2 as one measurement area R1. The auxiliary landmarks a1, b1, b2, and a2 each have vertex coordinates corresponding to the measurement area R1. Similarly, the measurement area setting unit 13 sets one measurement area R2 surrounded by the auxiliary landmarks a2, b2, b3, and a3, and the vertex coordinates of the measurement area R2.

[0025] Although an example of setting the measurement region ri(k) in the part corresponding to the cheek has been described here, the measurement region setting unit 13 similarly sets the measurement region ri(k) and the vertex coordinates of the measurement region ri(k) for, for example, other parts of the cheek and the skin region sr in the part corresponding to the chin. Although not shown in Fig. 3C, the measurement region setting unit 13 may set the measurement region ri(k) in the part of the skin region sr of the subject corresponding to the forehead or the part corresponding to the nose.

[0026] The measurement region setting unit 13 outputs the generated measurement region information R(k) to the pulse wave source signal extracting unit 14 and the simulation signal generating unit 15.

[0027] Based on frame Im(k) of the captured image acquired by captured image acquisition unit 11 and measurement region information R(k) output from measurement region setting unit 13, pulse wave source signal extraction unit 14 extracts a pulse wave source signal indicating a luminance change during a predetermined period, in other words, a period corresponding to frame number Tp, from each of a plurality of measurement regions ri(k) on frame Im(k) indicated by measurement region information R(k). The pulse wave source signal is a signal that is the source of a pulse wave. Pulse wave estimation device 1 estimates the subject's pulse wave using the pulse wave source signal. The subject's pulse wave is estimated by pulse wave estimation unit 16. Details of pulse wave estimation unit 16 will be described later. The pulse wave original signal extraction section 14 may acquire the captured image acquired by the captured image acquisition section 11 via the skin area detection section 12 and the measurement area setting section 13 . After extracting the pulse wave source signal, pulse wave source signal extractor 14 generates pulse wave source signal information W(t) indicating the extracted pulse wave source signal.

[0028] The pulse wave source signal information W(t) includes information indicative of the pulse wave source signal wi(t) extracted from the measurement region ri(k). The pulse wave source signal wi(t) is time-series data for Tp and is extracted, for example, based on frames Im(k-Tp+1), Im(k-Tp+2), . . . , Im(k) for the past Tp and measurement region information R(k-Tp+1), R(k-Tp+2), . . . , R(k). When extracting the pulse wave original signal wi(t), the pulse wave original signal extraction unit 14 calculates a luminance feature amount Gi(j) (j=k-Tp+1, k-Tp+2, . . . , k) of each measurement region ri(k) for each frame Im(k) of the captured image. The luminance feature amount Gi(j) is a value calculated for each measurement region ri(j) based on the luminance value in the frame Im(j) of the captured image. The luminance feature amount Gi(j) is the average or variance of the luminance values ​​of the pixels contained in the measurement region ri(j). In the first embodiment, as an example, the luminance feature amount Gi(j) is the average of the luminance values ​​of the pixels contained in the measurement region ri(j). The pulse wave original signal extraction unit 14 calculates a luminance feature amount Gi(j) (j=k-Tp+1, k-Tp+2, . . . , k) of each measurement region ri(k) of the captured image for each frame Im(k). Luminance feature The pulse wave source signal wi(t) is generated by arranging Gi(j) in a time series. That is, the pulse wave source signal extractor 14 sets the pulse wave source signal wi(t)=[Gi(k-Tp+1), Gi(k-Tp+2), . . . , Gi(k)].

[0029] The pulse wave source signal extractor 14 generates pulse wave source signal information W(t) by compiling the pulse wave source signals wi(t) for each measurement region ri(k). Pulse wave source signal extractor 14 outputs the generated pulse wave source signal information W(t) to pulse wave estimator 16.

[0030] The pulse wave source signal wi(t) contains various noise components in addition to the above-mentioned pulse wave components and the subject's (face in this case) movement components. Examples of noise components include noise caused by element defects in the imaging device. In order to remove these noise components, it is desirable to perform filtering as pre-processing on the pulse wave source signal wi(t). For example, the pulse wave source signal extractor 14 performs this filtering. In the filtering process, the pulse wave source signal wi(t) is filtered using, for example, a low-pass filter, a high-pass filter, or a band-pass filter. In the following description, it is assumed that the signal has been filtered using a band-pass filter. The bandpass filter may be, for example, a Butterworth filter, etc. As for the cutoff frequency of the bandpass filter, it is desirable that the lower cutoff frequency is 0.5 Hz and the higher cutoff frequency is 5.0 Hz, for example.

[0031] The simulation signal generating unit 15 generates a signal (hereinafter referred to as a "simulation signal") simulating a luminance change in the measurement area caused by a position change of the measurement area in a predetermined period, in other words, a period corresponding to the number of frames Tp, under ambient light, based on the frame Im(k) of the captured image acquired by the captured image acquiring unit 11, the measurement area information R(k) output from the measurement area setting unit 13, and the ambient light model. The simulation signal is a signal that becomes noise, and is removed from the pulse wave original signal when the pulse wave is estimated. The simulation signal generating unit 15 generates the simulation signal for each of a plurality of measurement regions ri(k) indicated by the measurement region information R(k). The simulation signal generating unit 15 may obtain the captured image obtained by the captured image obtaining unit 11 via the skin region detecting unit 12 and the measurement region setting unit 13 . After generating a simulation signal for each of the measurement regions ri(k), the simulation signal generating unit 15 generates simulation signal information M(t) by combining the generated simulation signals.

[0032] The simulated signal information M(t) includes information indicating a simulated signal mi(t) generated for a measurement region ri(k). The simulated signal mi(t) is time series data for Tp, and is generated based on, for example, frames Im(k-Tp+1), Im(k-Tp+2), . . . , Im(k) for the past Tp, measurement region information R(k-Tp+1), R(k-Tp+2), . . . , R(k), and an ambient light model. The environmental light model is stored in the simulation signal generating unit 15 in advance. The ambient light model is an ambient light distribution model that outputs a value simulating the luminance value of a coordinate on a captured image based on the spatial luminance distribution in an area illuminated by ambient light.

[0033] For example, when the ambient light when an image is captured is the illumination light from the illumination unit of the imaging device, in other words, when the ambient light is irradiated onto the imaging range so that the image is captured such that the center of the image is bright and the periphery is dark, the ambient light model is a two-dimensional Gaussian distribution as expressed by the following equation (1). TIFF0007630062000001.tif14166 q_(x,y) is a value that simulates the luminance value at the coordinates (x,y) on the captured image, and (x_c,y_c) is the central coordinate of the two-dimensional Gaussian distribution. a is a parameter that determines the height of the two-dimensional Gaussian distribution. σ is a parameter that determines the spread of the two-dimensional Gaussian distribution. The central coordinate of the two-dimensional Gaussian distribution is determined based on the spatial luminance distribution when the illumination light emitted by the illumination unit is captured by the imaging unit.

[0034] FIG. 4 is a diagram showing an example of the distribution of ambient light in a captured image represented by the ambient light model when a two-dimensional Gaussian distribution is used for the ambient light model in the first embodiment. The ambient light model shown in FIG. 4 is an ambient light model in which the ambient light is illumination light from a lighting unit of an imaging device.

[0035] The simulation signal generating unit 15 converts the coordinate values ​​(x, y) on the captured image included in the measurement region ri(k) into a simulation signal mi(t) using the above-mentioned ambient light model. In generating the simulation signal mi(t), the simulation signal generating unit 15 calculates a value Vi(j) (j=k-Tp+1, k-Tp+2, . . . , k) simulating the luminance value at the coordinates in each measurement region ri(k) for each frame Im(k) of the captured image using the above-mentioned ambient light model. The simulation signal generating unit 15 arranges the calculated Vi(j) for each frame Im(k) of the captured image in time series to generate the simulation signal mi(t). That is, the simulation signal generating unit 15 sets the simulation signal mi(t)=[Vi(k-Tp+1), Vi(k-Tp+2), . . . , Vi(k)]. Then, the simulation signal generating unit 15 generates a compilation of the simulation signals mi(t) in each measurement region ri(k) as simulation signal information M(t).

[0036] In the above description, the ambient light model is a two-dimensional Gaussian distribution, but this is merely an example. Any model other than a two-dimensional Gaussian distribution may be used as the ambient light model. For example, when the ambient light is the illumination light of the illumination unit of the imaging device, the actual measurement of the distribution of the illumination light may be used as the ambient light model. In other words, the ambient light model is not limited to one modeled as a mathematical formula, but may be one generated based on values ​​obtained by actually measuring the brightness of each position on the captured image. Furthermore, the simulation signal generating unit 15 may calculate one simulation signal mi(t) for each measurement region ri(k), or may calculate as many simulation signals as the number of coordinate points of each measurement region ri(k). In the first embodiment, as an example, the simulation signal generating unit 15 calculates one simulation signal mi(t) for each measurement region ri(k). The simulation signal generating unit 15 calculates one simulation signal mi(t) for each measurement region ri(k) using, for example, the coordinates of the center of gravity of the quadrangle of the measurement region ri(k).

[0037] Simulation signal generating section 15 outputs the generated simulation signal information M(t) to pulse wave estimating section 16.

[0038] Pulse wave estimation unit 16 estimates the subject's pulse wave based on pulse wave source signal information W(t) output from pulse wave source signal extraction unit 14 and simulation signal information M(t) output from simulation signal generation unit 15. In other words, pulse wave estimation unit 16 estimates the subject's pulse wave based on pulse wave source signal wi(t) extracted by pulse wave source signal extraction unit 14 and simulation signal mi(t) generated by simulation signal generation unit 15. Pulse wave estimation section 16 outputs pulse wave estimation result P(t), which is pulse wave information indicative of the estimated pulse wave, to output section 17. The pulse wave information may be, for example, time series data of the subject's pulse wave estimated by pulse wave estimation unit 16, or the subject's pulse rate. For ease of explanation, the pulse wave information is assumed to be the subject's pulse rate (number of beats per minute).

[0039] A method for estimating the subject's pulse wave by pulse wave estimator 16 will now be described in detail. Pulse wave estimation unit 16 calculates pulse wave signal di(t) from which luminance changes due to movement of the subject's skin region, specifically the subject's face, have been removed, for example, by subtracting simulation signal mi(t) from pulse wave original signal wi(t). That is, pulse wave estimation unit 16 calculates pulse wave signal di(t) for each measurement region ri(k). Next, pulse wave estimation unit 16 calculates pulse wave signal information D(t) by adding up the pulse wave signals di(t) corresponding to each measurement region ri(k). In other words, pulse wave estimation unit 16 calculates one piece of pulse wave signal information D(t) for all measurement regions ri(k). Pulse wave estimation unit 16 then performs a Fourier transform on pulse wave signal information D(t) and calculates the peak frequency in the frequency power spectrum as the pulse rate.

[0040] For example, when simulation signal generating unit 15 calculates simulation signals mi(t) for the number of coordinate points in each measurement region ri(k), pulse wave estimating unit 16 multiplies each simulation signal mi(t) by a coefficient (hereinafter referred to as a "simulation signal coefficient") corresponding to the number of simulation signals mi(t) calculated for one measurement region ri(k). Pulse wave estimating unit 16 then subtracts the simulation signal mi(t) multiplied by the simulation signal coefficient from the pulse wave original signal wi(t). As a specific example, in each measurement region ri(k), simulation signal generation unit 15 sets the maximum amplitude value of simulation signal mi(t) for the number of coordinate points as mi_amp(t) and the maximum amplitude value of pulse wave original signal wi(t) as wi_amp(t). In this case, pulse wave estimation unit 16 multiplies each simulation signal mi(t) by a simulation signal coefficient "wi_amp / mi_amp". Then, pulse wave estimation unit 16 Coefficients for simulated signals Each simulation signal mi(t) after multiplication by "wi_amp / mi_amp" is subtracted from the pulse wave original signal wi(t).

[0041] Pulse wave estimation section 16 outputs pulse wave estimation result P(t) to output section 17.

[0042] Output section 17 outputs pulse wave estimation result P(t) output from pulse wave estimating section 16 to pulse wave information acquiring section 21 of state estimation device 2. The function of output section 17 may be provided in pulse wave estimating section 16.

[0043] Next, a configuration example of the state estimating device 2 according to the first embodiment will be described. As shown in FIG. 1, condition estimation device 2 includes pulse wave estimation device 1, pulse wave information acquisition section 21, condition estimation section 22, and output section .

[0044] Pulse wave information acquisition section 21 acquires pulse wave estimation result P(t) output from pulse wave estimation device 1. Pulse wave information acquirer 21 outputs the acquired pulse wave estimation result P(t) to state estimater 22.

[0045] State estimation unit 22 estimates the state of the subject based on pulse wave estimation result P(t) output from pulse wave information acquisition unit 21, in other words, based on the subject's pulse wave estimated by pulse wave estimation device 1. In the first embodiment, state estimation unit 22 estimates the state of the subject based on the subject's pulse rate estimated by pulse wave estimation device 1. In detail, state estimation unit 22 estimates the driver's alertness as the state of the driver based on the subject's, in other words, the driver's pulse rate. For example, the alertness is represented by two levels of alertness (1: sleepy, 2: awake).

[0046] For example, state estimation unit 22 stores pulse wave estimation result P(t) when it is output from pulse wave information acquisition unit 21, and calculates the subject's pulse rate for 10 minutes after starting driving as the subject's reference pulse rate based on the stored pulse wave estimation result P(t). State estimation unit 22 may determine that driving has started, for example, when the power of the vehicle is turned on. State estimation unit 22 may also determine that driving has started based on signals from various sensors mounted on the vehicle, such as a shift position sensor. In general, people are assumed to be in an awake state immediately after starting to drive. Therefore, the state estimation unit 22 calculates the subject's pulse rate for 10 minutes after starting to drive as a reference pulse rate that is a criterion for determining whether the subject is in an awake state or not. If the subject's pulse rate falls below the reference pulse rate, it can be estimated that the subject's level of wakefulness has decreased. State estimation unit 22 estimates the level of alertness of the subject by comparing the pulse wave estimation result P(t) output from pulse wave information acquisition unit 21, in other words, the subject's pulse rate estimated by pulse wave estimation device 1 with a reference pulse rate. When the subject's pulse rate falls from the reference pulse rate by a preset threshold value (hereinafter referred to as the "decrease determination threshold value") or more, state estimation unit 22 determines that the alertness level has decreased, and estimates the alertness level to be "1."

[0047] In the first embodiment, state estimation unit 22 calculates the reference pulse rate based on pulse wave estimation result P(t) output from pulse wave information acquisition unit 21, in other words, the subject's pulse wave estimated by pulse wave estimation device 1, but this is merely an example. For example, the reference pulse rate may be set in advance to a pulse rate in a general human wakefulness state, and the reference pulse rate may be stored in state estimation unit 22. State estimation unit 22 estimates the subject's wakefulness level based on the stored reference pulse rate. However, as described above, state estimation unit 22 can estimate the wakefulness level according to the subject, taking into account individual differences, by calculating the reference pulse rate for the subject based on pulse wave estimation result P(t) output from pulse wave estimation device 1.

[0048] The state estimation unit 22 may estimate the level of wakefulness of the subject by combining a method of estimating the level of wakefulness of the subject based on the pulse wave estimation result P(t) with a known method of estimating the level of wakefulness. The known method of estimating the level of wakefulness may be, for example, a method of estimating the level of wakefulness by detecting whether the proportion of time the eyes are closed is increasing.

[0049] The state estimation unit 22 outputs state information Z(t) relating to the estimated state of the subject to the output unit 23.

[0050] The output unit 23 outputs the state information Z(t) output from the state estimation unit 22 to the output device 3. The function of the output unit 23 may be included in the state estimation unit 22 .

[0051] In the first embodiment, as described above, the state information Z(t) regarding the state of the subject estimated by the state estimation unit 22 is output to the output device 3, but this is merely an example. For example, the state estimation unit 22 may store the state information Z(t). In this case, it is not essential that the state estimation device 2 is connected to the output device 3. Furthermore, the state estimation device 2 may be configured not to include the output unit 23.

[0052] The operation of pulse wave estimation device 1 according to the first embodiment will be described. FIG. 5 is a flowchart for explaining the operation of pulse wave estimation device 1 according to the first embodiment.

[0053] The captured image acquisition unit 11 acquires a captured image of a subject (step ST1). The captured image acquisition unit 11 outputs the acquired captured image to the skin area detection unit 12.

[0054] The skin region detection section 12 detects a skin region from a frame Im(k) included in the captured image acquired by the captured image acquisition section 11 in step ST1 (step ST2). The skin area detection unit 12 generates skin area information S(k) indicating the detected skin area. The skin region detection unit 12 outputs the generated skin region information S(k) to the measurement region setting unit 13.

[0055] Based on the frame Im(k) of the captured image acquired by the captured image acquisition unit 11 in step ST1 and the skin area information S(k) output by the skin area detection unit 12 in step ST2, the measurement area setting unit 13 sets multiple measurement areas in the image area on the frame Im(k) corresponding to the skin area indicated by the skin area information S(k) for extracting a pulse wave original signal indicating a change in brightness (step ST3). When the measurement region setting unit 13 sets a plurality of measurement regions, it generates measurement region information R(k) indicating the plurality of measurement regions that have been set. The measurement region setting unit 13 outputs the generated measurement region information R(k) to the pulse wave source signal extracting unit 14 and the simulation signal generating unit 15.

[0056] Based on frame Im(k) of the captured image acquired by image acquisition unit 11 in step ST1 and measurement area information R(k) output from measurement area setting unit 13 in step ST3, pulse wave source signal extraction unit 14 extracts a pulse wave source signal indicating a change in brightness during a predetermined period, in other words, a period corresponding to frame number Tp, from each of a plurality of measurement areas ri(k) on frame Im(k) indicated by measurement area information R(k) (step ST4). After extracting the pulse wave source signal, pulse wave source signal extractor 14 generates pulse wave source signal information W(t) indicating the extracted pulse wave source signal. Pulse wave source signal extractor 14 outputs the generated pulse wave source signal information W(t) to pulse wave estimator 16.

[0057] The simulation signal generating unit 15 generates a simulation signal simulating a change in luminance of the measurement area caused by a change in position of the measurement area under ambient light during a predetermined period, in other words, a period corresponding to the number of frames Tp, based on the frame Im(k) of the captured image acquired by the captured image acquiring unit 11 in step ST1, the measurement area information R(k) output from the measurement area setting unit 13 in step ST4, and the ambient light model (step ST5). Then, the simulation signal generating unit 15 generates simulation signal information M(t) by combining the simulation signals generated for each of the measurement regions ri(k). Simulation signal generating section 15 outputs the generated simulation signal information M(t) to pulse wave estimating section 16.

[0058] The pulse wave estimation unit 16 estimates the subject's pulse wave based on the pulse wave source signal information W(t) output from the pulse wave source signal extraction unit 14 in step ST4 and the simulation signal information M(t) output from the simulation signal generation unit 15 in step ST5 (step ST6). Pulse wave estimation section 16 outputs pulse wave estimation result P(t), which is pulse wave information indicative of the estimated pulse wave, to output section 17. Output section 17 outputs pulse wave estimation result P(t) output from pulse wave estimating section 16 to pulse wave information acquiring section 21 of state estimation device 2.

[0059] 5, the process is performed in the order of step ST4 and step ST5, but this is merely an example. The process order of step ST4 and step ST5 may be reversed, or the process of step ST4 and the process of step ST5 may be performed in parallel.

[0060] The operation of the state estimating device 2 according to the first embodiment will be described. FIG. 6 is a flowchart for explaining the operation of the state estimating device 2 according to the first embodiment.

[0061] Pulse wave information acquiring section 21 acquires pulse wave estimation result P(t) output from pulse wave estimation device 1 (step ST11). Pulse wave information acquirer 21 outputs the acquired pulse wave estimation result P(t) to state estimater 22.

[0062] Condition estimating section 22 estimates the condition of the subject based on pulse wave estimation result P(t) output from pulse wave information acquiring section 21 in step ST11 (step ST12). The state estimation unit 22 outputs state information Z(t) relating to the estimated state of the subject to the output unit 23. The output unit 23 outputs the state information Z(t) output from the state estimation unit 22 to the output device 3.

[0063] As described above, a method is known for estimating a pulse wave in a non-contact manner from minute luminance changes in a human skin region in an image captured by an imaging device. For example, as described in the following reference 1, a method is known in which a plurality of measurement regions are set on a face image of a subject, a frequency power spectrum of a luminance signal acquired in each measurement region is calculated, a pulse wave is synthesized according to a peak frequency of the frequency power spectrum, and a pulse rate is estimated from a peak of the frequency power spectrum of the synthesized pulse wave.

[0064] [Reference 1] Mayank Kumar, et al., “DistancePPG: Robust non-contact vital signs monitoring using a camera”, Biomedical optics express, 6(5), 1565-1588, 2015

[0065] However, the method disclosed in Reference 1 has a problem in that the accuracy of estimating the pulse wave decreases when the subject's face moves. When the subject's face moves, a component corresponding to the facial movement appears as a peak in the frequency power spectrum, and the component corresponding to the facial movement is erroneously detected as the pulse wave, instead of the frequency component corresponding to the pulse wave.

[0066] To address this problem, as described above, there is known a technique for removing specific frequency components obtained by frequency analysis of position information of a face region in an image capture area as noise from an image signal from the face region. However, such conventional techniques do not take into consideration that in a scene where the distribution of ambient light in the imaging range is non-uniform, the pulse wave estimated from minute changes in luminance of the subject's skin surface in the captured image is affected not only by changes in the position of the skin area but also by ambient light. As a result, the conventional techniques are unable to obtain a sufficient noise removal effect, and the estimation accuracy of the subject's pulse wave is low.

[0067] FIG. 7 is a diagram for explaining the effect of environmental light, which occurs when the distribution of environmental light is non-uniform, on a signal used for estimating a subject's pulse wave based on a captured image. figure 7A and 7B are diagrams illustrating an example of a captured image captured in a scene with a uniform distribution of ambient light, and an example of a captured image captured in a scene with a non-uniform distribution of ambient light, respectively. In Figures 7A and 7B, the captured images are indicated by ImA and ImB, respectively. The color intensity of the captured images shown in Figures 7A and 7B represents the luminance distribution according to the ambient light. The lighter the color (whiter the image), the higher the luminance, and the darker the color (blacker the image), the lower the luminance. In the captured image, a subject (indicated as SA in FIG. 7A and as SB in FIG. 7B) is imaged. For example, assume that the subject moves his / her face, and thus the subject's face moves from left to right in the frames (Tp frames) of the time-series captured images in Fig. 7. For convenience, Fig. 7 shows only one frame of the captured image, and the movement of the subject's face position in the frames of the time-series captured images is indicated by an arrow. As shown in Fig. 7A, in a scene with uniform ambient light, even if the subject moves his / her face, in other words, no matter where the subject's face is in multiple frames of the captured images in time series, both the signal indicating the position of the face and the signal indicating the luminance of the surface of the face are not affected by the ambient light. On the other hand, as shown in Fig. 7B, in a scene with non-uniform ambient light, even if the subject moves his / her face, the signal indicating the position of the face is not affected by the ambient light, whereas the signal indicating the luminance of the surface of the subject's face is affected by the ambient light when the subject moves his / her face, in other words, when the position of the subject's face changes in multiple frames of the captured images in time series. The closer the subject's face is to the center of the captured images, the higher the luminance of the surface of the subject's face is affected by the ambient light. The above-mentioned conventional technology can handle the scene described using FIG. 7A, but cannot handle the scene described using FIG. 7B.

[0068] In response to this, pulse wave estimation device 1 according to embodiment 1 generates a simulation signal mi(t) simulating a change in luminance of a measurement region caused by a change in position of the measurement region in a predetermined period, in other words, a period corresponding to the number of frames Tp, under ambient light, based on a frame Im(k) of a captured image, measurement region information R(k), and an ambient light model. Pulse wave estimation device 1 then uses the generated simulation signal mi(t) for noise removal. Pulse wave estimation device 1 generates simulation signal mi(t) by taking into consideration that in a scene where the distribution of ambient light in the imaging range is non-uniform, the pulse wave estimated from minute changes in luminance of the subject's skin region in the captured image is affected by the ambient light, so that even in a scene where the distribution of ambient light in the imaging range is non-uniform, noise due to changes in the position of the subject's skin region can be removed and the subject's pulse wave can be estimated. Furthermore, state estimation device 2 according to embodiment 1 can estimate the state of the subject even in a scene where the distribution of ambient light in the imaging range is non-uniform.

[0069] In the above first embodiment, it is assumed that there is one ambient light model, but there may be multiple ambient light models. The simulation signal generating unit 15 can generate a simulation signal mi(t) and simulation signal information M(t) that combines the simulation signal mi(t) by using a plurality of ambient light models.

[0070] For example, the simulated signal generating unit 15 can determine the imaging conditions based on information acquired from sensors etc. provided in the vehicle, and switch between a plurality of ambient light models to be used, thereby generating a simulated signal mi(t) and simulated signal information M(t) that combines the simulated signal mi(t). To explain with a specific example, the simulated signal generating unit 15 can generate the simulated signal mi(t) and the simulated signal information M(t) combining the simulated signal mi(t) using three ambient light models: an ambient light model (first ambient light model) in which the illumination light emitted by the illumination unit of the imaging device is the ambient light; an ambient light model (second ambient light model) in which the light entering the imaging range so that the luminance on the left side of the captured image is high is the ambient light; and an ambient light model (third ambient light model) in which the light entering the imaging range so that the luminance on the right side of the captured image is high is the ambient light. The light entering the imaging range so that the luminance on the left or right side of the captured image is high is assumed to be light entering from outside the vehicle, such as sunlight. The first ambient light model, the second ambient light model, and the third ambient light model are generated in advance and are provided in the simulated signal generating unit 15.

[0071] FIG. 8 is a diagram for explaining an example of a method for generating a simulation signal mi(t) based on a plurality of environmental light models by the simulation signal generating unit 15 in the first embodiment. In Fig. 8, ImC shows an image captured by an imaging device in a vehicle traveling at night, ImD shows an image captured by an imaging device in a vehicle traveling during the day in an environment where light enters the imaging range so that the luminance on the left side of the captured image is higher, and ImE shows an image captured by an imaging device in a vehicle traveling during the day in an environment where light enters the imaging range so that the luminance on the right side of the captured image is higher. Also, in Fig. 8, 81a shows a light intensity distribution represented by an environmental light model based on environmental light in a situation where the captured image shown in ImC is captured, 81b shows a light intensity distribution represented by an environmental light model based on environmental light in a situation where the captured image shown in ImD is captured, and 81c shows a light intensity distribution represented by an environmental light model based on environmental light in a situation where the captured image shown in ImE is captured. In 81a, 81b, and 81c, the intensity of the environmental light is shown by color density, and the whiter the color, the higher the intensity. For ease of understanding, Fig. 8 illustrates an imaging unit (indicated by 401 in Fig. 8) and an illumination unit (indicated by 402 in Fig. 8) that the imaging device includes. Also, in Fig. 8, Dr indicates the face of the subject, here the driver.

[0072] For example, in a nighttime driving scene, the only ambient light is the illumination light emitted by the illumination unit provided in the imaging device. Therefore, the simulated signal generating unit 15 generates the simulated signal mi(t) and the simulated signal information M(t) using a first ambient light model representing a light intensity distribution as shown in 81a in Fig. 8. The simulated signal generating unit 15 may determine that it is nighttime based on, for example, sensor information from an illuminance sensor that detects the brightness of the surroundings of the vehicle. For example, in a daytime driving scene where light enters from outside the vehicle so that the luminance on the left side of the captured image is high, the ambient light is the light entering from outside the vehicle. Therefore, the simulated signal generating unit 15 generates the simulated signal mi(t) and the simulated signal information M(t) using a second ambient light model that represents a light intensity distribution as shown in 81b in Fig. 8. The simulated signal generating unit 15 may determine, for example, based on the sensor information from the illuminance sensor, that the situation is daytime and light enters from outside the vehicle so that the luminance on the left side of the captured image is high. For example, in a daytime driving scene where light enters from outside the vehicle so that the luminance on the right side of the captured image is high, the ambient light is the light entering from outside the vehicle. Therefore, the simulated signal generating unit 15 generates the simulated signal mi(t) and the simulated signal information M(t) using a third ambient light model that represents a light intensity distribution as shown in 81c in Fig. 8. The simulated signal generating unit 15 may determine, for example, based on the sensor information from the illuminance sensor, that the situation is one in which light enters from outside the vehicle so that the luminance on the right side of the captured image is high during the day.

[0073] In the above-described example, the simulation signal generating unit 15 switches between a plurality of ambient light models to generate the simulation signal mi(t) and the simulation signal information M(t) that combines the simulation signal mi(t), but this is merely one example. For example, the simulation signal generating unit 15 may use a plurality of ambient light models to generate a simulation signal mi(t) for each of the ambient light models used, and multiply the generated simulation signal mi(t) by a coefficient (hereinafter referred to as a "model coefficient") according to the situation. Then, the simulation signal generating unit 15 generates simulation signal information M(t) including the simulation signal mi(t) after multiplication by the model coefficient.

[0074] As a specific example, in a nighttime driving scene, the simulation signal generating unit 15 sets the model coefficient corresponding to the first ambient light model to "1" and the model coefficients corresponding to the second ambient light model and the third ambient light model to "0." In this way, the simulation signal generating unit 15 sets the model coefficients so that the model coefficient corresponding to the ambient light model according to the situation is large. The simulation signal generating unit 15 generates a simulation signal mi(t) (hereinafter referred to as "first simulation signal") using a first environmental light model, a simulation signal mi(t) (hereinafter referred to as "second simulation signal") using a second environmental light model, and a simulation signal mi(t) (hereinafter referred to as "third simulation signal") using a third environmental light model. Next, the simulation signal generating unit 15 multiplies the first simulation signal by a model coefficient "1", and multiplies the second and third simulation signals by a model coefficient "0". Then, the simulation signal generating unit 15 generates simulation signal information M(t) that combines the first, second, and third simulation signals after multiplication by the model coefficients.

[0075] In this way, by configuring the simulated signal generating unit 15 to generate the simulated signal mi(t) and the simulated signal information M(t) using a plurality of ambient light models, the pulse wave estimation device 1 can estimate the subject's pulse wave with high accuracy for scenes with a variety of ambient light.

[0076] In the above embodiment 1, pulse wave estimation unit 16 in pulse wave estimation device 1 unconditionally subtracts simulated signal mi(t) from pulse wave source signal wi(t) to calculate pulse wave signal di(t) from which luminance changes due to movement of the subject's skin region, more specifically, the subject's face, have been removed, but this is merely an example. For example, pulse wave estimation unit 16 may subtract simulated signal mi(t) from pulse wave source signal wi(t) to calculate pulse wave signal di(t) when a certain condition is met, and may use pulse wave source signal wi(t) as pulse wave signal di(t) without subtracting simulated signal mi(t) from pulse wave source signal wi(t) when the condition is not met. Specifically, when the maximum amplitude value of simulation signal mi(t) is smaller than a preset threshold value (hereinafter referred to as the "amplitude determination threshold value"), pulse wave estimation unit 16 determines that noise due to the subject's movement is sufficiently small and does not perform the above subtraction, but uses pulse wave source signal wi(t) as pulse wave signal di(t). However, when the maximum amplitude value of simulation signal mi(t) is equal to or greater than the amplitude determination threshold value, pulse wave estimation unit 16 may perform the above subtraction to calculate pulse wave signal di(t).

[0077] The subject's movement is not limited to left-right movement, but may also be forward-backward movement. Therefore, in the above-described first embodiment, the simulation signal generating unit 15 may take this into consideration when generating the simulation signal mi(t) and the simulation signal information M(t) that combines the simulation signal mi(t). In detail, the simulation signal generating unit 15 may generate the simulation signal mi(t) based on, for example, a signal obtained by normalizing the scale (amplitude) of the luminance change in the measurement area ri(k) caused by the left and right movement of the subject, and the scale of the luminance change in the measurement area ri(k) caused by the forward and backward movement of the subject.

[0078] The simulation signal generating unit 15 generates the simulation signal mi(t) by the method described in the above embodiment 1 as a simulation signal mi(t) (hereinafter referred to as the "first simulation signal mi1(t)") that simulates the luminance change in the measurement area ri(k) caused by the left and right movement of the subject. The simulation signal generating unit 15 generates a simulation signal mi(t) (hereinafter referred to as the "second simulation signal mi2(t)") that simulates a change in luminance in the measurement region ri(k) caused by the subject's forward and backward movements, in addition to the first simulation signal mi1(t). A specific method in which the simulation signal generating unit 15 generates the second simulation signal mi2(t) will be described. The simulation signal generating unit 15 first calculates the circumscribing rectangle Rec(k) of all the measurement regions ri(k) and obtains the center coordinates O(k)(x, y)=(xc, yc). above The diagram on the left shows an example of a circumscribing rectangle Rec(k) of the measurement region ri(k) and its center coordinates O(k). Next, the simulation signal generating unit 15 calculates the distance Dis(k) between the coordinates of each vertex of each measurement area ri(k) and the center coordinate O(k). For example, if the coordinates of one corner of the measurement area ri(k) are (x, y)=(xi_rb, yi_rb) and the corresponding distance to the center coordinate O(k) is Dis_i_rb, the simulation signal generating unit 15 calculates Dis_i_rb using the following formula (2). under The diagram on the right shows an image of the distance between the vertex coordinates of one corner of the measurement area ri(k) and the corresponding center coordinates O(k). TIFF0007630062000002.tif12166

[0079] Then, the simulation signal generating unit 15 calculates the vertex coordinates and the circumscribing rectangle of each measurement region ri(k) obtained as described above. Rec(k) The distance Dis(k) between the center coordinate O(k) of the signal mi2(t) is defined as the second simulation signal mi2(t).

[0080] In the above example, the simulation signal generating unit 15 calculates the vertex coordinates of each measurement region ri(k) and the circumscribing rectangle Rec(k) However, the present invention is not limited to this. For example, the simulation signal generator 15 may calculate the distance between the coordinates of the center of gravity of each measurement region ri(k) and the coordinates of the center of the circumscribing rectangle Rec(k). In the above example, the simulation signal generator 15 may use the average value of the vertex coordinates of each measurement region ri(k), in other words, the center of gravity, instead of the center coordinate O(k) of the circumscribed rectangle Rec(k), and calculate the distance Dis(k) between the center of gravity and the vertex coordinates of each measurement region ri(k). The simulation signal generator 15 may determine the reference coordinates for each measurement region ri(k).

[0081] Next, the simulation signal generating unit 15 multiplies the first simulation signal mi1(t) and the second simulation signal mi2(t) by a coefficient (hereinafter referred to as the "normalization coefficient") to normalize the scale. In other words, the simulation signal generating unit 15 multiplies the first simulation signal mi1(t) and the second simulation signal mi2(t) by the normalization coefficient to even out the difference in scale between the first simulation signal mi1(t) and the second simulation signal mi2(t). For example, the simulation signal generating unit 15 multiplies either the first simulation signal mi1(t) or the second simulation signal mi2(t) by a normalization coefficient. For example, the simulation signal generating unit 15 may multiply both the first simulation signal mi1(t) and the second simulation signal mi2(t) by a normalization coefficient. Note that the normalization coefficient may be a preset value, or the simulation signal generating unit 15 may calculate the normalization coefficient so that the average values ​​of the scales of the first simulation signal mi1(t) and the second simulation signal mi2(t) are the same. FIG. 10 is a diagram showing an example of the first test signal mi1(t) and the second test signal mi2(t) in which the difference in scale is made uniform by multiplying the signal by a normalization coefficient in the test signal generating unit 15 in the first embodiment. The simulation signal generating unit 15 then combines the first simulation signal mi1(t) and the second simulation signal mi2(t) after multiplication by the normalization coefficient to generate a simulation signal mi(t).

[0082] In this way, the simulation signal generating unit 15 generates the simulation signal mi(t) based on a signal obtained by normalizing the scale of a signal (first simulation signal mi1(t)) simulating the change in luminance of the measurement area ri(k) caused by the left and right movement of the subject, and the scale of a signal (second simulation signal mi2(t)) simulating the change in luminance of the measurement area ri(k) caused by the forward and backward movement of the subject.As a result, the pulse wave estimation device 1 can eliminate noise caused by changes in the position of the subject's skin area while also taking into account the forward and backward movement of the subject, and estimate the subject's pulse wave.

[0083] In the above embodiment 1, in the pulse wave estimation device 1, the measurement region setting unit 13 sets a plurality of measurement regions in the image region on the frame Im(k) corresponding to the skin region indicated by the skin region information S(k), but this is merely an example. For example, the measurement region setting unit 13 may set only one measurement region. In addition, in the pulse wave estimation device 1, when the simulation signal generating unit 15 generates a simulation signal, it may select one measurement area from the multiple measurement areas set by the measurement area setting unit 13, and generate a simulation signal for only the selected one measurement area. When the measurement area setting unit 13 sets only one measurement area, or when the simulation signal generating unit 15 selects one measurement area, it is preferable that the one measurement area that is set or the one measurement area that is selected is set to a skin area with little movement, such as mouth movement due to conversation or movement due to changes in facial expression. The "movement of the skin area" here refers to "movement of the skin area" that is not accompanied by movement of the head position itself, but occurs due to factors other than the movement of the head position itself. For example, when the measurement region setting unit 13 sets only one measurement region, or when the simulation signal generating unit 15 selects one measurement region, it is preferable that the one measurement region that is set or selected is set to a skin region corresponding to the cheek or forehead. However, since the skin region corresponding to the forehead may be hidden by bangs, etc., the skin region corresponding to the cheek is more preferable than the forehead.

[0084] In addition, in the above-described first embodiment, pulse wave estimation device 1 is provided in state estimation device 2, but this is merely an example. Pulse wave estimation device 1 may be provided outside state estimation device 2 and connected to state estimation device 2 outside state estimation device 2. In the above embodiment 1, the illumination unit is provided in the imaging device, but this is merely an example. For example, an illumination device may be provided outside the imaging device, and the illumination device may irradiate light onto the imaging range from outside the imaging device.

[0085] In the above embodiment 1, the subject is the driver of the vehicle, but this is merely an example The subject may be a passenger other than the driver of the vehicle.

[0086] In addition, in the above embodiment 1, pulse wave estimation device 1 and state estimation device 2 are in-vehicle devices, and image acquisition unit 11, skin area detection unit 12, measurement area setting unit 13, pulse wave original signal extraction unit 14, simulation signal generation unit 15, pulse wave estimation unit 16, output unit 17, pulse wave information acquisition unit 21, state estimation unit 22, and output unit 23 are provided in the in-vehicle device. Without being limited to this, some of the image acquisition unit 11, skin area detection unit 12, measurement area setting unit 13, pulse wave original signal extraction unit 14, simulation signal generation unit 15, pulse wave estimation unit 16, output unit 17, pulse wave information acquisition unit 21, state estimation unit 22, and output unit 23 may be mounted on an in-vehicle device of a vehicle, and the others may be provided on a server connected to the in-vehicle device via a network, so that the system is formed by the in-vehicle device and the server. In addition, the image acquisition unit 11, skin area detection unit 12, measurement area setting unit 13, pulse wave original signal extraction unit 14, simulation signal generation unit 15, pulse wave estimation unit 16, output unit 17, pulse wave information acquisition unit 21, state estimation unit 22, and output unit 23 may all be provided in the server.

[0087] Moreover, the pulse wave estimation device 1 and the condition estimation device 2 according to the above-mentioned embodiment 1 are not limited to in-vehicle devices mounted on vehicles, but can also be applied to, for example, home appliances. Moreover, the subjects are not limited to vehicle occupants, but can be various people. As a specific example, pulse wave estimation device 1 and condition estimation device 2 may be mounted on a television installed in a living room of a house. In this case, the subject is a user such as a resident of the house. Pulse wave estimation device 1 estimates the user's pulse wave based on an image captured by an imaging device mounted on the television. Note that it is sufficient for the imaging device to be able to capture at least an image of the subject's skin. Condition estimation device 2 estimates and records the user's physical condition from an increase or decrease in pulse rate based on a pulse wave estimation result P(t) based on the user's pulse wave estimated by pulse wave estimation device 1, for example.

[0088] 11A and 11B are diagrams illustrating an example of a hardware configuration of pulse wave estimation device 1 according to embodiment 1. FIG. In the first embodiment, the functions of captured image acquisition section 11, skin area detection section 12, measurement area setting section 13, pulse wave original signal extraction section 14, simulation signal generation section 15, pulse wave estimation section 16, and output section 17 are realized by processing circuit 101. That is, pulse wave estimation device 1 includes processing circuit 101 for controlling the estimation of the subject's pulse wave using an environmental light model. The processing circuit 101 may be dedicated hardware as shown in FIG. 11A, or may be a processor 104 that executes a program stored in a memory as shown in FIG. 11B.

[0089] When the processing circuit 101 is dedicated hardware, the processing circuit 101 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof.

[0090] When the processing circuit is the processor 104, the functions of the captured image acquisition unit 11, skin area detection unit 12, measurement area setting unit 13, pulse wave source signal extraction unit 14, simulation signal generation unit 15, pulse wave estimation unit 16, and output unit 17 are realized by software, firmware, or a combination of software and firmware. The software or firmware is written as a program and stored in the memory 105. The processor 104 reads out and executes the program stored in the memory 105 to execute the functions of the captured image acquisition unit 11, skin area detection unit 12, measurement area setting unit 13, pulse wave source signal extraction unit 14, simulation signal generation unit 15, pulse wave estimation unit 16, and output unit 17. That is, the pulse wave estimation device 1 includes a memory 105 for storing a program that, when executed by the processor 104, results in the execution of steps ST1 to ST6 in FIG. 5 described above. It can also be said that the programs stored in memory 105 cause the computer to execute the procedures or methods of the processes of image acquisition unit 11, skin area detection unit 12, measurement area setting unit 13, pulse wave source signal extraction unit 14, simulation signal generation unit 15, pulse wave estimation unit 16, and output unit 17. Here, memory 105 corresponds to, for example, non-volatile or volatile semiconductor memory such as RAM, ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), etc., or a magnetic disk, flexible disk, optical disk, compact disk, mini disk, DVD (Digital Versatile Disc), etc.

[0091] The functions of the captured image acquisition unit 11, skin area detection unit 12, measurement area setting unit 13, pulse wave source signal extraction unit 14, simulation signal generation unit 15, pulse wave estimation unit 16, and output unit 17 may be partially realized by dedicated hardware and partially realized by software or firmware. For example, the functions of the captured image acquisition unit 11 and output unit 17 may be realized by a processing circuit 101 as dedicated hardware, and the functions of the skin area detection unit 12, measurement area setting unit 13, pulse wave source signal extraction unit 14, simulation signal generation unit 15, and pulse wave estimation unit 16 may be realized by a processor 104 reading and executing a program stored in a memory 105. The pulse wave estimation device 1 also includes a device such as an imaging device, an input interface device 102 that performs wired or wireless communication, and an output interface device 103.

[0092] An example of the hardware configuration of the state estimating device 2 according to the first embodiment is also configured as shown in FIGS. 11A and 11B. In the first embodiment, the functions of pulse wave information acquisition section 21, condition estimation section 22, and output section 23 are realized by processing circuit 101. That is, condition estimation device 2 includes processing circuit 101 for performing control to estimate the condition of a subject, based on pulse wave information relating to the subject's pulse wave estimated by pulse wave estimation device 1.

[0093] When the processing circuit is processor 104, the functions of pulse wave information acquisition section 21, state estimation section 22, and output section 23 are realized by software, firmware, or a combination of software and firmware. The software or firmware is written as a program and stored in memory 105. Processor 104 executes the functions of pulse wave information acquisition section 21, state estimation section 22, and output section 23 by reading and executing the program stored in memory 105. That is, state estimation device 2 includes memory 105 for storing a program that, when executed by processor 104, results in the execution of steps ST11 to ST12 in FIG. 6 described above. It can also be said that the program stored in memory 105 causes a computer to execute the procedures or methods of the processes of pulse wave information acquisition section 21, state estimation section 22, and output section 23.

[0094] It is also possible to realize some of the functions of pulse wave information acquisition unit 21, state estimation unit 22, and output unit 23 with dedicated hardware and some with software or firmware. For example, the functions of pulse wave information acquisition unit 21 and output unit 23 can be realized by processing circuit 101 as dedicated hardware, and the function of state estimation unit 22 can be realized by processor 104 reading and executing a program stored in memory 105. Moreover, the state estimation device 2 includes an input interface device 102 and an output interface device 103 that perform wired or wireless communication with devices such as the pulse wave estimation device 1 or the output device 3.

[0095] As described above, pulse wave estimation device 1 according to embodiment 1 is configured to include captured image acquisition unit 11 that acquires an image of a person (subject), skin area detection unit 12 that detects the person's skin area from the captured image, measurement area setting unit 13 that sets measurement area ri(k) in an area on the captured image corresponding to the skin area for extracting a pulse wave original signal indicating a change in luminance, pulse wave original signal extraction unit 14 that extracts a pulse wave original signal wi(t) based on the change in luminance in the measurement area ri(k) on the captured image, simulation signal generation unit 15 that generates a simulation signal mi(t) that simulates the change in luminance of measurement area ri(k) caused by the change in position of measurement area ri(k) under ambient light, based on the change in position of measurement area ri(k) and a distribution model of ambient light in the imaging range, and pulse wave estimation unit 16 that estimates the person's pulse wave based on the pulse wave original signal wi(t) and the simulation signal mi(t). Therefore, pulse wave estimation device 1 can estimate the subject's pulse wave even in a scene in which the distribution of ambient light in the imaging range is non-uniform.

[0096] Moreover, the state estimation device 2 according to the above-described embodiment 1 is configured to estimate the state of a person (subject) based on the pulse wave of the person (subject) estimated by the pulse wave estimation device 1 having the above-described configuration. Therefore, the state estimation device 2 can estimate the state of the subject based on the pulse wave estimated with high accuracy even in a scene in which the distribution of ambient light in the imaging range is non-uniform. As a result, the state estimation device 2 can estimate the state of the subject with high accuracy.

[0097] Any of the components of the embodiments may be modified or omitted.

[0098] Various aspects of the present disclosure are summarized below as appendices.

[0099] (Appendix 1) an image acquisition unit for acquiring an image of a person; a skin area detection unit that detects a skin area of ​​the person from the captured image; a measurement area setting unit that sets a measurement area for extracting a pulse wave source signal indicating a luminance change in an area corresponding to the skin area on the captured image; a pulse wave source signal extracting unit that extracts the pulse wave source signal based on the luminance change in the measurement region on the captured image; a simulation signal generating unit that generates a simulation signal simulating the luminance change of the measurement area caused by the position change of the measurement area under the ambient light, based on a position change of the measurement area and a distribution model of the ambient light in an imaging range; a pulse wave estimation unit that estimates the pulse wave of the person based on the pulse wave original signal and the simulation signal; A pulse wave estimation device comprising: (Appendix 2) The pulse wave estimation unit estimates the pulse wave based on a signal obtained by removing the simulation signal from the pulse wave original signal. 2. A pulse wave estimation device according to claim 1. (Appendix 3) The distribution model of the ambient light is A model that outputs a value simulating the luminance value of a coordinate on the captured image based on the spatial luminance distribution of the range illuminated by the environmental light. 3. The pulse wave estimation device according to claim 1 or 2. (Appendix 4) The ambient light is illumination light emitted by an illumination unit included in an imaging device that captured the captured image. 4. The pulse wave estimation device according to claim 1, (Appendix 5) The simulation signal generating unit generates the simulation signal based on a plurality of different distribution models of the ambient light according to a situation in which the ambient light is irradiated and a position change of the measurement area. 5. The pulse wave estimation device according to claim 1, (Appendix 6) The simulation signal generating unit generates the simulation signal based on a signal obtained by normalizing a scale of a signal indicating the luminance change in the measurement area caused by the left-right movement of the person and a scale of a signal indicating the luminance change in the measurement area caused by the forward-backward movement of the person. 5. The pulse wave estimation device according to claim 1, (Appendix 7) A state estimation unit that estimates a state of the person based on the pulse wave of the person estimated by the pulse wave estimation device according to any one of Supplementary Note 1 to Supplementary Note 6. A state estimation device comprising: (Appendix 8) The state estimation unit estimates a level of alertness of the person as the state of the person. 8. The state estimation device according to claim 7. (Appendix 9) The state estimation unit calculates a reference pulse rate based on the pulse wave of the person estimated by the pulse wave estimation device when the person is assumed to be awake, and estimates the wakefulness of the person by comparing the pulse wave estimated by the pulse wave estimation device with the reference pulse rate. 9. The state estimation device according to claim 8. (Appendix 10) A step in which a captured image acquisition unit acquires a captured image of a person; a skin area detection unit detecting a skin area of ​​the person from the captured image; a measurement area setting unit setting a measurement area for extracting a pulse wave source signal indicating a luminance change in a region corresponding to the skin region on the captured image; a pulse wave source signal extracting unit extracting the pulse wave source signal based on the luminance change in the measurement region on the captured image; a simulation signal generating unit generating a simulation signal simulating the luminance change of the measurement area caused by the position change of the measurement area under the ambient light, based on the position change of the measurement area and a distribution model of the ambient light in an imaging range; a pulse wave estimating unit estimating a pulse wave of the person based on the pulse wave original signal and the simulation signal; A pulse wave estimation method comprising: [Industrial Applicability]

[0100] The pulse wave estimation device of the present disclosure estimates the subject's pulse wave by taking into account that in a scene where the distribution of ambient light in the imaging range is uneven, the pulse wave estimated from minute changes in brightness of the subject's skin surface in the captured image is affected by ambient light, and therefore can estimate the subject's pulse wave with higher accuracy than when this effect is not taken into account. [Explanation of symbols]

[0101] 1 Pulse wave estimation device, 11 Image acquisition unit, 12 Skin area detection unit, 13 Measurement area setting unit, 14 Pulse wave original signal extraction unit, 15 Simulation signal generation unit, 16 Pulse wave estimation unit, 17 Output unit, 2 State estimation device, 21 Pulse wave information acquisition unit, 22 State estimation unit, 23 Output unit, 3 Output device, 401 Imaging unit, 402 Illumination unit, 101 Processing circuit, 102 Input interface device, 103 Output interface device, 104 Processor, 105 Memory.

Claims

1. an image acquisition unit for acquiring an image of a person; a skin area detection unit that detects a skin area of ​​the person from the captured image; a measurement area setting unit that sets a measurement area for extracting a pulse wave source signal indicating a luminance change in an area corresponding to the skin area on the captured image; a pulse wave source signal extracting unit that extracts the pulse wave source signal based on the luminance change in the measurement region on the captured image; a simulation signal generating unit that generates a simulation signal simulating the luminance change of the measurement area caused by the position change of the measurement area under the ambient light, based on a position change of the measurement area and a distribution model of the ambient light in an imaging range; a pulse wave estimation unit that estimates a pulse wave of the person based on the pulse wave original signal and the simulation signal, The simulation signal generating unit generates the simulation signal based on a plurality of different distribution models of the ambient light according to a situation in which the ambient light is irradiated and a position change of the measurement area. A pulse wave estimation device comprising:

2. an image acquisition unit for acquiring an image of a person; a skin area detection unit that detects a skin area of ​​the person from the captured image; a measurement area setting unit that sets a measurement area for extracting a pulse wave source signal indicating a luminance change in an area corresponding to the skin area on the captured image; a pulse wave source signal extracting unit that extracts the pulse wave source signal based on the luminance change in the measurement region on the captured image; a simulation signal generating unit that generates a simulation signal simulating the luminance change of the measurement area caused by the position change of the measurement area under the ambient light, based on a position change of the measurement area and a distribution model of the ambient light in an imaging range; a pulse wave estimation unit that estimates a pulse wave of the person based on the pulse wave original signal and the simulation signal, The simulation signal generating unit generates the simulation signal based on a signal obtained by normalizing a scale of a signal indicating the luminance change in the measurement area caused by the left-right movement of the person and a scale of a signal indicating the luminance change in the measurement area caused by the forward-backward movement of the person. A pulse wave estimation device comprising:

3. The pulse wave estimation unit estimates the pulse wave based on a signal obtained by removing the simulation signal from the pulse wave original signal.

3. The pulse wave estimating device according to claim 1 or 2.

4. The distribution model of the ambient light is A model that outputs a value simulating the luminance value of a coordinate on the captured image based on the spatial luminance distribution of the range illuminated by the environmental light.

3. The pulse wave estimating device according to claim 1 or 2.

5. The ambient light is illumination light emitted by an illumination unit included in an imaging device that captured the captured image.

3. The pulse wave estimating device according to claim 1 or 2.

6. a state estimation unit for estimating a state of the person based on the pulse wave of the person estimated by the pulse wave estimation device according to claim 1 or 2; A state estimation device comprising:

7. The state estimation unit estimates a level of alertness of the person as the state of the person.

7. The state estimation device according to claim 6.

8. The state estimation unit calculates a reference pulse rate based on the pulse wave of the person estimated by the pulse wave estimation device when the person is assumed to be awake, and estimates the wakefulness of the person by comparing the pulse wave estimated by the pulse wave estimation device with the reference pulse rate.

8. The state estimation device according to claim 7.

9. A step in which a captured image acquisition unit acquires a captured image of a person; a skin area detection unit detecting a skin area of ​​the person from the captured image; a measurement area setting unit setting a measurement area for extracting a pulse wave source signal indicating a luminance change in a region corresponding to the skin region on the captured image; a pulse wave source signal extracting unit extracting the pulse wave source signal based on the luminance change in the measurement region on the captured image; a simulation signal generating unit generating a simulation signal simulating the luminance change of the measurement area caused by the position change of the measurement area under the ambient light, based on the position change of the measurement area and a distribution model of the ambient light in an imaging range; a pulse wave estimating unit estimating a pulse wave of the person based on the pulse wave original signal and the simulation signal; The simulation signal generating unit generates the simulation signal based on a plurality of different distribution models of the ambient light according to a situation in which the ambient light is irradiated and a position change of the measurement area. A pulse wave estimating method comprising:

10. A step of an image acquisition unit acquiring an image of a person; a skin area detection unit detecting a skin area of ​​the person from the captured image; a measurement area setting unit setting a measurement area for extracting a pulse wave source signal indicating a luminance change in a region corresponding to the skin region on the captured image; a pulse wave source signal extracting unit extracting the pulse wave source signal based on the luminance change in the measurement region on the captured image; a simulation signal generating unit generating a simulation signal simulating the luminance change of the measurement area caused by the position change of the measurement area under the ambient light, based on the position change of the measurement area and a distribution model of the ambient light in an imaging range; a pulse wave estimating unit estimating a pulse wave of the person based on the pulse wave original signal and the simulation signal; The simulation signal generating unit generates the simulation signal based on a signal obtained by normalizing a scale of a signal indicating the luminance change in the measurement area caused by the left-right movement of the person and a scale of a signal indicating the luminance change in the measurement area caused by the forward-backward movement of the person. A pulse wave estimating method comprising:

Images