Pulse wave estimation device and pulse wave estimation method

Abstract

A pulse wave estimation device will be provided that can suppress a deterioration in the accuracy of the pulse wave estimation. A pulse wave estimation device 1 comprises: an image acquisition unit 11 for acquiring a captured image of a person; a skin area detection unit 12 for detecting a skin area of ​​the person in the captured image; a measurement area setting unit 13 for setting, as a plurality of measurement areas for extracting luminance signals representing luminance changes, areas formed by subdividing the skin area in the captured image in a predetermined manner; a luminance signal extraction unit 14 for extracting the luminance signals based on the luminance changes in the measurement areas in the captured image;and a pulse wave estimation unit 16 for estimating a human pulse wave based on the extracted luminance signals, wherein the measurement range setting unit 13 performs a measurement range reset when a ratio of measurement ranges in which the number of pixels is equal to or greater than a preset reference value among the set plurality of measurement ranges is lower than a threshold, in such a way that the ratio of measurement ranges is equal to or greater than the threshold.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to a pulse wave estimation apparatus and a pulse wave estimation method. STATE OF THE ART

[0002] As a technique for estimating pulse waves of a subject in a non-contact manner without applying stress to the subject, there is a technique in which images of the subject's face are captured with a camera, and the pulse waves are estimated from a minute change in luminance on the subject's facial surface (e.g., Non-Patent Literature 1). For example, in this technique, a plurality of measurement areas are set on the images of the subject's face, and a frequency power spectrum of a luminance signal obtained from each set measurement area is calculated. Then, a pulse wave of the subject is estimated from a peak frequency of the frequency power spectrum calculated for each measurement area. In addition, the estimated pulse waves are synthesized, and a pulse rate is also estimated from a peak value of a frequency power spectrum of a pulse wave obtained by the synthesis. CITATION LISTPATENT LITERATURE

[0003] Non-Patent Literature 1: Mayank Kumar, et al., “DistancePPG: Robust non-contact vital signs monitoring using a camera,” Biomedical optics express, 6 (5), 1565-1588, 2015 SUMMARY OF THE INVENTION TECHNICAL PROBLEM

[0004] However, the above-described technique has a problem that deterioration of the image resolution of the acquired images of the subject due to a change in the arrangement of a camera or the like leads to deterioration of the resolving power and deterioration of the accuracy of pulse wave estimation.

[0005] The present disclosure has been made to solve the problem described above, and its object is to obtain a pulse wave estimation apparatus that can suppress deterioration of the accuracy of pulse wave estimation. SOLUTION TO THE PROBLEM

[0006] A pulse wave estimation device according to the present disclosure comprises: a captured image acquisition unit for acquiring a captured image of a human; a skin area detection unit for detecting a skin area of ​​the human in the captured image; a measurement area setting unit for setting, as a plurality of measurement areas for extracting luminance signals representing luminance changes, areas formed by dividing the skin area on the captured image in a predetermined manner; a luminance signal extraction unit for extracting the luminance signals based on the luminance changes in the measurement areas on the captured image; and a pulse wave estimation unit for estimating a human pulse wave based on the extracted luminance signals, wherein the measurement area setting unit performs a measurement area reset when a ratio of measurement areas in which a number of pixels is equal to or higher than a preset reference value among the set plurality of measurement areas is lower than a threshold, in such a manner that the ratio of measurement areas is equal to or higher than the threshold. ADVANTAGEOUS EFFECTS OF THE INVENTION

[0007] Since the present disclosure adopts the configuration described above, it is possible to suppress deterioration of the accuracy of pulse wave estimation. BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 is a figure illustrating a configuration example of a pulse wave estimation apparatus according to a first embodiment. Fig. 2A, Fig. 2B and Fig. 2C are figures for explaining an example of a measurement range setting process performed by a measurement range setting unit in the first embodiment. Fig. 3A and Fig. 3B are figures for explaining examples of the measurement range setting performed by the measurement range setting unit in the first embodiment. Fig. 4 is a flowchart showing an example of operations performed by the pulse wave estimation apparatus according to the first embodiment. Fig. 5 is a flowchart for explaining details of a process performed by the measurement range setting unit in the first embodiment. Fig. 6 is a figure for explaining an example of the measurement range resetting performed by the measurement range setting unit in the first embodiment. Fig. 7A, Fig. 7B, Fig. 7C and Fig. 7D are figures for explaining an example of the measurement range resetting performed by the measurement range setting unit in the first embodiment. Fig. 8A and Fig. 8B are figures showing an example of the hardware configuration of the pulse wave estimation apparatus according to the first embodiment. DESCRIPTION OF THE EMBODIMENTS

[0008] Below, an embodiment is explained in detail using the figures. First embodiment.

[0009] Fig. 1 is a figure showing a configuration example of a pulse wave estimation device 1 according to a first embodiment. The pulse wave estimation device 1 estimates the pulse waves of a human based on the acquired images of the human. In the following explanation, the human whose pulse waves are to be estimated by the pulse wave estimation device 1 is also referred to as the "subject."

[0010] The pulse wave estimation device 1 obtains captured images including a series of frames Im(k) obtained by capturing images of an area where a skin region, which is an area including at least the skin of the subject, is located at a preset pulse rate Fr. Here, k represents a frame number assigned to each frame. For example, a frame obtained at a time adjacent to frame Im(k) is a frame Im(k+1). Then, the pulse wave estimation device 1 estimates the pulse waves of the subject for each frame Im(k) and outputs a pulse wave estimation result P(t) that is information representing the estimated pulse wave (hereinafter referred to as "pulse wave information"). For example, the pulse wave information may be time-series data of the pulse waves of the subject estimated by a pulse wave estimation unit 16 mentioned later, or it may be a pulse rate of the subject.

[0011] Note that the number of human subjects captured in the captured images may be one or more than one. For convenience of explanation, the following explanation assumes that the number of subjects captured in the captured images is one. In addition, the following explanation assumes that the pulse wave estimation device 1 is mounted on a vehicle as an example (an illustration of the vehicle is omitted), and that the subject is the driver of the vehicle. That is, the pulse wave estimation device 1 estimates the pulse waves of the driver of the vehicle.

[0012] For example, as in Fig. 1, the pulse wave estimation device 1 includes a captured image acquisition unit 11, a skin area detection unit 12, a measurement area setting unit 13, a luminance signal extraction unit 14, a luminance signal segment generation unit 15, and a pulse wave estimation unit 16.

[0013] The captured image acquisition unit 11 receives a captured image of the subject driver. Specifically, the captured image acquisition unit 11 acquires a captured image of the driver of the vehicle captured with an image acquisition device mounted on the vehicle (an illustration of the image acquisition device is omitted). Note that the image acquisition device is installed so that images of an area where a skin region of the driver is located can be captured. The captured image acquisition unit 11 outputs the captured image to the skin region detection unit 12.

[0014] The skin area detection unit 12 detects a skin area of ​​the subject driver in a frame Im(k) included in the image captured by the captured image acquisition unit 11. Then, the skin area detection unit 12 outputs information representing the detected skin area as skin area information S(k) to the measurement area setting unit 13, along with the frame Im(k) on which the skin area information S(k) was detected.

[0015] Note that in the following explanation, the skin area is assumed to be information representing a rectangular area that represents the position and size of the driver's face on the frame Im(k). Note that the driver's face detection can be implemented using a known means. For example, the skin area detection unit 12 can extract a rectangular area surrounding the driver's face using a cascade face detector that utilizes hair-like features.

[0016] It should be noted that the skin area may be an area corresponding to a part other than the driver's face. For example, the skin area may be an area belonging to a part of the face, such as an eye, an eyebrow, the driver's nose, mouth, forehead, cheek, or chin. Additionally, the skin area may be an area corresponding to a body part other than the face, such as the driver's head, a shoulder, a hand, the neck, or a leg. Additionally, the skin area may not be just one area, but may include multiple areas.

[0017] The measurement area setting unit 13 receives the frame Im(k) and the skin area information S(k) output from the skin area detection unit 12. In addition, the measurement area setting unit 13 receives vehicle location information from a control device (not shown) mounted on the vehicle. The vehicle location information includes at least information representing the positional relationship between the image pickup device installed in the vehicle and the driver. For example, the vehicle location information includes: information representing the position in the vehicle where the image pickup device is installed; information representing the orientation of the image pickup device; information representing the position of the seat on which the driver sits relative to the position in the vehicle where the image pickup device is installed; and the like.

[0018] Based on the acquired frame Im(k), the skin area information S(k), and the above-described vehicle arrangement information, the measurement area setting unit 13 divides the skin area on the frame Im(k) represented by the skin area information S(k) in a predetermined manner, and sets the areas obtained by the division as a plurality of measurement areas for extracting luminance signals representing luminance changes.

[0019] After setting the plurality of measurement areas, the measurement area setting unit 13 generates measurement area information R(k) representing the set plurality of measurement areas. The measurement area information R(k) includes information representing the position and size of Rn (positive integer) measurement areas on the captured image. The measurement areas are assumed to be measurement areas fi(k) (i = 1, 2, ..., Rn). In the first embodiment, it is assumed that each measurement area fi(k) is a quadrilateral, and the position and size of the measurement area fi(k) are represented by the coordinate values ​​of the four vertices of the quadrilateral on the captured image.

[0020] Fig. 2A, Fig. 2B and Fig. 2C are figures for explaining an example of a measurement range setting process performed by the measurement range setting unit 13 in the pulse wave estimation device 1 according to the first embodiment. An example of the process performed by the measurement range setting unit 13 for setting the plurality of measurement ranges is described using Fig. 2 explained.

[0021] First, the measuring range setting unit 13 detects, as shown in Fig. 2A and Fig. 2B, Ln (positive integer) landmarks of facial organs, such as the eye tails, the inner corners of the eyes, the nose or the mouth in a skin area sr, which is represented by the skin area information S(k). In Fig. 2A and Fig. 2B, the landmarks are represented by circles. The measurement area setting unit 13 sets a vector that stores the coordinate values ​​of a detected landmark as L(k). Note that the measurement area setting unit 13 can detect facial organs using a known means, such as using a model called a "Constrained Local Model" (CLM).

[0022] Next, the measurement area setting unit 13 sets the vertex coordinates of the quadrilaterals of the measurement areas fi(k) based on the detected landmarks. For example, the measurement area setting unit 13 sets the vertex coordinates of quadrilaterals as shown in Fig. 2C, and sets Rn measuring ranges fi(k).

[0023] The setting of measurement areas fi(k) in a section corresponding to a cheek in the skin area sr by the measurement area setting unit 13 will be explained as an example. The measurement area setting unit 13 selects a landmark LA1 on the facial contour and a landmark LA2 on the nose. The measurement area setting unit 13 may first select the landmark LA2 on the nose and select the landmark LA1 on the facial contour closest to the landmark LA2 on the nose. Then, the measurement area setting unit 13 sets the auxiliary landmarks a1, a2, and a3 so that the auxiliary landmarks a1, a2, and a3 evenly divide the line segment between the landmark LA1 and the landmark LA2 into four areas.

[0024] Similarly, the measurement area setting unit 13 selects a landmark LB1 on the contour of the face and a landmark LB2 on the nose. Additionally, the measurement area setting unit 13 sets the auxiliary landmarks b1, b2, and b3 so that the auxiliary landmarks b1, b2, and b3 evenly divide the line segment between the landmark LB1 and the landmark LB2 into four parts. Note that, for example, the landmarks LB1 and LB2 can be selected from landmarks on the facial contour and the nose, respectively, that are adjacent to the landmarks LA1 and LA2.

[0025] The measurement area setting unit 13 sets a quadrangular area surrounded by the auxiliary landmarks a1, b1, b2, and a2 as a measurement area R1. Each of the auxiliary landmarks a1, b1, b2, and a2 is located at the coordinates of a vertex of the measurement area R1. Similarly, the measurement area setting unit 13 sets a measurement area R2 surrounded by the auxiliary landmarks a2, b2, b3, and a3 and sets the vertex coordinates of the measurement area R2.

[0026] It should be noted that the measurement area setting unit 13 sets the measurement areas fi(k) and the vertex coordinates of the measurement areas fi(k) similarly for the other portion of the cheeks and the portion corresponding to the chin in the skin area sr, while in the example explained here, measurement areas fi(k) are set at a portion corresponding to the cheeks. It should be noted that the measurement area setting unit 13 may set measurement areas fi(k) for the portion corresponding to the driver's forehead or for the portion corresponding to the driver's nose tip in the skin area sr, although this is not possible in Fig. 2C is not shown.

[0027] For example, as in Fig. As shown in Figure 3A, the measurement area setting unit 13 divides the areas corresponding to the left and right cheeks of a driver DR in the skin area represented by the skin area information S(k) in a predetermined manner and sets the measurement areas fi(k) (i = 1, 2, 3, ..., n). For example, in this case, the measurement areas fi(k) are set so that the measurement areas fi(k) form a U-shape in the skin area represented by the skin area information S(k).

[0028] It should be noted that the measurement area setting unit 13 can decide where to set the measurement areas depending on the orientation of the driver's face. For example, if the driver's face is directed slightly to the right relative to the front, for example, as in Fig. 3B, the measurement area setting unit 13 may set measurement areas fi(k) only in the area corresponding to the driver's left cheek (the area closer to the image pickup device), namely, in the skin area. In this case, the measurement areas are set so that the measurement areas form a J-shape, for example, in the skin area represented by the skin area information S(k). Note that the measurement area setting unit 13 may determine the orientation of the driver's face by a known technique. In the following explanation, measurement areas that the measurement area setting unit 13 initially sets for a skin area are also referred to as "initial measurement areas."

[0029] In addition, the measurement area setting unit 13 can determine the maximum possible number of measurement areas that can be set in advance from the above-described obtained arrangement information about the vehicle and a predetermined angular range of the orientation of the driver's face.

[0030] For example, the measurement area setting unit 13 may decide in advance that the maximum number of measurement areas to be set is 28, for example, in a case where the face of the driver DR is slightly directed to the right relative to the front, as in Fig. 3B, and the image pickup device has captured an image of the driver DR diagonally from the front of the driver DR. Then, the measurement area setting unit 13 may set 28 measurement areas as initial measurement areas for the skin area represented by the skin area information S(k) obtained when the image pickup device actually captured an image of the driver DR diagonally from the front.

[0031] Similarly, the measurement area setting unit 13 may decide in advance that the maximum number of measurement areas to be set is 40, for example, in a case where the driver's face DR faces the front, as in Fig. 3A, and the image pickup device has captured an image of the driver DR from the front of the driver DR. Then, the measurement area setting unit 13 may set 40 measurement areas as initial measurement areas for the skin area represented by the skin area information S(k) obtained when the image pickup device actually captured an image of the driver DR from the front.

[0032] After setting the initial measurement areas as described above, the measurement area setting unit 13 calculates the number of pixels included in each of the set initial measurement areas. Then, the measurement area setting unit 13 calculates the ratio of initial measurement areas with the number of pixels equal to or higher than a preset reference value in the set plurality of initial measurement areas, and in a case where the calculated ratio is lower than the threshold, performs a measurement area reset to make the ratio equal to or higher than the threshold. Note that details of the measurement area reset will be mentioned later.

[0033] The measurement range setting unit 13 outputs the measurement range information R(k) representing the set or reset plurality of measurement ranges fi(k) to the luminance signal extraction unit 14.

[0034] The luminance signal extraction unit 14 extracts, based on the measurement range information R(k) output from the measurement range setting unit 13, a luminance signal representing a luminance change of each measurement range fi(k) changed by the measurement range information R(k).

[0035] For example, for each measurement area fi(k) set by the measurement area setting unit 13 on the current frame, the luminance signal extraction unit 14 calculates the average value of the luminance values ​​of the pixels included in the measurement area. Then, the luminance signal extraction unit 14 sets, as luminance signal information Gi(k), information representing the calculated average value of the luminance values ​​of each measurement area. Similarly, for each measurement area fi(k-1) set by the measurement area setting unit 13 on the previous frame, which is one frame before the current frame, the luminance signal extraction unit 14 calculates the average value of the luminance values ​​of the pixels included in the measurement area. Then, the luminance signal extraction unit 14 sets, as luminance signal information Gi(k-1), information representing the calculated average value of the luminance values ​​of each measurement area.

[0036] Then, the luminance signal extraction unit 14 calculates the difference between the luminance signal information Gi(k) and the luminance signal information Gi(k-1). Specifically, the luminance signal extraction unit 14 calculates the difference between the average values ​​of the luminance values ​​of the corresponding measurement ranges in the luminance signal information Gi(k) and the luminance signal information Gi(k-1). At this time, in a case where either the luminance signal information Gi(k) or the luminance signal information Gi(k-1) has a measurement range that is not included in the other, the luminance signal extraction unit 14 does not calculate the difference thereof.

[0037] Then, the luminance signal extraction unit 14 outputs information representing the calculated differences between the luminance signal information Gi(k) and the luminance signal information Gi(k-1) to the luminance signal segment generation unit 15 as luminance signal segment information W(t). At this time, the luminance signal extraction unit 14 includes, in the luminance signal segment information W(t), information representing the orientation of the subject's face on the current frame (k) and information representing the orientation of the subject's face on the previous frame (k-1).

[0038] Here, t represents a number corresponding to the time difference between the current frame (k) and the previous frame (k-1). For example, if the luminance signal segment information representing the difference between the luminance signal information Gi(k) and the luminance signal information Gi(k-1) is assumed to be W(t), the luminance signal segment information representing the difference between the luminance signal information Gi(k+1) and the luminance signal information Gi(k) is assumed to be W(t+1).

[0039] Note that in the case exemplified in the above explanation, the luminance signal extraction unit 14 uses information representing the calculated average value of the luminance values ​​of each measurement area as the luminance signal information Gi(k). However, this is not the only example; for example, the luminance signal extraction unit 14 may use information representing the calculated variance of the luminance values ​​of each measurement area as the luminance signal information Gi(k).

[0040] The luminance signal segment generation unit 15 continuously collects (stores) the luminance signal segment information W(t) output from the luminance signal extraction unit 14 in a storage unit such as a memory until an amount of information corresponding to a predetermined period of time required for estimating a pulse wave is accumulated. At this time, the luminance signal segment generation unit 15 generates a time-series luminance signal (luminance signal segment) representing the appearance of time-series changes in luminance values ​​of each measurement range.

[0041] Then, the luminance signal segment generation unit 15 selects, from the generated time-series luminance signals of the measurement areas, a time-series luminance signal to be used to estimate a pulse wave in the pulse wave estimation unit 16 mentioned later. For example, the luminance signal segment generation unit 15 selects, from the generated time-series luminance signals of the measurement areas, a time-series luminance signal obtained when the deviation between the orientation of the face on the current frame (k) and the previous frame (k-1) of the driver DR included in the luminance signal segment information W(t) and a preset reference face orientation is within a predetermined range (e.g., the range from "-5 degrees" to "+5 degrees") as a time-series luminance signal to be used to estimate a pulse wave of the subject.

[0042] One reason why the luminance signal segment generation unit 15 selects a time-series luminance signal in this way is as follows. That is, in principle, a pulse wave (pulse) is detected by detecting a luminance change in the same measurement range during pulse wave estimation, but a luminance change may also be caused by factors other than blood flow. In particular, if the orientation of the subject driver DR's face changes, the incident direction of light relative to the driver DR's face changes, and accordingly, the luminance changes significantly. Accordingly, in pulse wave estimation, it is desirable that luminance values ​​(luminance signals) obtained when the deviation between the face orientation and the reference face orientation is significant are not used to estimate a pulse wave.In view of this, the luminance signal segment generation unit 15 selects a time-series luminance signal used to estimate a pulse wave of the subject using the above-described orientation of the driver DR's face as a condition.

[0043] As segment information X(t), the luminance signal segment generation unit 15 outputs information representing the selected time series luminance signal to the pulse wave estimation unit 16.

[0044] The pulse wave estimation unit 16 estimates a pulse wave of the subject based on the segment information X(t) output from the luminance signal segment generation unit 15. For example, the pulse wave estimation unit 16 adds the differences between the average values ​​of the luminance values ​​of the measurement areas fi(k) included in the segment information X(t) output from the luminance signal segment generation unit 15 and calculates a piece of pulse wave signal information. Then, the pulse wave estimation unit 16 performs Fourier transformation on the calculated pulse wave signal information and estimates a pulse wave of the subject based on the peak frequency of a frequency power spectrum.

[0045] Note that, while the pulse wave estimation device 1 includes the luminance signal segment generation unit 15 in the example explained in the above explanation, the luminance signal segment generation unit 15 is not an essential element and may be omitted. If so, the pulse wave estimation unit 16 may collect (store) the luminance signal segment information W(t) output from the luminance signal extraction unit 14 in the storage unit such as a memory until an amount of information corresponding to a predetermined period of time necessary for estimating a pulse wave is collected, and estimate a pulse wave of the subject based on the collected information.

[0046] Next, an example of operations is explained that are performed by the Fig. 1 shown pulse wave estimation device 1. Fig. 4 is a flowchart representing an example of operations performed by the pulse wave estimation device 1.

[0047] First, the captured image acquisition unit 11 receives a captured image of the subject driver DR (step ST1). The captured image acquisition unit 11 outputs the captured image to the skin area detection unit 12.

[0048] Next, the skin area detection unit 12 detects a skin area in a frame Im(k) included in the image acquired by the captured image acquisition unit 11 in step ST1 (step ST2). The skin area detection unit 12 generates skin area information S(k) representing the detected skin area and outputs the generated skin area information S(k) to the measurement area setting unit 13 along with the frame Im(k).

[0049] Next, based on the frame Im(k), the skin area information S(k), and the vehicle arrangement information, the measurement area setting unit 13 divides the skin area represented by the skin area information S(k) on the frame Im(k) in a predetermined manner, and sets the areas obtained by the division as a plurality of measurement areas for extracting luminance signals representing luminance changes (step ST3).

[0050] Here, details of the process performed by the measurement range setting unit 13 will be described with reference to a Fig. The flowchart shown in Figure 5 explains this.

[0051] First, the measurement area setting unit 13 divides the skin area represented by the skin area information S(k) on the frame Im(k) in a predetermined manner and sets the areas obtained by the division as a plurality of measurement areas (initial measurement areas) for extracting luminance signals representing luminance changes (step ST31).

[0052] Next, the measurement area setting unit 13 calculates the number of pixels included in each of the set initial measurement areas. Then, the measurement area setting unit 13 calculates the ratio of initial measurement areas with the number of pixels equal to or higher than a preset reference value in the set plurality of initial measurement areas, and judges whether the calculated ratio is equal to or higher than a threshold (step ST32).

[0053] If, as a result of the evaluation, the calculated ratio is equal to or higher than the threshold (step ST32; YES), the measurement range setting unit 13 terminates the process. Thereafter, the process returns to step ST4, which is shown in Fig. 4. On the other hand, if the calculated ratio is not equal to or higher than the threshold (step ST32; NO), the measurement range setting unit 13 performs a measurement range reset to make the ratio equal to or higher than the threshold (step ST33).

[0054] Note that the above-described reference value and threshold can be appropriately set, for example, by an administrator of the pulse wave estimation device 1 based on the accuracy required for pulse wave estimation or the like. For example, the administrator of the pulse wave estimation device 1 increases the reference value and the above-described threshold to be set as the accuracy required for pulse wave estimation increases. For example, in a case where the above-described threshold is set to "100%", the measurement range setting unit 13 performs the measurement range reset if there is even one initial measurement range with a pixel count lower than the preset reference value in the set initial measurement ranges.In addition, for example, if the threshold described above is set to “50%”, the measurement area setting unit 13 does not perform the measurement area reset when the number of initial measurement areas with a pixel number equal to or greater than the preset reference value is equal to or greater than half of the number of set initial measurement areas.

[0055] For example, the range setting unit 13 performs the range resetting according to the following three methods. (Procedure 1).

[0056] For example, the measurement range setting unit 13 performs the measurement range reset by making the number of divisions of the skin area smaller than at the time of the initial measurement range setting so that the above-described ratio is equal to or higher than the threshold. For example, if the above-described ratio is in a case where 28 initial measurement ranges are set as in Fig. 3B is not equal to or higher than the threshold, the measurement range setting unit 13 performs the measurement range reset by reducing the number of divisions of the skin area to eight, as shown in Fig. 6. In addition, the measurement area setting unit 13 at this time makes a pixel number per measurement area according to the Fig. 6 shown settings greater than one pixel number per measuring area before the in Fig. 3B shown settings.

[0057] When the measurement area setting unit 13 reduces the number of skin areas, it can reduce the number of skin areas by selecting some adjacent initial measurement areas from the plurality of Fig. 3B. Alternatively, the measuring range setting unit 13 can reset measuring ranges by selecting the plurality of Fig. 3B and re-divides the skin area with a number of subdivisions that is smaller than that at the time of setting the initial measurement ranges. (Procedure 2)

[0058] For example, the measurement range setting unit 13 performs the measurement range reset by setting new measurement ranges each extending over a predetermined plurality of adjacent initial measurement ranges in the plurality of initial measurement ranges to make the above-described ratio equal to or higher than the threshold. For example, if the above-described ratio is set to 28 in a case where 28 initial measurement ranges are set as shown in Fig. 3B is not equal to or higher than the threshold as shown in Fig. 7A to Fig. As shown in Figure 7D, the measurement range setting unit 13 performs the measurement range reset by setting new measurement ranges, each of which extends over two or four adjacent initial measurement ranges among the 28 initial measurement ranges. In this example, thirteen measurement ranges from fi(1) to fi(13) are reset.

[0059] Additionally, at this time, the thirteen new measurement ranges are allowed to overlap each other. For example, the measurement range fi(9) overlaps the measurement ranges fi(1) and fi(3), and the measurement range fi(10) overlaps the measurement ranges fi(2) and fi(4).

[0060] In addition, in this case, the measuring area setting unit 13 also makes a number of pixels per measuring area according to the Fig. 7A to Fig. 7D shown settings are greater than one pixel number per measuring area before the Fig. 3B shown settings. (Procedure 3)

[0061] For example, the measurement range setting unit 13 may combine Method 1 and Method 2. For example, the measurement range setting unit 13 may perform the measurement range reset by reducing the number of divisions of the skin area and also setting new measurement ranges each extending over a predetermined plurality of adjacent measurement ranges in a plurality of measurement ranges obtained by reducing the number of divisions to make the above-described ratio equal to or higher than the threshold. For example, a new measurement range may be set extending over two adjacent measurement ranges in the eight in Fig. 6 shown measuring ranges.

[0062] The measuring range setting unit 13 performs the measuring range reset according to one of the above-mentioned methods 1 to 3. After that, the process returns to the one shown in Fig. 4 shown step ST4.

[0063] Next, advantageous effects of the pulse wave estimation device 1 according to the first embodiment will be explained. By performing the measurement range reset as described above, the pulse wave estimation device 1 can suppress deterioration in the accuracy of pulse wave estimation compared to conventional techniques.

[0064] For example, in conventional pulse wave estimation techniques, if the arrangement of an image pickup device for acquiring images of the subject driver DR is changed or the image pickup device itself is replaced, and accordingly, the image resolution of the acquired images of the subject driver DR deteriorates, the resolving power deteriorates, and the accuracy of pulse wave estimation deteriorates in some cases. In addition, in conventional pulse wave estimation techniques, in a case where the driver DR moves back the vehicle seat or changes the orientation of the face, and accordingly, the driver DR's face moves away from the image pickup device, the image resolution of the acquired images of the driver DR deteriorates, and a similar problem occurs in some cases.

[0065] In contrast, in a case where the ratio of measurement areas with the number of pixels equal to or higher than the preset reference value in a plurality of measurement areas set by the measurement area setting unit 13 is lower than the threshold, the pulse wave estimation device 1 dynamically performs measurement area resetting to make the ratio equal to or higher than the threshold. Thereby, the pulse wave estimation device 1 makes the ratio of measurement areas with the number of pixels equal to or higher than the reference value equal to or higher than the threshold after the resetting, suppressing deterioration of the resolver and suppressing deterioration of the pulse wave estimation accuracy.

[0066] In addition, the pulse wave estimation device 1 also achieves the following advantageous effect in a case where the measurement range setting unit 13 adopts the above-described method 2. For example, in a case where the measurement range setting unit 13 resets measurement ranges by reducing the number of divisions of the skin area as in the above-described method 1, if local external light undesirably enters some of the measurement ranges obtained after the reset, these measurement ranges cannot be used for the estimation of a pulse wave, and accordingly, there is a possibility that the number of measurement ranges that can be used for the estimation of a pulse wave decreases, and the robustness deteriorates.

[0067] In this regard, in Method 2 described above, the measurement area setting process unit 13 sets new measurement areas each extending over a predetermined number of adjacent initial measurement areas, while allowing the new measurement areas to overlap with each other. Accordingly, it is also possible to ensure that there is a sufficient number of measurement areas that can be used for estimating a pulse wave while ensuring that the measurement areas have a sufficient size (there are a sufficient number of pixels in each measurement area), and it is possible to suppress deterioration in robustness compared with Method 1.

[0068] In addition, in a case where the measurement area setting unit 13 applies the above-described method 3, the pulse wave estimation device 1 can make the sizes of the measurement areas after setting (the number of pixels in the measurement areas) larger than in a case where the method 1 or the method 2 is applied individually, and can also suppress deterioration of robustness, which is a problem with the method 1.

[0069] Next, an example of the hardware configuration of the pulse wave estimation device 1 according to the first embodiment will be described with reference to Fig. 8. The respective functions of the captured image acquisition unit 11, the skin area detection unit 12, the measurement area setting unit 13, the luminance signal extraction unit 14, the luminance signal segment generation unit 15, and the pulse wave estimation unit 16 in the pulse wave estimation device 1 are implemented by a processing circuit. The processing circuit may be dedicated hardware, as shown in Fig. 8A, or a central processing unit (CPU; also referred to as central processor, processing unit, computing device, microprocessor, microcomputer, processor or digital signal processor (DSP)) 52 to execute programs stored in a memory 53, as shown in Fig. 14B.

[0070] For example, in a case where the processing circuit is dedicated hardware, a processing circuit 51 is a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a combination thereof. Each of the functions of the captured image acquisition unit 11, the skin area detection unit 12, the measurement area setting unit 13, the luminance signal extraction unit 14, the luminance signal segment generation unit 15, and the pulse wave estimation unit 16 can be implemented by the processing circuit 51, or the functions of the respective units can be implemented jointly by the processing circuit 51.

[0071] If the processing circuit is the CPU 52, the functions of the captured image acquisition unit 11, the skin area detection unit 12, the measurement area setting unit 13, the luminance signal extraction unit 14, the luminance signal segment generation unit 15, and the pulse wave estimation unit 16 are implemented by software, firmware, or a combination of software and firmware. Software and firmware are written as programs and stored in the memory 53. The processing circuit implements the function of each unit by reading and executing the programs stored in the memory 53. That is, the pulse wave estimation device 1 includes a memory for storing the programs, the execution of which by the processing circuit leads to the execution of the individual steps described, for example, in Fig.4. In addition, these programs may also be referred to as programs that cause a computer to execute procedures and processes of the captured image acquisition unit 11, the skin area detection unit 12, the measurement area setting unit 13, the luminance signal extraction unit 14, the luminance signal segment generation unit 15, and the pulse wave estimation unit 16. For example, a non-volatile or volatile semiconductor memory such as a Random Access Memory (RAM), a Read Only Memory (ROM), a flash memory, an Erasable Programmable ROM (EPROM), or an Electrically EPROM (EEPROM), a magnetic disk, a flexible disk, an optical disk, a compact disc, a mini-disc, a Digital Versatile Disc (DVD), or the like may be used as the memory 53.

[0072] Note that some of the respective functions of the captured image acquisition unit 11, the skin area detection unit 12, the measurement area setting unit 13, the luminance signal extraction unit 14, the luminance signal segment generation unit 15, and the pulse wave estimation unit 16 can be implemented by dedicated hardware, while others can be implemented by software or firmware. For example, it is possible to implement the function of the captured image acquisition unit 11 using the processing circuit as dedicated hardware, and to implement the functions of the skin area detection unit 12, the measurement area setting unit 13, the luminance signal extraction unit 14, the luminance signal segment generation unit 15, and the pulse wave estimation unit 16 by having the processing circuit read and execute programs stored in the memory 53.

[0073] In this way, the processing circuit can implement the corresponding functions mentioned above through hardware, software, firmware or a combination thereof.

[0074] As mentioned above, the pulse wave estimation device 1 according to the first embodiment includes: the captured image acquisition unit 11 for acquiring a captured image of a human; the skin area detection unit 12 for detecting a skin area of ​​the human on the captured image; the measurement area setting unit 13 for setting, as a plurality of measurement areas for extracting luminance signals representing luminance changes, areas formed by dividing the skin area on the captured image in a predetermined manner; the luminance signal extraction unit 14 for extracting the luminance signals based on the luminance changes in the measurement areas on the captured image;and the pulse wave estimation unit 16 for estimating a human pulse wave based on the extracted luminance signal. The measurement range setting unit 13 performs a measurement range reset to make a ratio of measurement ranges with the number of pixels equal to or higher than a preset reference value in the set plurality of measurement ranges equal to or higher than a threshold in a case where the ratio is lower than the threshold. This makes it possible for the pulse wave estimation device 1 according to the first embodiment to suppress deterioration in the accuracy of the pulse wave estimation.

[0075] Additionally, the measurement area setting unit 13 performs measurement area resetting by reducing the number of divisions of the skin area on the acquired image if the above-described ratio is lower than the threshold. Thus, the pulse wave estimation device 1 according to the first embodiment can easily reset measurement areas.

[0076] In addition, the measurement range setting unit 13 performs the measurement range reset by setting a new measurement range extending over a predetermined plurality of adjacent measurement ranges in the set plurality of measurement ranges if the above-described ratio is lower than the threshold. This allows the pulse wave estimation device 1 according to the first embodiment to easily reset measurement ranges.

[0077] In addition, a plurality of the new measurement ranges described above can be set, and mutually overlapping measurement ranges can be set as the new measurement ranges. Thus, the pulse wave estimation device 1 according to the first embodiment can also ensure that there are a sufficient number of measurement ranges that can be used to estimate a pulse wave, while ensuring that the measurement ranges have sufficient sizes.

[0078] In addition, the measurement area setting unit 13 performs the measurement area reset by reducing the number of divisions of the skin area on the acquired image, and also sets a new measurement area extending over a predetermined number of adjacent measurement areas in a plurality of measurement areas obtained by reducing the number of divisions if the above-described ratio is lower than the threshold. Thus, the pulse wave estimation device 1 according to the first embodiment can increase the sizes of the measurement areas after the reset.

[0079] In addition, the measurement range adjustment unit 13 makes the number of pixels per measurement range after adjustment larger than the number of pixels per measurement range before adjustment. This allows the pulse wave estimation device 1 according to the first embodiment to suppress deterioration in resolver performance and suppress deterioration in pulse wave estimation precision.

[0080] Additionally, the pulse wave estimation device 1 includes the luminance signal segment generation unit 15 for selecting a luminance signal to be used for estimating the human pulse wave from the luminance signals extracted by the luminance signal extraction unit 14. The pulse wave estimation unit 16 estimates the human pulse wave based on the luminance signal selected by the luminance signal segment generation unit 15. Therefore, the pulse wave estimation device 1 according to the first embodiment can effectively suppress deterioration in the accuracy of the pulse wave estimation.

[0081] In addition, the luminance signal segment generation unit 15 selects, as the luminance signal to be used for estimating the human pulse wave, a luminance signal extracted when the deviation between the human face orientation and a preset reference face orientation is within a predetermined range from the luminance signals extracted by the luminance signal extraction unit 14. Thus, the pulse wave estimation device 1 according to the first embodiment can select a time-series luminance signal used for estimating a pulse wave using the human face orientation as a condition.

[0082] Although a preferred embodiment and the like have been explained in detail so far, the present disclosure and the like are not limitative, and various modifications and replacements of the present embodiment and the like are possible without departing from the scope described in the claims.

[0083] Several aspects of the present disclosure are described below collectively as notes. (Note 1)

[0084] Pulse wave estimation device comprising: a captured image acquiring unit for acquiring a captured image of a human; a skin area detection unit for detecting a skin area of ​​the human in the captured image; a measurement area setting unit for setting, as a plurality of measurement areas for extracting luminance signals representing luminance changes, areas formed by dividing the skin area on the captured image in a predetermined manner; a luminance signal extraction unit for extracting the luminance signals based on the luminance changes in the measurement areas on the captured image; and a pulse wave estimation unit for estimating a pulse wave of the human based on the extracted luminance signals, wherein the measurement area setting unit performs a measurement area reset when a ratio of measurement areas in which a number of pixels is equal to or larger than a preset reference value among the set plurality of measurement areas is lower than a threshold, in such a manner that the ratio of measurement areas is made equal to or higher than the threshold. (Note 2)

[0085] The pulse wave estimation device according to Note 1, wherein the measurement range setting unit performs the measurement range resetting by decreasing the number of divisions of the skin area on the acquired image in a case where the ratio is lower than the threshold. (Note 3)

[0086] The pulse wave estimation device according to Note 1, wherein the measurement range setting unit performs the measurement range resetting by setting a new measurement range extending over a predetermined number of adjacent measurement ranges in the set plurality of measurement ranges if the ratio is lower than the threshold. (Note 4)

[0087] A pulse wave estimating device according to Note 3, in which a plurality of the new measurement ranges can be set, and mutually overlapping measurement ranges can be set as the new measurement ranges. (Note 5)

[0088] The pulse wave estimation device according to note 1, wherein the measurement area setting unit performs the measurement area resetting by reducing the number of divisions of the skin area on the captured image, and also sets a new measurement area extending over a predetermined number of adjacent measurement areas in the plurality of measurement areas obtained by reducing the number of divisions if the ratio is lower than the threshold. (Note 6)

[0089] A pulse wave estimation device according to any one of notes 1 to 5, wherein the measurement area setting unit makes the number of pixels per measurement area after setting larger than the number of pixels per measurement area before setting. (Note 7)

[0090] A pulse wave estimation device according to any one of notes 1 to 6, comprising a luminance signal segment generation unit for selecting a luminance signal to be used for estimating the pulse wave of the human from the luminance signals extracted by the luminance signal extraction unit, wherein the pulse wave estimation unit estimates the human pulse wave based on the luminance signal selected by the luminance signal segment generation unit. (Note 8)

[0091] The pulse wave estimation device according to Note 7, wherein the luminance signal segment generation unit selects, from the luminance signals extracted by the luminance signal extraction unit, a luminance signal extracted when a deviation between an orientation of a face of the human and a preset reference face orientation is within a predetermined range as the luminance signal to be used for estimating the pulse wave of the human, (Note 9)

[0092] A pulse wave estimation method performed by a pulse wave estimation device, the pulse wave estimation method comprising: Capturing, by a capture image capture unit, a captured image of a human; Detecting, by a skin area detection unit, a skin area of ​​the human in the captured image; Setting, by a measurement area setting unit, areas formed by dividing the skin area on the captured image in a predetermined manner as a plurality of measurement areas for extracting luminance signals representing luminance changes; Extracting, by a luminance signal extraction unit, the luminance signals based on the luminance changes in the measurement areas on the captured image; Estimating, by a pulse wave estimation unit, a pulse wave of the human based on the extracted luminance signals; and Performing, by the measurement area setting unit, a measurement area reset when a ratio of measurement areas in which a number of pixels is equal to or larger than a preset reference value among the set plurality of measurement areas is lower than a threshold, in such a manner that the ratio of measurement areas is made equal to or higher than the threshold. INDUSTRIAL APPLICABILITY

[0093] The present disclosure makes it possible to suppress deterioration of the accuracy of pulse wave estimation and is suitable for use for a pulse wave estimation device. LIST OF REFERENCE SYMBOLS

[0094] 1: Pulse wave estimation device, 11: Captured image acquisition unit, 12: Skin area detection unit, 13: Measurement area setting unit, 14: Luminance signal extraction unit, 15: Luminance signal segment generation unit, 16: Pulse wave estimation unit, 51: Processing circuit, 52: CPU, 53: Memory, a1 to a3: Auxiliary landmark, b1 to b3: Auxiliary landmark, DR: Driver, fi: Measurement area, LA1 to LA2: Landmark, LB1 to LB2: Landmark, sr: Skin area QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited non-patent literature

[0000] Mayank Kumar, et al., „DistancePPG: Robust non-contact vital signs monitoring using a camera,“ Biomedical optics express, 6 (5), 1565-1588, 2015

[0003]

Claims

[1] Pulse wave estimation device (1) comprising: a captured image acquisition unit (11) for acquiring a captured image of a human; a skin area detection unit (12) for detecting a skin area of the human in the captured image; a measurement area setting unit (13) for setting, as a plurality of measurement areas for extracting luminance signals representing luminance changes, areas formed by dividing the skin area on the captured image in a predetermined manner; a luminance signal extraction unit (14) for extracting the luminance signals based on the luminance changes in the measurement areas on the captured image; and a pulse wave estimation unit (16) for estimating a pulse wave of the human based on the extracted luminance signals, wherein the measurement area setting unit (13) performs a measurement area reset when a ratio of measurement areas in which the number of pixels is equal to or greater than a preset reference value among the set plurality of measurement areas is lower than a threshold, in such a manner that the ratio of measurement areas is equal to or higher than the threshold. [2] The pulse wave estimation device according to claim 1, wherein the measurement area setting unit (13) performs the measurement area resetting by decreasing the number of divisions of the skin area on the acquired image if the ratio is lower than the threshold. [3] The pulse wave estimation device according to claim 1, wherein the measurement range setting unit (13) performs the measurement range resetting by setting at least one new measurement range extending over a predetermined number of adjacent measurement ranges in the set plurality of measurement ranges in a case where the ratio is lower than the threshold. [4] The pulse wave estimation apparatus according to claim 3, wherein the at least one new measurement range comprises a plurality of new measurement ranges that can be overlapped with each other. [5] The pulse wave estimation device according to claim 1, wherein the measurement area setting unit (13) performs the measurement area resetting by reducing the number of divisions of the skin area on the acquired image, and also sets at least one new measurement area extending over a predetermined number of adjacent measurement areas in the plurality of measurement areas obtained by reducing the number of divisions if the ratio is lower than the threshold. [6] The pulse wave estimation device according to any one of claims 1 to 5, wherein the measurement area setting unit (13) makes the number of pixels per measurement area after the reset larger than the number of pixels per measurement area before the reset. [7] Pulse wave estimation device according to claim 1, comprising a luminance signal segment generation unit (15) for selecting a luminance signal to be used for estimating the human pulse wave from the luminance signals extracted by the luminance signal extraction unit (14), wherein the pulse wave estimation unit (16) estimates the pulse wave of the human based on the luminance signal selected by the luminance signal segment generation unit. [8] The pulse wave estimation device according to claim 7, wherein the luminance signal segment generation unit selects, as the luminance signal to be used for estimating the pulse wave of the human, a luminance signal extracted when a deviation between an orientation of a face of the human and a preset reference face orientation is within a predetermined range from the luminance signals extracted by the luminance signal extraction unit (14). [9] A pulse wave estimation method performed by a pulse wave estimation device, the pulse wave estimation method comprising: Capturing, by a captured image capturing unit (11), a captured image of a human; Detecting, by a skin area detection unit (12), a skin area of the human in the captured image; Setting, by a measurement area setting unit (13), areas formed by dividing the skin area on the captured image in a predetermined manner as a plurality of measurement areas for extracting luminance signals representing luminance changes; Extracting, by a luminance signal extraction unit (14), the luminance signals based on the luminance changes in the measurement areas on the captured image; Estimating, by a pulse wave estimation unit (16), a pulse wave of the human based on the extracted luminance signals; and Performing, by the measurement area setting unit (13), a measurement area reset when a ratio of measurement areas in which a number of pixels is equal to or greater than a preset reference value among the set plurality of measurement areas is lower than a threshold, in such a manner that the ratio of measurement areas is equal to or higher than the threshold.

Images