Pulse wave estimation device and pulse wave estimation method

Abstract

To obtain a pulse wave estimation device capable of suppressing a decrease in estimation accuracy of a pulse wave.SOLUTION: A pulse wave estimation device 1 includes: a photographed image acquisition unit 11 that acquires a photographed image obtained by imaging a person; a skin region detection unit 12 that detects skin regions of the person from the photographed image; a measurement region setting unit 13 that sets regions obtained by dividing the skin region in the photographed image into prescribed regions as multiple measurement regions for extracting luminance signals indicating luminance variations; a luminance signal extraction unit 14 that extracts the luminance signals on the basis of the luminance variations in the measurement regions on the photographed image; and a pulse wave estimation unit 16 that estimates the pulse wave of the person on the basis of the extracted luminance signals. The measurement region setting unit 13 resets, when a ratio of the measurement regions with the number of pixels being equal to or greater than a predetermined reference value among the set multiple measurement regions is less than a threshold, the measurement regions such that the ratio becomes equal to or greater than the threshold.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a pulse wave estimation device and a pulse wave estimation method. [Background technology]

[0002] As a method for estimating a subject's pulse wave in a non-contact manner without imposing a burden on the subject, there is a method in which the subject's face is imaged with a camera and the pulse wave is estimated from minute changes in brightness on the subject's facial surface (for example, Non-Patent Document 1). In this method, for example, multiple measurement areas are set on the subject's facial image, and the frequency power spectrum of the brightness signal acquired in each of the set measurement areas is calculated. Then, the subject's pulse wave is estimated according to the peak frequency of the frequency power spectrum calculated in each measurement area. In addition, the estimated pulse waves are synthesized, and the pulse rate is estimated from the peak of the frequency power spectrum of the synthesized pulse wave. [Prior art documents] [Patent documents]

[0003] [Non-Patent Document 1] Mayank Kumar, et al., "DistancePPG: Robust non-contact vital signs monitoring using a camera", Biomedical optics express, 6(5), 1565-1588, 2015 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the above method has a problem that if the resolution of the image of the subject is reduced due to a change in the installation layout of the cameras, the resolution is reduced and the accuracy of estimating the pulse wave is reduced.

[0005] The present disclosure has been made to solve the above-mentioned problems, and has an object to provide a pulse wave estimation device that can suppress a decrease in the accuracy of estimating a pulse wave. [Means for solving the problem]

[0006] The pulse wave estimation device according to the present disclosure comprises an image acquisition unit that acquires an image of a person, a skin area detection unit that detects the person's skin area from the image, a measurement area setting unit that sets areas obtained by dividing the skin area in the image into predetermined parts as multiple measurement areas for extracting a luminance signal indicating a change in luminance, a luminance signal extraction unit that extracts the luminance signal based on the luminance change in the measurement area on the image, and a pulse wave estimation unit that estimates the person's pulse wave based on the extracted luminance signal, and is characterized in that, when the proportion of the measurement areas having a pixel count equal to or greater than a predetermined reference value among the multiple set measurement areas is less than a threshold value, the measurement area setting unit resets the measurement area so that the proportion is equal to or greater than the threshold value. Effect of the Invention

[0007] According to the present disclosure, with the above-described configuration, it is possible to suppress a decrease in the accuracy of estimating a pulse wave. [Brief description of the drawings]

[0008] [Figure 1] 1 is a diagram illustrating an example of the configuration of a pulse wave estimation device according to a first embodiment. [Diagram 2] 2A, 2B, and 2C are diagrams for explaining an example of a method for setting a measurement region by the measurement region setting unit in the first embodiment. [Diagram 3] 3A and 3B are diagrams for explaining an example of setting of a measurement region by the measurement region setting unit in the first embodiment. [Figure 4] 4 is a flowchart showing an example of the operation of the pulse wave estimation device according to the first embodiment. [Diagram 5] 5 is a flowchart illustrating details of a process performed by a measurement area setting unit in the first embodiment. [Figure 6] 6 is a diagram for explaining an example of resetting a measurement region by a measurement region setting unit in the first embodiment. FIG. [Figure 7]7A, 7B, 7C, and 7D are diagrams for explaining an example of resetting the measurement region by the measurement region setting unit in the first embodiment. [Figure 8] 8A and 8B are diagrams illustrating an example of a hardware configuration of a pulse wave estimation device according to the first embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, the embodiments will be described in detail with reference to the drawings. Embodiment 1 1 is a diagram showing an example of the configuration of a pulse wave estimation device 1 according to embodiment 1. Pulse wave estimation device 1 estimates the pulse wave of a person based on a captured image of the person. In the following description, a person whose pulse wave is to be estimated by pulse wave estimation device 1 is also referred to as a "subject."

[0010] Pulse wave estimation device 1 acquires captured images consisting of a series of frames Im(k) capturing an area including at least the skin region of the subject at a predetermined frame rate Fr. Here, k indicates a frame number assigned to each frame. For example, the frame provided next to frame Im(k) is frame Im(k+1). Pulse wave estimation device 1 estimates the subject's pulse wave for each frame Im(k) and outputs pulse wave estimation result P(t) which is information indicating the estimated pulse wave (hereinafter referred to as "pulse wave information"). The pulse wave information may be, for example, time series data of the subject's pulse wave estimated by pulse wave estimation unit 16 described later, or the subject's pulse rate.

[0011] The number of subjects, which are people, included in the captured image may be one or more. In the following description, for simplicity, the number of subjects included in the captured image is assumed to be one. In the following description, as an example, pulse wave estimation device 1 is mounted on a vehicle (not shown), and the subject is the driver of the vehicle. In other words, pulse wave estimation device 1 estimates the pulse wave of the driver of the vehicle.

[0012] As shown in FIG. 1, the pulse wave estimation device 1 includes an image acquisition unit 11, a skin area detection unit 12, a measurement area setting unit 13, a luminance signal extraction unit 14, a luminance signal segment generation unit 15, and a pulse wave estimation unit 16.

[0013] The captured image acquisition unit 11 acquires a captured image of the driver, who is a subject. Specifically, the captured image acquisition unit 11 acquires a captured image of the driver of the vehicle by an imaging device (not shown) mounted on the vehicle. The imaging device is installed so as to be able to capture an image of the range in which the skin area of ​​the driver exists. The captured image acquisition unit 11 outputs the acquired captured image to the skin area detection unit 12.

[0014] The skin area detection unit 12 detects the skin area of ​​the driver, who is the subject, from a frame Im(k) included in the captured image acquired by the captured image acquisition unit 11. Then, the skin area detection unit 12 sets information indicating the detected skin area as skin area information S(k) and outputs this skin area information S(k) to the measurement area setting unit 13 together with the frame Im(k) in which the skin area information S(k) was detected.

[0015] In the following description, the skin region is assumed to be information indicating a rectangular region that represents the position and size of the driver's face on the frame Im(k). The detection of the driver's face can be realized by using a known means. For example, the skin region detection unit 12 can extract a rectangular region surrounding the driver's face by using a cascade-type face detector that uses Haar-like features.

[0016] However, the skin region may be a region corresponding to a part of the driver other than the face. For example, the skin region may be a region corresponding to a part belonging to the face, such as the driver's eyes, eyebrows, nose, mouth, forehead, cheeks, chin, etc. Also, the skin region may be a region corresponding to a body part other than the face, such as the driver's head, shoulders, hands, neck, feet, etc. Also, the skin region may be a plurality of regions instead of one region.

[0017] The measurement area setting unit 13 acquires the frame Im(k) and the skin area information S(k) output from the skin area detection unit 12. The measurement area setting unit 13 also acquires vehicle layout information from a control device (not shown) mounted on the vehicle. The vehicle layout information includes at least information indicating the positional relationship between the imaging device installed in the vehicle and the driver. For example, the vehicle layout information includes information indicating the position where the imaging device is installed in the vehicle, information indicating the orientation of the imaging device, and information indicating the position of the seat where the driver sits based on the position where the imaging device is installed in the vehicle.

[0018] The measurement area setting unit 13 divides the skin area indicated by the skin area information S(k) on the frame Im(k) into predetermined parts based on the acquired frame Im(k), skin area information S(k), and vehicle layout information, and sets the areas obtained by the division as a plurality of measurement areas for extracting a luminance signal indicating a change in luminance.

[0019] When the measurement area setting unit 13 sets a plurality of measurement areas, it generates measurement area information R(k) indicating the set plurality of measurement areas. The measurement area information R(k) includes information indicating the position and size of Rn (positive integer) measurement areas on the captured image. Each measurement area is defined as measurement area fi(k) (i=1, 2, . . . , Rn). In the first embodiment, the measurement area fi(k) is defined as a quadrangle, and the position and size of the measurement area fi(k) are defined as the coordinate values ​​of the four vertices of the quadrangle on the captured image.

[0020] 2A, 2B, and 2C are diagrams for explaining an example of a method for setting a measurement region by the measurement region setting unit 13 in the pulse wave estimation device 1 according to embodiment 1. An example of a method for the measurement region setting unit 13 to set a plurality of measurement regions will be explained with reference to FIG.

[0021] First, as shown in Fig. 2A and Fig. 2B, the measurement area setting unit 13 detects Ln (positive integer) landmarks of facial parts such as the corners and corners of the eyes, the nose, and the mouth in the skin area sr indicated by the skin area information S(k). In Fig. 2A and Fig. 2B, the landmarks are indicated by circles. The measurement area setting unit 13 sets a vector storing the coordinate values ​​of the detected landmarks as L(k). The measurement area setting unit 13 may detect the facial parts using a known means such as a model called Constrained Local Model (CLM).

[0022] Next, the measurement area setting unit 13 sets the vertex coordinates of the quadrangle of the measurement area fi(k) based on the detected landmark. For example, the measurement area setting unit 13 sets the vertex coordinates of a quadrangle as shown in Fig. 2C, and sets Rn measurement areas fi(k).

[0023] To explain an example in which the measurement area setting unit 13 sets the measurement area fi(k) in a portion of the skin area sr corresponding to the cheek, the measurement area setting unit 13 selects a landmark LA1 on the facial contour and a landmark LA2 on the nose. The measurement area setting unit 13 first selects the landmark LA2 on the nose, and then selects the landmark LA1 on the facial contour that is closest to the landmark LA2 on the nose. The measurement area setting unit 13 then sets auxiliary landmarks a1, a2, and a3 so as to divide the line segment between the landmark LA1 and the landmark LA2 into four equal parts.

[0024] Similarly, the measurement area setting unit 13 selects a landmark LB1 on the face outline and a landmark LB2 on the nose. The measurement area setting unit 13 also sets auxiliary landmarks b1, b2, and b3 so as to divide the line segment between the landmarks LB1 and LB2 into four equal parts. Note that the landmarks LB1 and LB2 may be selected, for example, from the landmarks on the face outline or the nose adjacent to the landmarks LA1 and LA2, respectively.

[0025] The measurement area setting unit 13 sets a quadrilateral area surrounded by the auxiliary landmarks a1, b1, b2, and a2 as one measurement area R1. The auxiliary landmarks a1, b1, b2, and a2 each have vertex coordinates corresponding to the measurement area R1. Similarly, the measurement area setting unit 13 sets one measurement area R2 surrounded by the auxiliary landmarks a2, b2, b3, and a3 and the vertex coordinates of the measurement area R2.

[0026] Although an example of setting the measurement area fi(k) in the part corresponding to the cheek has been described here, the measurement area setting unit 13 similarly sets the measurement area fi(k) and the vertex coordinates of the measurement area fi(k) in the skin area sr of the other part of the cheek and the part corresponding to the chin. Although not shown in FIG. 2C, the measurement area setting unit 13 may set the measurement area fi(k) in the part of the skin area sr of the driver corresponding to the forehead or the part corresponding to the nose.

[0027] In this manner, the measurement region setting unit 13 divides the region corresponding to the left and right cheeks of the driver DR in the skin region indicated by the skin region information S(k) into predetermined regions as shown in Fig. 3A, for example, and sets the measurement regions fi(k) (i = 1, 2, 3, ..., n). In this case, the measurement region fi(k) is set, for example, in a U-shape with respect to the skin region indicated by the skin region information S(k).

[0028] The measurement area setting unit 13 may appropriately determine where to set the measurement area depending on the driver's facial orientation. For example, when the driver faces slightly rightward from the front, the measurement area setting unit 13 may set the measurement area fi(k) only in an area of ​​the skin area corresponding to the driver's left cheek (the area closer to the imaging device) as shown in FIG. 3B. In this case, the measurement area is set, for example, in a J-shape with respect to the skin area indicated by the skin area information S(k). The measurement area setting unit 13 may determine the driver's facial orientation using a known method. In the following description, the measurement area initially set by the measurement area setting unit 13 with respect to the skin area is also referred to as an "initial measurement area."

[0029] In addition, the measurement area setting unit 13 may determine the maximum number of measurement areas that can be set based on the acquired vehicle layout information and a predetermined angle range of the driver's face direction.

[0030] For example, when the driver DR faces slightly to the right from the front as shown in Fig. 3B and the imaging device captures an image of the driver DR from diagonally ahead, the measurement area setting unit 13 may set the maximum number of measurement areas to, for example, 28. Then, the measurement area setting unit 13 may set 28 measurement areas as initial measurement areas for the skin area indicated by the skin area information S(k) when the imaging device actually captures an image of the driver DR from diagonally ahead.

[0031] 3A, when the driver DR faces the front and the imaging device captures an image of the driver DR from the front, the measurement area setting unit 13 may set the maximum number of measurement areas to, for example, 40. Then, the measurement area setting unit 13 may set 40 measurement areas as initial measurement areas for the skin area indicated by the skin area information S(k) when the imaging device actually captures an image of the driver DR from the front.

[0032] When the measurement area setting unit 13 sets the initial measurement areas as described above, it calculates the number of pixels included in each of the set initial measurement areas.Then, the measurement area setting unit 13 calculates the ratio of the initial measurement areas in which the number of pixels is equal to or greater than a predetermined reference value among the set initial measurement areas, and if the calculated ratio is less than a threshold value, it resets the measurement areas so that the ratio is equal to or greater than the threshold value.The details of this resetting of the measurement areas will be described later.

[0033] The measurement region setting unit 13 outputs measurement region information R(k) indicating the multiple measurement regions fi(k) that have been set and reset to the luminance signal extraction unit .

[0034] Based on the measurement region information R(k) output from the measurement region setting unit 13, the luminance signal extraction unit 14 extracts a luminance signal indicating a change in luminance from each measurement region fi(k) indicated by the measurement region information R(k).

[0035] For example, the luminance signal extraction unit 14 calculates, for each measurement region, an average value of the luminance values ​​of pixels included in each measurement region fi(k) set by the measurement region setting unit 13 in the current frame. Then, the luminance signal extraction unit 14 sets information indicating the calculated average value of the luminance values ​​for each measurement region as luminance signal information Gi(k). Similarly, the luminance signal extraction unit 14 calculates, for each measurement region, an average value of the luminance values ​​of pixels included in each measurement region fi(k-1) set by the measurement region setting unit 13 in the frame immediately preceding the current frame. Then, the luminance signal extraction unit 14 sets information indicating the calculated average value of the luminance values ​​for each measurement region as luminance signal information Gi(k-1).

[0036] Then, the luminance signal extraction unit 14 calculates the difference between the luminance signal information Gi(k) and the luminance signal information Gi(k-1). Specifically, the luminance signal extraction unit 14 calculates the difference between the average values ​​of the luminance values ​​in the corresponding measurement regions between the luminance signal information Gi(k) and the luminance signal information Gi(k-1). At this time, if there is no corresponding measurement region between the luminance signal information Gi(k) and the luminance signal information Gi(k-1), the luminance signal extraction unit 14 does not calculate the difference for that measurement region.

[0037] Then, the luminance signal extraction unit 14 outputs information indicating the difference between the calculated luminance signal information Gi(k) and luminance signal information Gi(k-1) as luminance signal segment information W(t) to the luminance signal segment generation unit 15. At this time, the luminance signal extraction unit 14 includes information indicating the facial direction of the subject in the current frame (k) and information indicating the facial direction of the subject in the previous frame (k-1) in the luminance signal segment information W(t).

[0038] Here, t is a number corresponding to the time difference between the current frame (k) and the previous frame (k-1). For example, when luminance signal segment information indicating the difference between luminance signal information Gi(k) and luminance signal information Gi(k-1) is W(t), luminance signal segment information indicating the difference between luminance signal information Gi(k+1) and luminance signal information Gi(k) is W(t+1).

[0039] In the above description, the luminance signal extraction unit 14 uses information indicating the average value of the calculated luminance values ​​for each measurement region as the luminance signal information Gi(k). However, the luminance signal extraction unit 14 is not limited to this, and may use information indicating the variance of the calculated luminance values ​​for each measurement region as the luminance signal information Gi(k).

[0040] The luminance signal segment generating unit 15 accumulates (stores) the luminance signal segment information W(t) output from the luminance signal extracting unit 14 in a storage unit such as a memory until a predetermined amount of information required for estimating the pulse wave is accumulated. In this way, the luminance signal segment generating unit 15 generates a time-series luminance signal (luminance signal segment) that represents the change in luminance value over time for each measurement region.

[0041] Then, the luminance signal segment generating unit 15 selects, from the generated time-series luminance signals for each measurement region, a time-series luminance signal to be used for estimating a pulse wave by the pulse wave estimating unit 16, which will be described later. For example, the luminance signal segment generating unit 15 selects, from the generated time-series luminance signals for each measurement region, a time-series luminance signal in which the deviation between the face orientation of the driver DR in the current frame (k) and the previous frame (k-1) included in the luminance signal segment information W(t) and a predetermined reference face orientation is within a predetermined range (for example, a range from "-5 degrees" to "+5 degrees"), as a time-series luminance signal to be used for estimating the subject's pulse wave.

[0042] The reason why the luminance signal segment generating unit 15 selects the time-series luminance signal in this way is as follows. That is, in pulse wave estimation, a pulse wave (pulse) is basically captured by capturing a luminance change within the same measurement area, but a change in luminance can occur due to causes other than blood flow. In particular, when the face direction of the driver DR, who is the subject, changes, the direction in which light enters the face of the driver DR changes, which causes a large change in luminance. Therefore, in pulse wave estimation, it is desirable not to use a luminance value (luminance signal) when the face direction is significantly different from the reference face direction for estimating the pulse wave. Therefore, the luminance signal segment generating unit 15 selects a time-series luminance signal to be used for estimating the subject's pulse wave, using the face direction of the driver DR as described above as a condition.

[0043] Luminance signal segment generating section 15 outputs information indicating the selected time-series luminance signal to pulse wave estimating section 16 as segment information X(t).

[0044] Pulse wave estimation unit 16 estimates the subject's pulse wave based on segment information X(t) output from luminance signal segment generation unit 15. For example, pulse wave estimation unit 16 adds up differences in average luminance values ​​for each measurement region fi(k) included in segment information X(t) output from luminance signal segment generation unit 15 to calculate a single piece of pulse wave signal information. Pulse wave estimation unit 16 then performs a Fourier transform on the calculated pulse wave signal information, and estimates the subject's pulse wave based on the peak frequency in the frequency power spectrum.

[0045] In the above description, an example has been described in which pulse wave estimation device 1 includes luminance signal segment generation section 15, but luminance signal segment generation section 15 is not an essential component and may be omitted. In that case, pulse wave estimation section 16 accumulates (stores) luminance signal segment information W(t) output from luminance signal extraction section 14 in a storage section such as a memory until a predetermined amount of information required for estimating the pulse wave has been accumulated, and estimates the subject's pulse wave based on the accumulated information.

[0046] Next, a description will be given of an example of the operation of pulse wave estimation device 1 shown in Fig. 1. Fig. 4 is a flowchart showing an example of the operation of pulse wave estimation device 1.

[0047] First, the captured image acquisition unit 11 acquires a captured image of the driver DR who is a subject (step ST1). The captured image acquisition unit 11 outputs the acquired captured image to the skin region detection unit 12.

[0048] Next, the skin region detection unit 12 detects a skin region from a frame Im(k) included in the captured image acquired by the captured image acquisition unit 11 in step ST1 (step ST2). The skin region detection unit 12 generates skin region information S(k) indicating the detected skin region, and outputs the generated skin region information S(k) to the measurement region setting unit 13 together with the frame Im(k).

[0049] Next, the measurement area setting unit 13 divides the skin area indicated by the skin area information S(k) on the frame Im(k) into predetermined parts based on the frame Im(k), the skin area information S(k), and the vehicle layout information, and sets the areas obtained by the division as multiple measurement areas for extracting a luminance signal indicating a change in luminance (step ST3).

[0050] Here, the details of the process performed by the measurement area setting unit 13 will be described with reference to the flowchart shown in FIG.

[0051] First, the measurement area setting unit 13 divides the skin area indicated by the skin area information S(k) on the frame Im(k) into predetermined parts, and sets the areas obtained by the division as a plurality of measurement areas (initial measurement areas) for extracting a luminance signal indicating a luminance change (step ST31).

[0052] Next, the measurement area setting unit 13 calculates the number of pixels included in each of the set initial measurement areas. Then, the measurement area setting unit 13 calculates the ratio of the initial measurement areas having a pixel count equal to or greater than a predetermined reference value among the set initial measurement areas, and determines whether the calculated ratio is equal to or greater than a threshold value (step ST32).

[0053] As a result, if the calculated ratio is equal to or greater than the threshold (step ST32; YES), the measurement area setting unit 13 ends the process. After that, the process returns to step ST4 shown in Fig. 4. On the other hand, if the calculated ratio is not equal to or greater than the threshold (step ST32; NO), the measurement area setting unit 13 resets the measurement area so that the ratio is equal to or greater than the threshold (step ST33).

[0054] The reference value and threshold value may be appropriately set by, for example, an administrator of the pulse wave estimation device 1 according to the accuracy required for pulse wave estimation. For example, the administrator of the pulse wave estimation device 1 sets the reference value and threshold value to be larger as the accuracy required for pulse wave estimation is higher. For example, when the threshold value is set to "100%", the measurement area setting unit 13 resets the measurement area if at least one of the set initial measurement areas has a pixel count less than a predetermined reference value. Also, when the threshold value is set to "50%", for example, the measurement area setting unit 13 does not reset the measurement area if half or more of the set initial measurement areas have a pixel count equal to or greater than a predetermined reference value.

[0055] The measurement area setting unit 13 resets the measurement area using, for example, the following three methods. (Method 1) For example, the measurement area setting unit 13 resets the measurement area by reducing the number of divisions of the skin area when the initial measurement area was set so that the ratio is equal to or greater than the threshold. For example, when the initial measurement area is set to 28 as shown in Fig. 3B, if the ratio is not equal to or greater than the threshold, the measurement area setting unit 13 resets the measurement area by reducing the number of divisions of the skin area to 8 as shown in Fig. 6. In addition, at this time, the measurement area setting unit 13 increases the number of pixels per measurement area after resetting shown in Fig. 6 from the number of pixels per measurement area before resetting shown in Fig. 3B.

[0056] Here, when reducing the number of divisions of the skin region, the measurement region setting unit 13 may reduce the number of divisions of the skin region by merging some of the adjacent initial measurement regions among the multiple initial measurement regions shown in Fig. 3B. Alternatively, the measurement region setting unit 13 may discard the multiple initial measurement regions shown in Fig. 3B and re-divide the skin region into a smaller number of divisions than when the initial measurement region was set, thereby re-setting the measurement region.

[0057] (Method 2) For example, the measurement area setting unit 13 resets the measurement area by setting a new measurement area that spans a predetermined number of adjacent initial measurement areas among the multiple initial measurement areas so that the ratio is equal to or greater than the threshold. For example, when 28 initial measurement areas are set as shown in Fig. 3B, if the ratio is not equal to or greater than the threshold, the measurement area setting unit 13 resets the measurement area by setting a new measurement area that spans two or four adjacent initial measurement areas among the 28 initial measurement areas as shown in Figs. 7A to 7D. In this example, 13 measurement areas from fi(1) to fi(13) are newly set.

[0058] In addition, the 13 new measurement regions are allowed to overlap with each other. For example, measurement region fi(9) overlaps with measurement regions fi(1) and fi(3), and measurement region fi(10) overlaps with measurement regions fi(2) and fi(4).

[0059] Even in this case, the measurement area setting unit 13 increases the number of pixels per measurement area after resetting shown in FIGS. 7A to 7D from the number of pixels per measurement area before resetting shown in FIG. 3B.

[0060] (Method 3) For example, the measurement area setting unit 13 may combine method 1 and method 2. For example, the measurement area setting unit 13 may reset the measurement area by reducing the number of divisions of the skin area and setting a new measurement area across a predetermined number of adjacent measurement areas among the multiple measurement areas obtained by reducing the number of divisions so that the ratio is equal to or greater than a threshold. For example, a new measurement area across two adjacent measurement areas among the eight measurement areas shown in FIG. 6 may be set.

[0061] The measurement area setting unit 13 resets the measurement area using any one of the above methods 1 to 3. After that, the process returns to step ST4 shown in FIG.

[0062] Next, a description will be given of the effects of the pulse wave estimation device 1 according to embodiment 1. As described above, the pulse wave estimation device 1 can prevent a decrease in the estimation accuracy of the pulse wave more effectively than before by resetting the measurement region.

[0063] For example, in the conventional pulse wave estimation method, when the resolution of the image of the driver DR is reduced due to a change in the installation layout of the imaging device that images the driver DR, who is the subject, or the imaging device itself is changed, the resolution is reduced and the estimation accuracy of the pulse wave is reduced. Also, in the conventional pulse wave estimation method, when the driver DR moves the face of the driver DR away from the imaging device by lowering the vehicle seat backward or changing the face direction, the resolution of the image of the driver DR is reduced, and the same problem may occur.

[0064] In contrast, in pulse wave estimation device 1, when the ratio of measurement areas in which the number of pixels is equal to or greater than a predetermined reference value among the multiple measurement areas set by measurement area setting unit 13 is less than a threshold value, the measurement areas are dynamically reset so that the ratio becomes equal to or greater than the threshold value. As a result, in pulse wave estimation device 1, the ratio of measurement areas in which the number of pixels is equal to or greater than the reference value after resetting becomes equal to or greater than the threshold value, suppressing a decrease in resolution and suppressing a decrease in pulse wave estimation accuracy.

[0065] Furthermore, in the pulse wave estimation device 1, when the above method 2 is adopted by the measurement area setting unit 13, the following effect is also achieved. For example, when the measurement area setting unit 13 resets the measurement area by reducing the number of divisions of the skin area as in the above method 1, if localized external light is reflected in part of the reset measurement area, that measurement area cannot be used for estimating the pulse wave, so that the measurement area that can be used for estimating the pulse wave is reduced, and there is a risk of reduced robustness.

[0066] In this regard, in the above-mentioned method 2, the measurement area setting unit 13 resets a new measurement area spanning a predetermined number of adjacent initial measurement areas while allowing them to overlap with each other, so that the size of the measurement area (the number of pixels in the measurement area) can be secured while also securing the number of measurement areas that can be used to estimate the pulse wave, thereby suppressing a decrease in robustness compared to method 1.

[0067] Furthermore, in the pulse wave estimation device 1, when the above-mentioned method 3 is adopted by the measurement area setting unit 13, the size of the measurement area after reconfiguration (the number of pixels in the measurement area) can be increased more than when method 1 or method 2 is adopted alone, and the decrease in robustness that is a problem with method 1 can also be suppressed.

[0068] Next, an example of a hardware configuration of pulse wave estimation device 1 according to embodiment 1 will be described with reference to Fig. 8. The functions of captured image acquisition section 11, skin region detection section 12, measurement region setting section 13, luminance signal extraction section 14, luminance signal segment generation section 15, and pulse wave estimation section 16 in pulse wave estimation device 1 are realized by processing circuits. The processing circuit may be dedicated hardware as shown in Fig. 8A, or may be a CPU (also called a Central Processing Unit, central processing unit, processing device, arithmetic unit, microprocessor, microcomputer, processor, or DSP (Digital Signal Processor)) 52 that executes a program stored in memory 53 as shown in Fig. 14B.

[0069] When the processing circuit is a dedicated hardware, the processing circuit 51 corresponds to, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination of these. The functions of each of the captured image acquisition unit 11, the skin region detection unit 12, the measurement region setting unit 13, the luminance signal extraction unit 14, the luminance signal segment generation unit 15, and the pulse wave estimation unit 16 may be realized by the processing circuit 51 individually, or the functions of each unit may be realized by the processing circuit 51 collectively.

[0070] When the processing circuit is a CPU 52, the functions of the captured image acquisition unit 11, the skin area detection unit 12, the measurement area setting unit 13, the luminance signal extraction unit 14, the luminance signal segment generation unit 15, and the pulse wave estimation unit 16 are realized by software, firmware, or a combination of software and firmware. The software and firmware are written as programs and stored in the memory 53. The processing circuit realizes the functions of each unit by reading and executing the programs stored in the memory 53. That is, the pulse wave estimation device 1 includes a memory for storing a program that, when executed by the processing circuit, results in the execution of each step shown in FIG. 4. It can also be said that these programs cause a computer to execute the procedures and methods of the captured image acquisition unit 11, the skin area detection unit 12, the measurement area setting unit 13, the luminance signal extraction unit 14, the luminance signal segment generation unit 15, and the pulse wave estimation unit 16. Here, examples of the memory 53 include non-volatile or volatile semiconductor memories such as RAM (RANdom access memory), ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable ROM), EEPROM (Electrically EPROM), magnetic disks, flexible disks, optical disks, compact disks, mini disks, and DVDs (Digital Versatile Discs).

[0071] The functions of the captured image acquisition unit 11, skin area detection unit 12, measurement area setting unit 13, luminance signal extraction unit 14, luminance signal segment generation unit 15, and pulse wave estimation unit 16 may be partially realized by dedicated hardware and partially realized by software or firmware. For example, the functions of the captured image acquisition unit 11 may be realized by a processing circuit as dedicated hardware, and the functions of the skin area detection unit 12, measurement area setting unit 13, luminance signal extraction unit 14, luminance signal segment generation unit 15, and pulse wave estimation unit 16 may be realized by the processing circuit reading and executing a program stored in the memory 53.

[0072] Thus, the processing circuitry may implement the above-described functions through hardware, software, firmware, or a combination of these.

[0073] As described above, according to the first embodiment, the pulse wave estimation device 1 includes an image acquisition unit 11 that acquires an image of a person, a skin region detection unit 12 that detects the skin region of the person from the image, a measurement region setting unit 13 that sets regions obtained by dividing the skin region in the image into predetermined regions as a plurality of measurement regions for extracting a luminance signal indicating a luminance change, a luminance signal extraction unit 14 that extracts a luminance signal based on a luminance change in the measurement region on the image, and a pulse wave estimation unit 16 that estimates the person's pulse wave based on the extracted luminance signal. When the ratio of the measurement regions having a pixel count equal to or greater than a predetermined reference value among the plurality of measurement regions that have been set is less than a threshold value, the measurement region setting unit 13 resets the measurement region so that the ratio is equal to or greater than the threshold value. This allows the pulse wave estimation device 1 according to the first embodiment to suppress a decrease in the estimation accuracy of the pulse wave.

[0074] Furthermore, when the ratio is less than the threshold, the measurement region setting unit 13 resets the measurement region by reducing the number of divisions of the skin region in the captured image, thereby enabling the pulse wave estimation device 1 according to the first embodiment to easily reset the measurement region.

[0075] Furthermore, when the ratio is less than the threshold value, the measurement region setting unit 13 resets the measurement region by setting a new measurement region that spans a predetermined number of adjacent measurement regions among the multiple measurement regions that have been set. This allows the pulse wave estimation device 1 according to the first embodiment to easily reset the measurement region.

[0076] In addition, multiple new measurement regions can be set and can overlap each other, so that the pulse wave estimation device 1 according to the first embodiment can ensure the size of the measurement region and the number of measurement regions that can be used for estimating a pulse wave.

[0077] Furthermore, when the ratio is less than the threshold, the measurement region setting unit 13 resets the measurement region by reducing the division number of the skin region in the captured image and setting a new measurement region that spans a predetermined number of adjacent measurement regions among the multiple measurement regions obtained by reducing the division number. This allows the pulse wave estimation device 1 according to the first embodiment to increase the size of the measurement region after resetting.

[0078] Furthermore, measurement area setting unit 13 increases the number of pixels per measurement area after the resetting compared to the number of pixels per measurement area before the resetting, thereby enabling pulse wave estimation device 1 according to embodiment 1 to suppress a decrease in resolution and a decrease in pulse wave estimation accuracy.

[0079] Pulse wave estimation device 1 also includes luminance signal segment generation unit 15 that selects a luminance signal to be used for estimating the person's pulse wave from among the luminance signals extracted by luminance signal extraction unit 14, and pulse wave estimation unit 16 estimates the person's pulse wave based on the luminance signal selected by luminance signal segment generation unit 15. This allows pulse wave estimation device 1 according to embodiment 1 to effectively suppress deterioration in pulse wave estimation accuracy.

[0080] Furthermore, luminance signal segment generating unit 15 selects, from among the luminance signals extracted by luminance signal extracting unit 14, a luminance signal whose deviation between the face direction of the person at the time the luminance signal was extracted and a predetermined reference face direction is within a predetermined range, as a luminance signal to be used for estimating the person's pulse wave. In this way, pulse wave estimation device 1 according to the first embodiment can select a time-series luminance signal to be used for estimating the pulse wave, with the face direction of the person as a condition.

[0081] Although the preferred embodiments have been described above, the present invention is not limited to the above-described embodiments. The above-described embodiments are not intended to be limiting unless otherwise specified, and are not intended to be limiting unless otherwise specified. Various modifications and substitutions can be made thereto.

[0082] Various aspects of the present disclosure are summarized below as appendices.

[0083] (Appendix 1) an image acquisition unit for acquiring an image of a person; a skin area detection unit that detects a skin area of ​​the person from the captured image; a measurement area setting unit that sets areas obtained by dividing the skin area in the captured image in a predetermined manner as a plurality of measurement areas for extracting a luminance signal indicating a luminance change; a luminance signal extraction unit that extracts the luminance signal based on a luminance change in the measurement area on the captured image; a pulse wave estimation unit that estimates a pulse wave of the person based on the extracted luminance signal, The measurement area setting unit is a pulse wave estimation device which, when a ratio of the measurement areas having a pixel count equal to or greater than a predetermined reference value among the plurality of measurement areas which have been set is less than a threshold value, resets the measurement areas so that the ratio becomes equal to or greater than the threshold value. (Appendix 2) The measurement area setting unit is If the ratio is less than the threshold, the measurement area is reset by reducing the number of divisions of the skin area in the captured image. 2. A pulse wave estimation device according to claim 1. (Appendix 3) The measurement area setting unit is When the ratio is less than a threshold value, a new measurement area is set across a predetermined number of adjacent measurement areas among the set measurement areas, thereby resetting the measurement area. 2. A pulse wave estimation device according to claim 1. (Appendix 4) The pulse wave estimation device according to claim 3, wherein the new measurement areas can be set in multiple areas and can overlap with each other. (Appendix 5) The measurement area setting unit is If the ratio is less than a threshold value, the number of divisions of the skin region in the captured image is reduced, and among the multiple measurement regions obtained by reducing the number of divisions, a new measurement region that spans a predetermined number of adjacent measurement regions is set, thereby resetting the measurement region. 2. A pulse wave estimation device according to claim 1. (Appendix 6) The measurement area setting unit is The number of pixels per measurement area after the resetting is increased to be greater than the number of pixels per measurement area before the resetting. 6. The pulse wave estimation device according to claim 1, (Appendix 7) a luminance signal segment generation unit that selects a luminance signal to be used for estimating a pulse wave of the person from among the luminance signals extracted by the luminance signal extraction unit, The pulse wave estimation unit estimates a pulse wave of the person based on the luminance signal selected by the luminance signal segment generation unit. 7. The pulse wave estimation device according to claim 1, (Appendix 8) The luminance signal segment generation unit Among the luminance signals extracted by the luminance signal extraction unit, a luminance signal in which a deviation between the face direction of the person when the luminance signal was extracted and a predetermined reference face direction is within a predetermined range is selected as a luminance signal to be used for estimating the pulse wave of the person. 8. The pulse wave estimation device according to claim 7. (Appendix 9) A pulse wave estimation method using a pulse wave estimation device, comprising: A step in which a captured image acquisition unit acquires a captured image of a person; a skin area detection unit detecting a skin area of ​​the person from the captured image; a measurement area setting unit setting areas obtained by dividing the skin area in the captured image into predetermined areas as a plurality of measurement areas for extracting a luminance signal indicating a luminance change; A luminance signal extraction unit extracts the luminance signal based on a luminance change in the measurement area on the captured image; and a pulse wave estimating unit estimating a pulse wave of the person based on the extracted luminance signal; The measurement area setting unit is a pulse wave estimation method comprising: when a ratio of the measurement areas having a pixel count equal to or greater than a predetermined reference value among the plurality of measurement areas that are set is less than a threshold value, resetting the measurement areas so that the ratio is equal to or greater than the threshold value. [Industrial Applicability]

[0084] The present disclosure is capable of suppressing a decrease in the accuracy of estimating a pulse wave, and is suitable for use in a pulse wave estimation device. [Explanation of symbols]

[0085] 1 Pulse wave estimation device, 11 Image acquisition unit, 12 Skin area detection unit, 13 Measurement area setting unit, 14 Luminance signal extraction unit, 15 Luminance signal segment generation unit, 16 Pulse wave estimation unit, 51 Processing circuit, 52 CPU, 53 Memory, a1 to a3 Auxiliary landmarks, b1 to b3 Auxiliary landmarks, DR Driver, fi Measurement area, LA1 to LA2 Landmarks, LB1 to LB2 Landmarks, sr Skin area.

Claims

1. an image acquisition unit for acquiring an image of a person; a skin area detection unit that detects a skin area of ​​the person from the captured image; a measurement area setting unit that sets areas obtained by dividing the skin area in the captured image in a predetermined manner as a plurality of measurement areas for extracting a luminance signal indicating a luminance change; a luminance signal extraction unit that extracts the luminance signal based on a luminance change in the measurement area on the captured image; a pulse wave estimation unit that estimates a pulse wave of the person based on the extracted luminance signal, The measurement area setting unit is a pulse wave estimation device which, when a ratio of the measurement areas having a pixel count equal to or greater than a predetermined reference value among the plurality of measurement areas which have been set is less than a threshold value, resets the measurement areas so that the ratio becomes equal to or greater than the threshold value.

2. The measurement area setting unit is If the ratio is less than the threshold, the measurement area is reset by reducing the number of divisions of the skin area in the captured image.

2. The pulse wave estimating device according to claim 1.

3. The measurement area setting unit is When the ratio is less than a threshold value, a new measurement area is set across a predetermined number of adjacent measurement areas among the set measurement areas, thereby resetting the measurement area.

2. The pulse wave estimating device according to claim 1.

4. 4. The pulse wave estimation device according to claim 3, wherein a plurality of the new measurement regions can be set, and the new measurement regions can overlap with each other.

5. The measurement area setting unit is If the ratio is less than a threshold value, the number of divisions of the skin region in the captured image is reduced, and among the multiple measurement regions obtained by reducing the number of divisions, a new measurement region that spans a predetermined number of adjacent measurement regions is set, thereby resetting the measurement region.

2. The pulse wave estimating device according to claim 1.

6. The measurement area setting unit is The number of pixels per measurement area after the resetting is increased to be greater than the number of pixels per measurement area before the resetting.

6. The pulse wave estimation device according to claim 1,

7. a luminance signal segment generation unit that selects a luminance signal to be used for estimating a pulse wave of the person from among the luminance signals extracted by the luminance signal extraction unit, The pulse wave estimation unit estimates a pulse wave of the person based on the luminance signal selected by the luminance signal segment generation unit.

2. The pulse wave estimating device according to claim 1.

8. The luminance signal segment generation unit Among the luminance signals extracted by the luminance signal extraction unit, a luminance signal in which a deviation between the face direction of the person when the luminance signal was extracted and a predetermined reference face direction is within a predetermined range is selected as a luminance signal to be used for estimating the pulse wave of the person.

8. The pulse wave estimating device according to claim 7.

9. A pulse wave estimation method using a pulse wave estimation device, comprising: A step in which a captured image acquisition unit acquires a captured image of a person; a skin area detection unit detecting a skin area of ​​the person from the captured image; a measurement area setting unit setting areas obtained by dividing the skin area in the captured image into predetermined areas as a plurality of measurement areas for extracting a luminance signal indicating a luminance change; A luminance signal extraction unit extracts the luminance signal based on a luminance change in the measurement area on the captured image; and a pulse wave estimating unit estimating a pulse wave of the person based on the extracted luminance signal; The measurement area setting unit is a pulse wave estimation method comprising: when a ratio of the measurement areas having a pixel count equal to or greater than a predetermined reference value among the plurality of measurement areas that are set is less than a threshold value, resetting the measurement areas so that the ratio is equal to or greater than the threshold value.

Images