Pulse wave estimation device and pulse wave estimation method

Abstract

To obtain a pulse wave estimation device capable of suppressing a decrease in estimation accuracy of a pulse wave.SOLUTION: A pulse wave estimation device 1 includes: a photographed image acquisition unit 11 that acquires a photographed image obtained by imaging a person; a skin region detection unit 12 that detects skin regions of the person from the photographed image; a measurement region setting unit 13 that sets regions obtained by dividing the skin region in the photographed image into prescribed regions as multiple measurement regions for extracting luminance signals indicating luminance variations; a luminance signal extraction unit 14 that extracts the luminance signals on the basis of the luminance variations in the measurement regions on the photographed image; and a pulse wave estimation unit 16 that estimates the pulse wave of the person on the basis of the extracted luminance signals. The measurement region setting unit 13 resets, when a ratio of the measurement regions with the number of pixels being equal to or greater than a predetermined reference value among the set multiple measurement regions is less than a threshold, the measurement regions such that the ratio becomes equal to or greater than the threshold.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a pulse wave estimation device and a pulse wave estimation method. [Background technology]

[0002] One method for estimating a subject's pulse wave without contact and without imposing a burden on the subject involves capturing an image of the subject's face with a camera and estimating the pulse wave from minute changes in luminance on the subject's facial surface (see, for example, Non-Patent Document 1). In this method, for example, multiple measurement areas are set on the subject's facial image, and the frequency power spectrum of the luminance signal acquired in each of the set measurement areas is calculated. The subject's pulse wave is then estimated based on the peak frequency of the frequency power spectrum calculated for each measurement area. Alternatively, the estimated pulse waves may be combined, and the pulse rate may be estimated from the peak of the frequency power spectrum of the combined pulse wave. [Prior art documents] [Patent documents]

[0003] [Non-Patent Document 1] Mayank Kumar, et al., "DistancePPG: Robust non-contact vital signs monitoring using a camera", Biomedical optics express, 6(5), 1565-1588, 2015 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the above method has a problem that if the resolution of the image of the subject is reduced due to a change in the installation layout of the camera, the resolution is reduced and the accuracy of estimating the pulse wave is reduced.

[0005] The present disclosure has been made to solve the above-mentioned problems, and aims to provide a pulse wave estimation device that can suppress a decrease in pulse wave estimation accuracy. [Means for solving the problem]

[0006] A pulse wave estimation device according to the present disclosure includes an image acquisition unit that acquires an image of a person, a skin region detection unit that detects a skin region of the person from the image, a measurement region setting unit that sets regions obtained by dividing the skin region in the image into predetermined parts as a plurality of measurement regions for extracting luminance signals indicating luminance changes, a luminance signal extraction unit that extracts luminance signals based on luminance changes in the measurement regions on the image, and a pulse wave estimation unit that estimates the person's pulse wave based on the extracted luminance signals, and when the proportion of measurement regions having a pixel count equal to or greater than a predetermined reference value among the plurality of set measurement regions is less than a threshold, the measurement region setting unit resets the measurement regions so that the proportion is equal to or greater than the threshold. The number of pixels per measurement area after the resetting is increased to be greater than the number of pixels per measurement area before the resetting. It is characterized by: [Effects of the Invention]

[0007] According to the present disclosure, with the above configuration, it is possible to suppress a decrease in the accuracy of estimating the pulse wave. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram illustrating an example of the configuration of a pulse wave estimation device according to a first embodiment. [Figure 2] 2A, 2B, and 2C are diagrams for explaining an example of a method for setting a measurement region by the measurement region setting unit according to the first embodiment. [Figure 3] 3A and 3B are diagrams for explaining an example of setting a measurement region by the measurement region setting unit according to the first embodiment. [Figure 4] 4 is a flowchart showing an example of the operation of the pulse wave estimation device according to the first embodiment. [Figure 5] 10 is a flowchart illustrating details of processing by a measurement region setting unit in the first embodiment. [Figure 6] 4 is a diagram for explaining an example of resetting a measurement region by a measurement region setting unit in the first embodiment. FIG. [Figure 7] 7A, 7B, 7C, and 7D are diagrams for explaining an example of resetting the measurement region by the measurement region setting unit in the first embodiment. [Figure 8] 8A and 8B are diagrams illustrating an example of a hardware configuration of a pulse wave estimation device according to the first embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments will be described in detail with reference to the drawings. Embodiment 1 1 is a diagram showing an example of the configuration of a pulse wave estimation device 1 according to embodiment 1. The pulse wave estimation device 1 estimates the pulse wave of a person based on a captured image of the person. In the following description, the person whose pulse wave is to be estimated by the pulse wave estimation device 1 is also referred to as a "subject."

[0010] Pulse wave estimation device 1 acquires captured images consisting of a series of frames Im(k), each capturing an area including at least the subject's skin, at a predetermined frame rate Fr. Here, k represents a frame number assigned to each frame. For example, the frame provided immediately following frame Im(k) is frame Im(k+1). Pulse wave estimation device 1 then estimates the subject's pulse wave for each frame Im(k) and outputs pulse wave estimation result P(t), which is information indicating the estimated pulse wave (hereinafter referred to as "pulse wave information"). The pulse wave information may be, for example, time-series data of the subject's pulse wave estimated by pulse wave estimation unit 16, which will be described later, or the subject's pulse rate.

[0011] The number of human subjects included in the captured image may be one or more. In the following description, for simplicity, the number of subjects included in the captured image will be described as one. In the following description, as an example, pulse wave estimation device 1 is mounted on a vehicle (not shown), and the subject is the driver of the vehicle. In other words, pulse wave estimation device 1 estimates the pulse wave of the vehicle driver.

[0012] As shown in FIG. 1, the pulse wave estimation device 1 includes an image acquisition unit 11, a skin area detection unit 12, a measurement area setting unit 13, a luminance signal extraction unit 14, a luminance signal segment generation unit 15, and a pulse wave estimation unit 16.

[0013] The captured image acquisition unit 11 acquires a captured image of the driver, who is the subject. Specifically, the captured image acquisition unit 11 acquires a captured image of the vehicle driver by an imaging device (not shown) mounted on the vehicle. The imaging device is installed so as to be able to capture an image of the range in which the driver's skin area exists. The captured image acquisition unit 11 outputs the acquired captured image to the skin area detection unit 12.

[0014] The skin region detection unit 12 detects the skin region of the driver, who is the subject, from the frame Im(k) included in the captured image acquired by the captured image acquisition unit 11. Then, the skin region detection unit 12 sets information indicating the detected skin region as skin region information S(k) and outputs this skin region information S(k) to the measurement region setting unit 13 together with the frame Im(k) in which it was detected.

[0015] In the following description, the skin region is assumed to be information indicating a rectangular region representing the position and size of the driver's face on the frame Im(k). The driver's face can be detected by using a known method. For example, the skin region detection unit 12 can extract a rectangular region surrounding the driver's face using a cascade face detector using Haar-like features.

[0016] However, the skin region may be a region corresponding to a part of the driver other than the face. For example, the skin region may be a region corresponding to a part of the face, such as the driver's eyes, eyebrows, nose, mouth, forehead, cheeks, or chin. The skin region may also be a region corresponding to a body part other than the face, such as the driver's head, shoulders, hands, neck, or feet. The skin region may be a single region or multiple regions.

[0017] The measurement area setting unit 13 acquires the frame Im(k) and the skin area information S(k) output from the skin area detection unit 12. The measurement area setting unit 13 also acquires vehicle layout information from a control device (not shown) mounted on the vehicle. The vehicle layout information includes at least information indicating the positional relationship between the imaging device installed in the vehicle and the driver. For example, the vehicle layout information includes information indicating the position where the imaging device is installed in the vehicle, information indicating the orientation of the imaging device, and information indicating the position of the seat where the driver sits based on the position where the imaging device is installed in the vehicle.

[0018] Based on the acquired frame Im(k), skin area information S(k), and vehicle layout information, the measurement area setting unit 13 divides the skin area indicated by the skin area information S(k) on the frame Im(k) into predetermined parts, and sets the areas obtained by the division as multiple measurement areas for extracting luminance signals indicating luminance changes.

[0019] When the measurement area setting unit 13 sets multiple measurement areas, it generates measurement area information R(k) that indicates the multiple measurement areas that have been set. The measurement area information R(k) includes information that indicates the position and size of Rn (a positive integer) measurement areas on the captured image. Each measurement area is defined as measurement area fi(k) (i=1, 2, . . . , Rn). In the first embodiment, the measurement area fi(k) is defined as a quadrangle, and the position and size of the measurement area fi(k) are defined as the coordinate values ​​of the four vertices of the quadrangle on the captured image.

[0020] 2A, 2B, and 2C are diagrams for explaining an example of a method for setting a measurement region by the measurement region setting unit 13 in the pulse wave estimation device 1 according to embodiment 1. An example of a method for the measurement region setting unit 13 to set a plurality of measurement regions will be explained using FIG.

[0021] First, as shown in FIGS. 2A and 2B, the measurement region setting unit 13 detects Ln (a positive integer) landmarks of facial features, such as the corners and corners of the eyes, the nose, and the mouth, in the skin region sr indicated by the skin region information S(k). In FIGS. 2A and 2B, landmarks are indicated by circles. The measurement region setting unit 13 defines a vector storing the coordinate values ​​of the detected landmarks as L(k). The measurement region setting unit 13 may detect facial features using a known method, such as using a model called a Constrained Local Model (CLM).

[0022] Next, the measurement area setting unit 13 sets the vertex coordinates of the quadrangle of the measurement area fi(k) based on the detected landmarks. For example, the measurement area setting unit 13 sets the vertex coordinates of a quadrangle as shown in Fig. 2C, and sets Rn measurement areas fi(k).

[0023] To explain this using an example in which the measurement area setting unit 13 sets the measurement area fi(k) in a portion of the skin area sr corresponding to the cheek, the measurement area setting unit 13 selects a landmark LA1 on the facial outline and a landmark LA2 on the nose. The measurement area setting unit 13 first selects the landmark LA2 on the nose, and then selects the landmark LA1 on the facial outline that is closest to the landmark LA2 on the nose. The measurement area setting unit 13 then sets auxiliary landmarks a1, a2, and a3 so as to divide the line segment between landmark LA1 and landmark LA2 into four equal parts.

[0024] Similarly, the measurement area setting unit 13 selects a landmark LB1 on the facial outline and a landmark LB2 on the nose. Furthermore, the measurement area setting unit 13 sets auxiliary landmarks b1, b2, and b3 so as to divide the line segment between the landmark LB1 and the landmark LB2 into four equal parts. Note that the landmarks LB1 and LB2 may be selected, for example, from landmarks on the facial outline or the nose adjacent to the landmarks LA1 and LA2, respectively.

[0025] The measurement area setting unit 13 sets a quadrilateral area surrounded by auxiliary landmarks a1, b1, b2, and a2 as one measurement area R1. The auxiliary landmarks a1, b1, b2, and a2 each have vertex coordinates corresponding to the measurement area R1. Similarly, the measurement area setting unit 13 sets one measurement area R2 surrounded by auxiliary landmarks a2, b2, b3, and a3 and the vertex coordinates of the measurement area R2.

[0026] Although an example of setting the measurement region fi(k) in the portion corresponding to the cheek has been described here, the measurement region setting unit 13 similarly sets the measurement region fi(k) and the vertex coordinates of the measurement region fi(k) for, for example, other portions of the cheek and the skin region sr in the portion corresponding to the chin. Although not shown in Fig. 2C, the measurement region setting unit 13 may set the measurement region fi(k) in the portion of the skin region sr of the driver corresponding to the forehead or the portion corresponding to the tip of the nose.

[0027] In this way, the measurement area setting unit 13 divides the skin area indicated by the skin area information S(k) into predetermined areas corresponding to the left and right cheeks of the driver DR, and sets measurement areas fi(k) (i=1, 2, 3, ..., n), as shown in Fig. 3A for example. In this case, the measurement area fi(k) is set to have, for example, a U-shape with respect to the skin area indicated by the skin area information S(k).

[0028] The measurement area setting unit 13 may appropriately determine where to set the measurement area depending on the driver's facial orientation. For example, when the driver is facing slightly to the right of the front, the measurement area setting unit 13 may set the measurement area fi(k) only in an area of ​​the skin area corresponding to the driver's left cheek (the area closer to the imaging device), as shown in FIG. 3B. In this case, the measurement area is set, for example, in a J-shape with respect to the skin area indicated by the skin area information S(k). The measurement area setting unit 13 may determine the driver's facial orientation using a known method. In the following description, the measurement area initially set by the measurement area setting unit 13 with respect to the skin area is also referred to as an "initial measurement area."

[0029] In addition, the measurement area setting unit 13 may determine the maximum number of measurement areas that can be set based on the acquired vehicle layout information and a predetermined angle range of the driver's face direction.

[0030] For example, when the driver DR faces slightly right of the front as shown in Fig. 3B and the imaging device captures an image of the driver DR from diagonally in front, the measurement area setting unit 13 may set the maximum number of measurement areas to, for example, 28. Then, the measurement area setting unit 13 may set 28 measurement areas as initial measurement areas for the skin area indicated by the skin area information S(k) when the imaging device actually captures an image of the driver DR from diagonally in front.

[0031] 3A, when the driver DR faces forward and the imaging device captures an image of the driver DR from the front, the measurement area setting unit 13 may set the maximum number of measurement areas to, for example, 40. Then, the measurement area setting unit 13 may set 40 measurement areas as initial measurement areas for the skin area indicated by the skin area information S(k) when the imaging device actually captures an image of the driver DR from the front.

[0032] After setting the initial measurement areas as described above, the measurement area setting unit 13 calculates the number of pixels included in each set initial measurement area. Then, among the multiple set initial measurement areas, the measurement area setting unit 13 calculates the ratio of initial measurement areas whose pixel count is equal to or greater than a predetermined reference value, and if the calculated ratio is less than a threshold, resets the measurement areas so that the ratio is equal to or greater than the threshold. Details of this resetting of the measurement areas will be described later.

[0033] The measurement region setting unit 13 outputs measurement region information R(k) indicating the set and reset measurement regions fi(k) to the luminance signal extraction unit 14.

[0034] Based on the measurement region information R(k) output from the measurement region setting unit 13, the luminance signal extraction unit 14 extracts a luminance signal indicating a change in luminance from each measurement region fi(k) indicated by the measurement region information R(k).

[0035] For example, the luminance signal extraction unit 14 calculates, for each measurement region, an average value of the luminance values ​​of pixels included in each measurement region fi(k) set by the measurement region setting unit 13 in the current frame. Then, the luminance signal extraction unit 14 sets information indicating the calculated average value of the luminance values ​​for each measurement region as luminance signal information Gi(k). Similarly, the luminance signal extraction unit 14 calculates, for each measurement region, an average value of the luminance values ​​of pixels included in each measurement region fi(k-1) set by the measurement region setting unit 13 in the frame immediately preceding the current frame. Then, the luminance signal extraction unit 14 sets information indicating the calculated average value of the luminance values ​​for each measurement region as luminance signal information Gi(k-1).

[0036] Then, the luminance signal extraction unit 14 calculates the difference between the luminance signal information Gi(k) and the luminance signal information Gi(k-1). Specifically, the luminance signal extraction unit 14 calculates the difference in average luminance values ​​between corresponding measurement areas between the luminance signal information Gi(k) and the luminance signal information Gi(k-1). At this time, if there is no corresponding measurement area between the luminance signal information Gi(k) and the luminance signal information Gi(k-1), the luminance signal extraction unit 14 does not calculate the difference for that measurement area.

[0037] Then, the luminance signal extraction unit 14 outputs information indicating the difference between the calculated luminance signal information Gi(k) and luminance signal information Gi(k-1) as luminance signal segment information W(t) to the luminance signal segment generation unit 15. At this time, the luminance signal extraction unit 14 includes information indicating the facial direction of the subject in the current frame (k) and information indicating the facial direction of the subject in the previous frame (k-1) in the luminance signal segment information W(t).

[0038] Here, t is a number corresponding to the time difference between the current frame (k) and the previous frame (k-1). For example, when luminance signal segment information indicating the difference between luminance signal information Gi(k) and luminance signal information Gi(k-1) is W(t), luminance signal segment information indicating the difference between luminance signal information Gi(k+1) and luminance signal information Gi(k) is W(t+1).

[0039] In the above description, the luminance signal extraction unit 14 has been described as taking, as an example, information indicating the average value of the calculated luminance values ​​for each measurement region as the luminance signal information Gi(k). However, the luminance signal extraction unit 14 is not limited to this, and may take, for example, information indicating the variance of the calculated luminance values ​​for each measurement region as the luminance signal information Gi(k).

[0040] The luminance signal segment generating unit 15 accumulates (stores) the luminance signal segment information W(t) output from the luminance signal extracting unit 14 in a storage unit such as a memory until it accumulates information for a predetermined time period required to estimate the pulse wave. As a result, the luminance signal segment generating unit 15 generates a time-series luminance signal (luminance signal segment) that represents the change in luminance value over time for each measurement region.

[0041] Then, the luminance signal segment generating unit 15 selects, from the generated time-series luminance signals for each measurement region, time-series luminance signals to be used for estimating a pulse wave by the pulse wave estimating unit 16, which will be described later. For example, from the generated time-series luminance signals for each measurement region, the luminance signal segment generating unit 15 selects time-series luminance signals in which the deviation between the face orientation of the driver DR in the current frame (k) and the previous frame (k-1) included in the luminance signal segment information W(t) and a predetermined reference face orientation is within a predetermined range (for example, a range from "-5 degrees" to "+5 degrees") as time-series luminance signals to be used for estimating the pulse wave of the subject.

[0042] The reason why the luminance signal segment generation unit 15 selects the time-series luminance signal in this way is as follows. Basically, in pulse wave estimation, the pulse wave (pulse rate) is detected by detecting luminance changes within the same measurement area. However, luminance changes can also occur due to factors other than blood flow. In particular, when the facial orientation of the driver DR, who is the subject, changes, the direction of light entering the face of the driver DR changes, resulting in a significant change in luminance. Therefore, in pulse wave estimation, it is desirable not to use luminance values ​​(luminance signals) when the facial orientation is significantly different from the reference facial orientation for pulse wave estimation. Therefore, the luminance signal segment generation unit 15 selects the time-series luminance signal to be used for estimating the subject's pulse wave, using the facial orientation of the driver DR as a condition.

[0043] Luminance signal segment generator 15 outputs information indicating the selected time-series luminance signal to pulse wave estimator 16 as segment information X(t).

[0044] Pulse wave estimation unit 16 estimates the subject's pulse wave based on segment information X(t) output from luminance signal segment generation unit 15. For example, pulse wave estimation unit 16 calculates one piece of pulse wave signal information by adding up the differences in the average values ​​of the luminance values ​​for each measurement region fi(k) included in segment information X(t) output from luminance signal segment generation unit 15. Pulse wave estimation unit 16 then performs a Fourier transform on the calculated pulse wave signal information and estimates the subject's pulse wave based on the peak frequency in the frequency power spectrum.

[0045] In the above description, an example has been described in which pulse wave estimation device 1 includes luminance signal segment generation unit 15, but luminance signal segment generation unit 15 is not an essential component and may be omitted. In this case, pulse wave estimation unit 16 accumulates (stores) luminance signal segment information W(t) output from luminance signal extraction unit 14 in a storage unit such as a memory until a predetermined amount of information necessary for estimating the pulse wave has been accumulated, and estimates the subject's pulse wave based on the accumulated information.

[0046] Next, an example of the operation of the pulse wave estimation device 1 shown in Fig. 1 will be described. Fig. 4 is a flowchart showing an example of the operation of the pulse wave estimation device 1.

[0047] First, the captured image acquisition unit 11 acquires a captured image of the driver DR who is the subject (step ST1). The captured image acquisition unit 11 outputs the acquired captured image to the skin region detection unit 12.

[0048] Next, the skin region detection unit 12 detects a skin region from a frame Im(k) included in the captured image acquired by the captured image acquisition unit 11 in step ST1 (step ST2). The skin region detection unit 12 generates skin region information S(k) indicating the detected skin region, and outputs the generated skin region information S(k) to the measurement region setting unit 13 together with the frame Im(k).

[0049] Next, the measurement area setting unit 13 divides the skin area indicated by the skin area information S(k) on the frame Im(k) into predetermined parts based on the frame Im(k), the skin area information S(k), and the vehicle layout information, and sets the areas obtained by the division as multiple measurement areas for extracting luminance signals indicating luminance changes (step ST3).

[0050] Here, the details of the processing by the measurement area setting unit 13 will be described with reference to the flowchart shown in FIG.

[0051] First, the measurement area setting unit 13 divides the skin area indicated by the skin area information S(k) on the frame Im(k) into predetermined parts, and sets the areas obtained by the division as a plurality of measurement areas (initial measurement areas) for extracting a luminance signal indicating a change in luminance (step ST31).

[0052] Next, the measurement area setting unit 13 calculates the number of pixels included in each of the set initial measurement areas. Then, the measurement area setting unit 13 calculates the ratio of the set initial measurement areas whose number of pixels is equal to or greater than a predetermined reference value, and determines whether the calculated ratio is equal to or greater than a threshold value (step ST32).

[0053] As a result, if the calculated ratio is equal to or greater than the threshold value (step ST32; YES), the measurement region setting unit 13 ends the process. Then, the process returns to step ST4 shown in Fig. 4. On the other hand, if the calculated ratio is not equal to or greater than the threshold value (step ST32; NO), the measurement region setting unit 13 resets the measurement region so that the ratio is equal to or greater than the threshold value (step ST33).

[0054] The reference value and threshold value may be set appropriately by, for example, an administrator of the pulse wave estimation device 1 depending on the accuracy required for pulse wave estimation. For example, the administrator of the pulse wave estimation device 1 sets the reference value and threshold value to be larger the higher the accuracy required for pulse wave estimation. For example, if the threshold value is set to "100%," the measurement area setting unit 13 will reset the measurement area if at least one of the set initial measurement areas has a pixel count less than a predetermined reference value. Also, if the threshold value is set to "50%, for example, the measurement area setting unit 13 will not reset the measurement area if half or more of the set initial measurement areas have a pixel count equal to or greater than the predetermined reference value.

[0055] The measurement area setting unit 13 resets the measurement area using, for example, the following three methods. (Method 1) For example, the measurement area setting unit 13 resets the measurement area by reducing the number of divisions of the skin area when the initial measurement area was set so that the ratio is equal to or greater than the threshold. For example, when 28 initial measurement areas are set as shown in Fig. 3B, if the ratio is not equal to or greater than the threshold, the measurement area setting unit 13 resets the measurement area by reducing the number of divisions of the skin area to 8, as shown in Fig. 6. In addition, at this time, the measurement area setting unit 13 increases the number of pixels per measurement area after resetting shown in Fig. 6 from the number of pixels per measurement area before resetting shown in Fig. 3B.

[0056] Here, when reducing the number of divisions of the skin region, the measurement region setting unit 13 may reduce the number of divisions of the skin region by merging some adjacent initial measurement regions among the multiple initial measurement regions shown in Fig. 3B. Alternatively, the measurement region setting unit 13 may discard the multiple initial measurement regions shown in Fig. 3B and re-divide the skin region into a smaller number of divisions than when the initial measurement region was set, thereby resetting the measurement region.

[0057] (Method 2) For example, the measurement area setting unit 13 resets the measurement areas by setting a new measurement area that spans a predetermined number of adjacent initial measurement areas among the multiple initial measurement areas so that the ratio is equal to or greater than a threshold. For example, when 28 initial measurement areas are set as shown in FIG. 3B, if the ratio is not equal to or greater than the threshold, the measurement area setting unit 13 resets the measurement areas by setting a new measurement area that spans two or four adjacent initial measurement areas among the 28 initial measurement areas, as shown in FIGS. 7A to 7D. In this example, 13 new measurement areas from fi(1) to fi(13) are set.

[0058] In addition, the 13 new measurement regions are allowed to overlap with one another. For example, measurement region fi(9) overlaps with measurement regions fi(1) and fi(3), and measurement region fi(10) overlaps with measurement regions fi(2) and fi(4).

[0059] Also in this case, the measurement area setting unit 13 increases the number of pixels per measurement area after resetting shown in FIGS. 7A to 7D compared to the number of pixels per measurement area before resetting shown in FIG. 3B.

[0060] (Method 3) For example, the measurement area setting unit 13 may combine Method 1 and Method 2. For example, the measurement area setting unit 13 may reset the measurement area by reducing the number of divisions of the skin area so that the ratio is equal to or greater than a threshold, and setting a new measurement area that spans a predetermined number of adjacent measurement areas from among the multiple measurement areas obtained by reducing the number of divisions. For example, of the eight measurement areas shown in FIG. 6, a new measurement area that spans two adjacent measurement areas may be set.

[0061] The measurement area setting unit 13 resets the measurement area using any one of the above methods 1 to 3. After that, the process returns to step ST4 shown in FIG.

[0062] Next, a description will be given of the effects of the pulse wave estimation device 1 according to embodiment 1. As described above, the pulse wave estimation device 1 can reduce the deterioration of pulse wave estimation accuracy more than ever before by resetting the measurement region.

[0063] For example, in the conventional pulse wave estimation method, if the resolution of the image of the driver DR is reduced due to a change in the installation layout of the imaging device that images the driver DR, who is the subject, or if the imaging device itself is changed, the resolution is reduced and the estimation accuracy of the pulse wave may be reduced. Also, in the conventional pulse wave estimation method, if the driver DR moves the face of the driver DR away from the imaging device by lowering the vehicle seat backward or changing the direction of his / her face, the resolution of the image of the driver DR is reduced and the same problem may occur.

[0064] In contrast, in pulse wave estimation device 1, if the proportion of measurement areas in which the number of pixels is equal to or greater than a predetermined reference value among the multiple measurement areas set by measurement area setting unit 13 is less than a threshold, the measurement areas are dynamically reset so that the proportion becomes equal to or greater than the threshold. As a result, in pulse wave estimation device 1, the proportion of measurement areas in which the number of pixels is equal to or greater than the reference value after resetting becomes equal to or greater than the threshold, thereby suppressing a decrease in resolution and a decrease in pulse wave estimation accuracy.

[0065] Furthermore, when the measurement area setting unit 13 employs the above method 2 in the pulse wave estimation device 1, the following effect is also achieved. For example, when the measurement area setting unit 13 resets the measurement area by reducing the number of divisions of the skin area as in the above method 1, if localized external light is reflected in part of the reset measurement area, that measurement area cannot be used to estimate the pulse wave, and therefore the measurement area that can be used to estimate the pulse wave is reduced, which may reduce robustness.

[0066] In this regard, in the above-mentioned method 2, the measurement area setting unit 13 resets a new measurement area spanning a predetermined number of adjacent initial measurement areas while allowing them to overlap with each other, so that it is possible to ensure the size of the measurement area (the number of pixels in the measurement area) while also ensuring the number of measurement areas that can be used to estimate the pulse wave, and it is possible to suppress a decrease in robustness compared to method 1.

[0067] Furthermore, in the pulse wave estimation device 1, when the measurement area setting unit 13 employs the above-mentioned method 3, the size of the measurement area after resetting (the number of pixels in the measurement area) can be increased compared to when method 1 or method 2 is employed alone, and the decrease in robustness that is a problem with method 1 can also be suppressed.

[0068] Next, an example of the hardware configuration of pulse wave estimation device 1 according to embodiment 1 will be described with reference to Fig. 8. The functions of captured image acquisition unit 11, skin region detection unit 12, measurement region setting unit 13, luminance signal extraction unit 14, luminance signal segment generation unit 15, and pulse wave estimation unit 16 in pulse wave estimation device 1 are realized by processing circuits. The processing circuit may be dedicated hardware as shown in Fig. 8A, or may be a CPU (also referred to as a central processing unit, processing unit, arithmetic unit, microprocessor, microcomputer, processor, or DSP (digital signal processor)) 52 that executes a program stored in memory 53 as shown in Fig. 14B.

[0069] When the processing circuit is dedicated hardware, the processing circuit 51 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination thereof. The functions of each of the captured image acquisition unit 11, the skin region detection unit 12, the measurement region setting unit 13, the luminance signal extraction unit 14, the luminance signal segment generation unit 15, and the pulse wave estimation unit 16 may be realized individually by the processing circuit 51, or the functions of each unit may be realized collectively by the processing circuit 51.

[0070] When the processing circuit is a CPU 52, the functions of the captured image acquisition unit 11, skin area detection unit 12, measurement area setting unit 13, luminance signal extraction unit 14, luminance signal segment generation unit 15, and pulse wave estimation unit 16 are realized by software, firmware, or a combination of software and firmware. The software and firmware are written as programs and stored in memory 53. The processing circuit realizes the functions of each unit by reading and executing the programs stored in memory 53. In other words, the pulse wave estimation device 1 includes a memory for storing programs that, when executed by the processing circuit, result in the execution of, for example, each step shown in FIG. 4. These programs can also be said to cause a computer to execute the procedures and methods of the captured image acquisition unit 11, skin area detection unit 12, measurement area setting unit 13, luminance signal extraction unit 14, luminance signal segment generation unit 15, and pulse wave estimation unit 16. Here, the memory 53 may be, for example, a RAM (R a Examples of such memory devices include non-volatile or volatile semiconductor memory such as Random Access Memory (RANDOM), ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable ROM), EEPROM (Electrically EPROM), magnetic disks, flexible disks, optical disks, compact disks, minidisks, and DVDs (Digital Versatile Discs).

[0071] Note that the functions of captured image acquisition unit 11, skin region detection unit 12, measurement region setting unit 13, luminance signal extraction unit 14, luminance signal segment generation unit 15, and pulse wave estimation unit 16 may be partially implemented by dedicated hardware and partially implemented by software or firmware. For example, the functions of captured image acquisition unit 11 may be implemented by a processing circuit as dedicated hardware, and the functions of skin region detection unit 12, measurement region setting unit 13, luminance signal extraction unit 14, luminance signal segment generation unit 15, and pulse wave estimation unit 16 may be implemented by the processing circuit reading and executing programs stored in memory 53.

[0072] Thus, the processing circuitry can implement each of the above-described functions by hardware, software, firmware, or a combination thereof.

[0073] As described above, according to this first embodiment, pulse wave estimation device 1 includes captured image acquisition unit 11 that acquires a captured image of a person, skin region detection unit 12 that detects the person's skin region from the captured image, measurement region setting unit 13 that sets regions obtained by dividing the skin region in the captured image into predetermined sections as multiple measurement regions from which luminance signals indicating luminance changes are extracted, luminance signal extraction unit 14 that extracts luminance signals based on luminance changes in the measurement regions on the captured image, and pulse wave estimation unit 16 that estimates the person's pulse wave based on the extracted luminance signals. If the proportion of the measurement regions that have a pixel count equal to or greater than a predetermined reference value is less than a threshold, measurement region setting unit 13 resets the measurement regions so that the proportion is equal to or greater than the threshold. This allows pulse wave estimation device 1 according to the first embodiment to prevent a decrease in pulse wave estimation accuracy.

[0074] Furthermore, if the ratio is less than the threshold, measurement region setting unit 13 resets the measurement region by reducing the number of divisions of the skin region in the captured image, thereby enabling pulse wave estimation device 1 according to embodiment 1 to easily reset the measurement region.

[0075] Furthermore, if the ratio is less than the threshold value, measurement region setting unit 13 resets the measurement region by setting a new measurement region that spans a predetermined number of adjacent measurement regions among the multiple measurement regions that have been set. This allows pulse wave estimation device 1 according to embodiment 1 to easily reset the measurement region.

[0076] Furthermore, multiple new measurement regions can be set, and they can overlap with each other. This allows the pulse wave estimation device 1 according to the first embodiment to ensure the size of the measurement region while also ensuring the number of measurement regions that can be used to estimate the pulse wave.

[0077] Furthermore, if the ratio is less than the threshold, the measurement region setting unit 13 resets the measurement region by reducing the number of divisions of the skin region in the captured image and setting a new measurement region that spans a predetermined number of adjacent measurement regions from among the multiple measurement regions obtained by reducing the number of divisions. This allows the pulse wave estimation device 1 according to the first embodiment to increase the size of the measurement region after resetting.

[0078] Furthermore, measurement region setting unit 13 increases the number of pixels per measurement region after resetting compared to the number of pixels per measurement region before resetting, thereby enabling pulse wave estimation device 1 according to embodiment 1 to suppress a decrease in resolution and a decrease in pulse wave estimation accuracy.

[0079] Pulse wave estimation device 1 also includes luminance signal segment generation unit 15 that selects luminance signals to be used for estimating the person's pulse wave from among the luminance signals extracted by luminance signal extraction unit 14, and pulse wave estimation unit 16 estimates the person's pulse wave based on the luminance signals selected by luminance signal segment generation unit 15. This allows pulse wave estimation device 1 according to embodiment 1 to effectively prevent a decrease in pulse wave estimation accuracy.

[0080] Furthermore, luminance signal segment generation unit 15 selects, from among the luminance signals extracted by luminance signal extraction unit 14, a luminance signal whose deviation from a predetermined reference face direction at the time the luminance signal was extracted is within a predetermined range, as a luminance signal to be used for estimating the person's pulse wave. This allows pulse wave estimation device 1 according to the first embodiment to select a time-series luminance signal to be used for estimating the pulse wave, using the person's face direction as a condition.

[0081] Although the preferred embodiments have been described above in detail, the present invention is not limited to the above-described embodiments. The above-described embodiments may be modified without departing from the scope of the claims. Various modifications and substitutions can be made to the above.

[0082] Various aspects of the present disclosure are summarized below as appendices.

[0083] (Appendix 1) a captured image acquisition unit that acquires a captured image of a person; a skin area detection unit that detects a skin area of ​​the person from the captured image; a measurement area setting unit that sets areas obtained by dividing the skin area in the captured image into predetermined parts as a plurality of measurement areas for extracting a luminance signal indicating a luminance change; a luminance signal extraction unit that extracts the luminance signal based on a luminance change in the measurement area on the captured image; a pulse wave estimation unit that estimates a pulse wave of the person based on the extracted luminance signal, The measurement area setting unit When the ratio of the measurement areas having a pixel count equal to or greater than a predetermined reference value among the plurality of measurement areas that have been set is less than a threshold value, the measurement areas are reset so that the ratio is equal to or greater than the threshold value. (Appendix 2) The measurement area setting unit If the ratio is less than the threshold, the measurement area is reset by reducing the number of divisions of the skin area in the captured image. 2. The pulse wave estimation device according to claim 1, (Appendix 3) The measurement area setting unit If the ratio is less than a threshold value, the measurement area is reset by setting a new measurement area that spans a predetermined number of adjacent measurement areas among the set measurement areas. 2. The pulse wave estimation device according to claim 1, (Appendix 4) 4. The pulse wave estimation device according to claim 3, wherein a plurality of the new measurement regions can be set and can overlap with each other. (Appendix 5) The measurement area setting unit If the ratio is less than the threshold value, the number of divisions of the skin region in the captured image is reduced, and a new measurement region that spans a predetermined number of adjacent measurement regions is set among the multiple measurement regions obtained by reducing the number of divisions, thereby resetting the measurement region. 2. The pulse wave estimation device according to claim 1, (Appendix 6) The measurement area setting unit The number of pixels per measurement area after the resetting is increased to be greater than the number of pixels per measurement area before the resetting. 6. The pulse wave estimation device according to claim 1, wherein: (Appendix 7) a luminance signal segment generation unit that selects a luminance signal to be used for estimating a pulse wave of the person from among the luminance signals extracted by the luminance signal extraction unit, The pulse wave estimation unit estimates the pulse wave of the person based on the luminance signal selected by the luminance signal segment generation unit. 7. The pulse wave estimation device according to claim 1, wherein: (Appendix 8) The luminance signal segment generation unit Among the luminance signals extracted by the luminance signal extraction unit, a luminance signal in which a deviation between the face orientation of the person when the luminance signal was extracted and a predetermined reference face orientation is within a predetermined range is selected as a luminance signal to be used for estimating the pulse wave of the person. 8. The pulse wave estimation device according to claim 7, (Appendix 9) A pulse wave estimation method using a pulse wave estimation device, comprising: A captured image acquisition unit acquires a captured image of a person; a step of detecting a skin region of the person from the captured image by a skin region detection unit; a measurement area setting unit setting areas obtained by dividing the skin area in the captured image into predetermined parts as a plurality of measurement areas for extracting a luminance signal indicating a luminance change; a luminance signal extraction unit extracting the luminance signal based on a luminance change in the measurement region on the captured image; a pulse wave estimation unit estimating a pulse wave of the person based on the extracted luminance signal; The measurement area setting unit a pulse wave estimation method, characterized in that, when a ratio of the measurement areas having a pixel count equal to or greater than a predetermined reference value among the plurality of measurement areas that have been set is less than a threshold value, the measurement areas are reset so that the ratio is equal to or greater than the threshold value. [Industrial Applicability]

[0084] The present disclosure is suitable for use in a pulse wave estimation device, as it is possible to suppress a decrease in the accuracy of estimating a pulse wave. [Explanation of symbols]

[0085] 1 Pulse wave estimation device, 11 Image acquisition unit, 12 Skin area detection unit, 13 Measurement area setting unit, 14 Luminance signal extraction unit, 15 Luminance signal segment generation unit, 16 Pulse wave estimation unit, 51 Processing circuit, 52 CPU, 53 Memory, a1 to a3 Auxiliary landmarks, b1 to b3 Auxiliary landmarks, DR Driver, fi Measurement area, LA1 to LA2 Landmarks, LB1 to LB2 Landmarks, sr Skin area.

Claims

1. a captured image acquisition unit that acquires a captured image of a person; a skin area detection unit that detects a skin area of ​​the person from the captured image; a measurement area setting unit that sets areas obtained by dividing the skin area in the captured image into predetermined parts as a plurality of measurement areas for extracting a luminance signal indicating a luminance change; a luminance signal extraction unit that extracts the luminance signal based on a luminance change in the measurement area on the captured image; a pulse wave estimation unit that estimates a pulse wave of the person based on the extracted luminance signal, The measurement area setting unit If the ratio of the measurement areas having a pixel count equal to or greater than a predetermined reference value among the plurality of measurement areas that have been set is less than a threshold value, the measurement areas are reset so that the ratio becomes equal to or greater than the threshold value, and the number of pixels per measurement area after the resetting is increased compared to the number of pixels per measurement area before the resetting. A pulse wave estimation device characterized by:

2. The measurement area setting unit If the ratio is less than the threshold, the measurement area is reset by reducing the number of divisions of the skin area in the captured image.

2. The pulse wave estimation device according to claim 1.

3. The measurement area setting unit If the ratio is less than a threshold value, the measurement area is reset by setting a new measurement area that spans a predetermined number of adjacent measurement areas among the set measurement areas.

2. The pulse wave estimation device according to claim 1.

4. 4. The pulse wave estimation device according to claim 3, wherein a plurality of new measurement regions can be set, and the new measurement regions can overlap with each other.

5. The measurement area setting unit If the ratio is less than the threshold value, the number of divisions of the skin region in the captured image is reduced, and a new measurement region that spans a predetermined number of adjacent measurement regions is set among the multiple measurement regions obtained by reducing the number of divisions, thereby resetting the measurement region.

2. The pulse wave estimation device according to claim 1.

6. a luminance signal segment generation unit that selects, from the luminance signals extracted by the luminance signal extraction unit, a luminance signal in which a deviation between the face orientation of the person at the time the luminance signal was extracted and a predetermined reference face orientation is within a predetermined range, as a luminance signal to be used for estimating the pulse wave of the person; The pulse wave estimation unit estimates the pulse wave of the person based on the luminance signal selected by the luminance signal segment generation unit.

2. The pulse wave estimation device according to claim 1.

7. A pulse wave estimation method using a pulse wave estimation device, comprising: A captured image acquisition unit acquires a captured image of a person; a step of detecting a skin region of the person from the captured image by a skin region detection unit; a measurement area setting unit setting areas obtained by dividing the skin area in the captured image into predetermined parts as a plurality of measurement areas for extracting a luminance signal indicating a luminance change; a luminance signal extraction unit extracting the luminance signal based on a luminance change in the measurement region on the captured image; a pulse wave estimation unit estimating a pulse wave of the person based on the extracted luminance signal; The measurement area setting unit When the proportion of the measurement areas having a pixel count equal to or greater than a predetermined reference value among the plurality of measurement areas that have been set is less than a threshold value, the measurement areas are reset so that the proportion becomes equal to or greater than the threshold value, and the number of pixels per measurement area after the resetting is increased compared to the number of pixels per measurement area before the resetting.

Images