Pulse wave inference device, and pulse wave inference method

Abstract

This pulse wave inference device comprises: a captured image acquisition unit (11) that acquires a captured image; a skin region detection unit (12) that detects a skin region of a person from the captured image; a measurement region setting unit (14) that sets, in a region corresponding to the skin region on the captured image, a measurement region that can be used for extracting a pulse wave source signal including a pulse wave component of the person; a face direction inference unit (13) that infers the direction of the face of the person on the basis of the captured image; a luminance signal extraction unit (15, 15a) that sets a use measurement region and extracts time series pulse wave source signals on the basis of a luminance change in the set use measurement region; a luminance signal selection unit (16, 16a, 16b) that selects a time series inference pulse wave source signal to be used for inferring the pulse wave of the person while taking into consideration the direction of the face of the person inferred by the face direction inference unit (13); and a pulse wave inference unit (17) that infers the pulse wave of the person on the basis of the time series inference pulse wave source signal selected by the luminance signal selection unit (16, 16a, 16b).

Description

Pulse wave estimation device and pulse wave estimation method

[0001] The present disclosure relates to a pulse wave estimation device and a pulse wave estimation method.

[0002] Conventionally, there has been known a technique for estimating a person's pulse wave from minute changes in luminance on the surface of the person's skin, based on time-series luminance signals extracted from multiple regions (hereinafter referred to as "measurement regions") set in a region including the person's skin (hereinafter referred to as "skin region") in an image captured by an imaging device (for example, Patent Document 1).

[0003] JP 2017-93760 A

[0004] However, conventional pulse wave estimation techniques, such as that disclosed in Patent Literature 1, do not take into account the possibility that, depending on the facial orientation of the person whose pulse wave is to be estimated, the skin area may not be captured in the captured image, or a shadow may be cast on the skin area, a phenomenon known as shadow cast. If the skin area is not captured in the captured image, or if a shadow cast occurs, the luminance signal extracted from the skin area may not contain enough pulse wave components to estimate the person's pulse wave. Therefore, conventional techniques have had the problem of a potential reduction in pulse wave estimation accuracy due to the skin area of ​​the person whose pulse wave is to be estimated not being captured in the captured image, or if a shadow cast occurs on the skin area.

[0005] The present disclosure has been made to solve the above-mentioned problems, and aims to provide a pulse wave estimation device that prevents a decrease in the accuracy of estimating a person's pulse wave due to, for example, the skin area of ​​the person from which the pulse wave is to be estimated not being captured in the captured image, or the occurrence of so-called shadow cast on that skin area.

[0006] a measurement area setting unit that sets a measurement area that can be used for extracting a pulse wave source signal that indicates a change in luminance and that includes a pulse wave component of the person in an area corresponding to the skin area on the captured image; a face direction estimation unit that estimates the person's facial direction on a frame-by-frame basis based on the captured image; a luminance signal extraction unit that sets a usable measurement area to be used for extracting the pulse wave source signal from the measurement areas set by the measurement area setting unit and extracts a time-series pulse wave source signal based on the change in luminance in the set usable measurement area; a luminance signal selection unit that selects a time-series estimation pulse wave source signal to be used for estimating the person's pulse wave from the time-series pulse wave source signals extracted by the luminance signal extraction unit during a pulse wave estimation period, taking into account the person's facial direction estimated by the face direction estimation unit; and a pulse wave estimation unit that estimates the person's pulse wave based on the time-series estimation pulse wave source signal selected by the luminance signal selection unit.

[0007] According to the present disclosure, it is possible to prevent a decrease in the accuracy of estimating a person's pulse wave due to, for example, the skin area of ​​the person from which the pulse wave is to be estimated not being captured in the captured image, or a shadow being cast on the skin area.

[0008] 1 is a diagram illustrating an example of the configuration of a pulse wave estimation device according to Embodiment 1. FIGS. 2A and 2B are diagrams illustrating an example of the contents of reference measurement region information according to Embodiment 1. FIGS. 3A and 3B are diagrams illustrating an example of the relationship between the position and orientation of the imaging device and the position and orientation of the assumed subject according to Embodiment 1. FIGS. 4A and 4B are diagrams illustrating an example of a set region determined by the measurement region setting unit according to Embodiment 1. FIGS. 5A and 5B are diagrams illustrating another example of the relationship between the position and orientation of the imaging device and the position and orientation of the assumed subject according to Embodiment 1. FIGS. 6A and 6B are diagrams illustrating another example of a set region determined by the measurement region setting unit according to Embodiment 1. FIGS. 7A, 7B, and 7C are diagrams illustrating an example of a method for setting a measurement region by the measurement region setting unit in a pulse wave estimation device according to Embodiment 1. FIGS. 1A, 1B, and 1C are diagrams illustrating an example of a histogram illustrating a distribution ratio of the facial orientation in the yaw direction of the subject corresponding to each frame of a captured image during a pulse wave estimation target period, as calculated by a luminance signal selecting unit, estimated by a facial orientation estimating unit. 15 is a flowchart illustrating the operation of the pulse wave estimation device according to embodiment 1. FIG. 15 is a flowchart illustrating the details of the processing of step ST5 in FIG. 9 . FIG. 15 is a flowchart illustrating the details of the processing of step ST6 in FIG. 9 . FIGS. 12A and 12B are diagrams illustrating an example of the hardware configuration of the pulse wave estimation device according to embodiment 1. FIG. 15 is a diagram illustrating an example of the configuration of a pulse wave estimation device according to embodiment 2. FIG. 15 is a diagram illustrating an example of a histogram showing the distribution ratio of the used measurement region from which the pulse wave source signal extracted by the luminance signal extraction unit was extracted during the pulse wave estimation target period, as calculated by the luminance signal selection unit, in embodiment 2. FIG. 15 is a flowchart illustrating the operation of the pulse wave estimation device according to embodiment 2. FIG. 15 is a flowchart illustrating the details of step ST6a in FIG. 15 . FIG. 15 is a diagram illustrating an example of the configuration of a pulse wave estimation device according to embodiment 3.19 is a diagram illustrating an example of a weighting coefficient for each measurement region that is set by a weight setting unit based on the face direction of the subject estimated on a frame-by-frame basis by a face direction estimation unit in embodiment 3.

[0034] FIG. 19 is a flowchart illustrating the operation of the pulse wave estimation device according to embodiment 3.

[0035] FIG. 19 is a flowchart illustrating details of step ST5a in FIG. 19.

[0036] FIG. 19 is a flowchart illustrating details of step ST6b in FIG.

[0009] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. EMBODIMENT 1 Fig. 1 is a diagram showing an example of the configuration of a pulse wave estimation device 1 according to a first embodiment.

[0010] The pulse wave estimation device 1 estimates the pulse wave of a person based on an image captured by the imaging device 2. In the following first embodiment, the person whose pulse wave is estimated by the pulse wave estimation device 1 is referred to as the "subject." The pulse wave estimation device 1 acquires a series of frames Im(k) captured at a predetermined frame rate Fr, capturing at least a range where the subject's skin region (hereinafter referred to as the "skin region") should be present. Here, k indicates a frame number assigned to each frame. For example, the frame provided next after frame Im(k) is frame Im(k+1). In the first embodiment, the skin region corresponds to the subject's face. Note that this is merely an example, and the skin region may be a region other than the subject's face. For example, the skin region may correspond to facial features such as the subject's eyes, eyebrows, nose, mouth, forehead, cheeks, or chin. The skin region may also be a region corresponding to a body part other than the face, such as the subject's head, shoulders, hands, neck, or feet. The skin region may be multiple regions.

[0011] The pulse wave estimation device 1 estimates the subject's pulse wave from a series of frames Im(k-Tp+1) to Im(k) every certain number of frames Tp and outputs a pulse wave estimation result P(t), which is information indicating the estimated pulse wave (hereinafter referred to as "pulse wave information"). Specifically, the pulse wave estimation device 1 estimates the subject's pulse wave from a luminance signal based on changes in luminance of the subject's skin region in the series of frames Im(k-Tp+1) to Im(k). In the first embodiment, the series of frames Im(k-Tp+1) to Im(k) are assumed to be captured images acquired by the imaging device 2 during a predetermined period (hereinafter referred to as the "pulse wave estimation target period"). The pulse wave estimation target period is set, for example, by an administrator or the like as the period during which one pulse wave estimation is performed.

[0012] Here, t represents an output number assigned to each specific number of frames Tp. For example, the pulse wave estimation result given at the timing following pulse wave estimation result P(t) is pulse wave estimation result P(t+1). Frame number k and output number t are integers equal to or greater than 1. Frame number Tp is an integer equal to or greater than 2.

[0013] Depending on the subject's facial orientation, the subject's skin area may not be adequately captured in the captured image captured by the imaging device 2. Examples of insufficient capture of the subject's skin area include the subject's skin not being captured in the captured image captured by the imaging device 2, or a shadow being cast on the skin area (so-called shadow cast). For example, when the imaging device 2 captures a subject from the front and the subject's face is facing 90 degrees to the left or right relative to the optical axis of the imaging device 2, the right or left side of the subject's face may not be captured or may be cast in shadow. In this case, all or part of the subject's skin area may not be detected, and the luminance signal based on the luminance change extracted from the skin area may not adequately contain the subject's pulse wave component. As a result, the accuracy of estimating a person's pulse wave may be reduced. Therefore, pulse wave estimation device 1 considers the subject's facial orientation to select from which region of the skin a luminance signal based on luminance changes that can be extracted from the subject's skin region to use as a luminance signal for estimating the subject's pulse wave. Taking the subject's facial orientation into consideration, pulse wave estimation device 1 selects, as the luminance signal to use for estimating the subject's pulse wave, a luminance signal extracted from a region from which it is assumed possible to extract a luminance signal that sufficiently contains the subject's pulse wave component. Pulse wave estimation device 1 then estimates the subject's pulse wave from the luminance signal selected taking the subject's facial orientation into consideration.

[0014] The number of subjects included in the captured image may be one or more. For simplicity of explanation, in the following embodiment 1, the number of subjects included in the captured image will be described as one.

[0015] The pulse wave estimation result P(t) is output from the pulse wave estimation device 1 to, for example, an arousal level estimation device that estimates a person's arousal level or an abnormality detection device that detects a person's abnormal physical condition. Note that the arousal level estimation device and the abnormality detection device are not shown in FIG. 1 . The arousal level estimation device estimates a decrease in the subject's arousal level based on the pulse wave estimation result P(t) output from the pulse wave estimation device 1. For example, the arousal level estimation device estimates that the subject's arousal level is decreasing when the subject's pulse rate tends to decrease slowly. If the arousal level estimation device estimates that the subject's arousal level is decreasing, it warns the subject or people around them of the decrease in the subject's arousal level. The abnormality detection device detects an abnormal physical condition of the subject based on the pulse wave estimation result P(t) output from the pulse wave estimation device 1. The abnormal physical condition of the subject may be, for example, epilepsy or heart disease. For example, the abnormality detection device detects that the subject is in an abnormal physical condition when the subject's pulse wave increases rapidly. When the abnormality detection device detects an abnormality in the physical condition of the subject, it issues a warning to the subject or people around the subject that the subject is in an abnormal physical condition.

[0016] The imaging device 2 includes an imaging section (not shown) and an illumination section (not shown). The illumination section is configured, for example, with an LED (Light Emitting Diode). The illumination section irradiates light onto the imaging range of the imaging section. The imaging section captures an image of the imaging range illuminated by light emitted from the illumination section. Note that the imaging device 2 may be provided with one or more illumination sections. The imaging device 2 is installed so as to be able to capture an image of the subject's skin area. That is, in this example, the imaging device 2 is installed so as to be able to capture an image of the driver's skin area.

[0017] In the following first embodiment, as an example, pulse wave estimation device 1, image capture device 2, a wakefulness estimation device (not shown), and an abnormality detection device (not shown) are assumed to be mounted on a vehicle (not shown), and the subject is the driver of the vehicle. In other words, pulse wave estimation device 1 estimates the pulse wave of the vehicle driver.

[0018] A detailed description will be given of an example configuration of the pulse wave estimation device 1 shown in Fig. 1. As shown in Fig. 1, the pulse wave estimation device 1 includes an image acquisition unit 11, a skin region detection unit 12, a face direction estimation unit 13, a measurement region setting unit 14, a luminance signal extraction unit 15, a luminance signal selection unit 16, a pulse wave estimation unit 17, and an output unit 18.

[0019] The captured image acquisition unit 11 acquires a captured image of the subject. Specifically, the captured image acquisition unit 11 acquires a captured image of the vehicle driver captured by the imaging device 2. The captured image acquisition unit 11 outputs the acquired captured image to the skin region detection unit 12 and the face direction estimation unit 13.

[0020] The skin area detection unit 12 detects the subject's skin area from a frame Im(k) included in the captured image acquired by the captured image acquisition unit 11. The skin area detection unit 12 may detect the skin area using a known means. For example, the skin area detection unit 12 may detect the skin area using a cascade face detector using Haar-like features. The skin area detection unit 12 generates skin area information S(k) indicating the detected skin area. The skin area information S(k) may include information indicating whether a skin area has been detected and information indicating the position and size of the detected skin area on the captured image. In the first embodiment, the skin area is represented by a rectangular area on the captured image, and the skin area information S(k) includes information indicating the position and size of the rectangular area on the captured image. Specifically, when the skin region corresponds to the subject's face, the skin region information S(k) indicates, for example, whether the subject's face is detected, the center coordinates Fc (Fcx, Fcy) of a rectangle surrounding the subject's face in the captured image, and the width Fcw and height Fch of the rectangle. The presence or absence of the subject's face is represented, for example, by "1" if the face is detected and "0" if the face is not detected. The center coordinates of the rectangle surrounding the face are expressed in the coordinate system of frame Im(k). The upper left corner of frame Im(k) is the origin, the rightward direction of frame Im(k) is the positive direction of the x-axis, and the downward direction of frame Im(k) is the positive direction of the y-axis. The skin region detection unit 12 outputs the generated skin region information S(k) to the measurement region setting unit 14.

[0021] The face direction estimation unit 13 estimates the face direction of the subject frame by frame, based on the frame Im(k) of the captured image acquired by the captured image acquisition unit 11. In the first embodiment, the face direction of the subject estimated by the face direction estimation unit 13 is an angle calculated using the front position of the subject, i.e., the driver in this case, as a reference (0 degrees), regardless of the installation position of the imaging device 2. Note that in the first embodiment, "front" does not necessarily mean strictly front, but also includes approximately front. In the first embodiment, the face direction of the subject estimated by the face direction estimation unit 13 with the front of the subject as a reference is also referred to as a "subject-referenced face direction." Note that the face direction estimation unit 13 can estimate the three directions of the subject's face direction: yaw [deg], pitch [deg], and roll [deg]. The face direction estimation unit 13 may estimate the face direction of the subject, i.e., the subject-referenced face direction, using a known, trained model (hereinafter referred to as a "machine learning model"), such as Hope-Net, for example. The face direction estimation unit 13 generates information F(k) relating to the estimated face direction of the subject (subject-based face direction) (hereinafter referred to as "face direction information"). The face direction information F(k) is information in which the estimated subject-based face direction is associated with the frame Im(k) of the captured image from which the face direction was estimated. The face direction estimation unit 13 outputs the generated face direction information F(k) to the luminance signal extraction unit 15.

[0022] Based on the frame Im(k) of the captured image acquired by the captured image acquisition unit 11 and the skin area information S(k) output by the skin area detection unit 12, the measurement area setting unit 14 sets, in the image area of ​​the frame Im(k) corresponding to the skin area indicated by the skin area information S(k), multiple measurement areas that can be used to extract a luminance signal indicating a luminance change, the luminance signal including the subject's pulse wave component. In the following description, the luminance signal indicating a luminance change extracted from the multiple measurement areas is also referred to as a "pulse wave source signal." The area of ​​the image area corresponding to the skin area to be used as the measurement area is determined based on the range of the subject's facial orientation (hereinafter referred to as the "target facial orientation range") for which the pulse wave is to be estimated and the expected positional relationship between the image capture device 2 and the subject. The target facial orientation range is appropriately determined by an administrator or the like. The administrator or the like determines the target facial orientation range depending on the intended use of the pulse wave estimation device 1, etc. The administrator or the like stores information indicating the target face direction range in a location that can be referenced by the measurement area setting unit 14, such as a storage unit (not shown).

[0023] For example, an administrator or the like may generate information (hereinafter referred to as "reference measurement area information") defining multiple measurement areas from which a pulse wave source signal containing the subject's pulse wave component can be extracted when the imaging device 2 images the subject at a reference position (hereinafter referred to as the "reference position") and orientation (hereinafter referred to as the "reference orientation"), and the information is stored in a location accessible by the pulse wave estimation device 1, such as a storage unit. The reference measurement area information is table-format information that associates information indicating the measurement area with the subject's facial orientation in the captured image when the imaging device 2 images the subject at the reference position and orientation. The information indicating the measurement area is, for example, a number (hereinafter referred to as the "measurement area number") that can identify the measurement area. Each measurement area is assigned a measurement area number. The subject is assumed to be a subject with a standard build (hereinafter referred to as the "expected subject") facing forward in an expected standard position. Here, the assumed subject is, for example, a driver with a standard physique who sits facing forward in a driver's seat set in a standard position with a standard posture without losing posture. In the first embodiment, when the subject (assumed subject) is imaged at the reference position and reference orientation defined in the reference measurement area information, the facial orientation of the subject (assumed subject) in the captured image is an angle calculated with the front of the imaging device 2 as the reference (0 degrees). In the first embodiment, the facial orientation of the subject (assumed subject) based on the front of the imaging device 2 as defined in the reference measurement area is also referred to as the "camera-based facial orientation."

[0024] Furthermore, information indicating the positional relationship between the actually installed imaging device 2 and the expected subject (hereinafter referred to as "layout information") is generated in advance by an administrator or the like and stored in a storage unit or the like. For example, when installing the imaging device 2, the administrator or the like generates layout information taking into consideration the position of the expected subject and stores it in a storage unit or the like.

[0025] That is, the information indicating the target face direction range, the reference measurement area information, and the layout information are generated in advance by an administrator or the like and stored in a storage unit or the like.

[0026] The measurement area setting unit 14 sets multiple measurement areas based on the multiple measurement areas defined in the reference measurement area information, based on the layout information and information indicating the target face direction range. Specifically, the measurement area setting unit 14 corrects the camera-based face direction defined in the reference measurement area information to a subject-based face direction based on the layout information, and sets multiple measurement areas based on the target face direction range, based on the multiple measurement areas defined in the reference measurement area information. More specifically, the measurement area setting unit 14 calculates a difference between the position and orientation of the image capture device 2 and a reference position and reference orientation based on the layout information, and offsets the face direction of the assumed subject (camera-based face direction) in the reference measurement area information based on the calculated difference, thereby aligning the face direction of the assumed subject (camera-based face direction) in frame Im(k) of the captured image captured at the reference position and reference orientation of the image capture device 2 assumed in the reference measurement area information with the face direction of the assumed subject (subject-based face direction) in frame Im(k) of the captured image captured at the current position and orientation of the image capture device 2. Then, the measurement area setting unit 14 determines a plurality of measurement areas to be set from among the plurality of measurement areas defined in the reference measurement area information, based on the target face direction range.

[0027] Here, a specific example will be described of a method in which the measurement area setting unit 14 determines a plurality of measurement areas to be set based on the layout information, the information indicating the target face direction range, and the reference measurement area information. For convenience, in the following specific example, it is assumed that the subject changes the face direction only in the yaw direction.

[0028] First, details of the reference measurement area information, which is the premise, will be described using drawings. As described above, it is assumed here that the subject changes their facial orientation only in the yaw direction, and the reference measurement area information is information in which the subject's facial orientation in the yaw direction is associated with information indicating the measurement area. FIG. 2 is a diagram for explaining an example of the content of the reference measurement area information in embodiment 1. FIG. 2 is a diagram for explaining an example of the content of the reference measurement area information when the subject's facial orientation in the yaw direction from "-30 degrees to +30 degrees" is associated with information indicating the measurement area. In embodiment 1, the subject's facial orientation in the yaw direction is represented as "0 degrees" when the subject's face is facing forward, and is represented as a more negative angle as the subject's face is turned leftward from the front, and a more positive angle as the subject's face is turned rightward from the front.

[0029] For example, as shown in FIG. 2A , the reference measurement area information includes information indicating a total of 24 measurement areas with measurement area numbers (1) to (24). As described above, in the reference measurement area information, the measurement area numbers are associated with the subject's facial orientation (face orientation relative to the camera), but information indicating the facial orientation is omitted from FIG. 2A . For ease of understanding, FIG. 2A illustrates all 24 measurement areas set in the reference measurement area information so that their positional relationships on the captured image can be seen. Of the 24 measurement areas set in the reference measurement area information shown in FIG. 2A , the measurement areas with measurement area numbers (1) to (12) indicate measurement areas set in the skin area corresponding to the left cheek. Furthermore, of the 24 measurement areas set in the reference measurement area information shown in FIG. 2A , the measurement areas with measurement area numbers (13) to (24) indicate measurement areas set in the skin area corresponding to the right cheek. When the imaging device 2 captures an image of the subject's face, the subject's left cheek is captured on the right side of the captured image, and the subject's right cheek is captured on the left side of the captured image.

[0030] Although not shown in FIG. 2A , in the reference measurement area information, information indicating each measurement area, more specifically, each measurement area number, is provided with information that can identify which area of ​​the skin area the measurement area is set in. The information that can identify which area of ​​the skin area the measurement area is set in includes, for example, information indicating the four vertices of the measurement area. In the first embodiment, the measurement area is assumed to be a quadrangle. The information indicating the four vertices is represented, for example, by landmarks of facial organs such as the outer corners and inner corners of the eyes, the nose, and the mouth, or auxiliary landmarks. In the reference measurement area information, the information indicating the four vertices of the measurement area includes, for example, information indicating which facial organ landmarks the landmarks are, or information indicating the number of auxiliary landmarks from which facial organ landmarks the landmarks are on the line segment between the facial organ landmarks. Details of the facial organ landmarks and auxiliary landmarks will be described later.

[0031] FIG. 2B is a diagram showing frames (indicated by "I" in FIG. 2B ) of an image of a potential subject captured by the imaging device 2 at intervals of 10 degrees of the potential subject's facial orientation in the yaw direction, and corresponding measurement areas (indicated by "M" in FIG. 2B ) determined by an administrator or the like. Here, the administrator or the like sets the measurement areas at intervals of 10 degrees of the potential subject's facial orientation in the yaw direction. Note that this is merely an example, and the administrator or the like can set the measurement areas in appropriate units, such as in 1-degree increments. Also, in FIG. 2B , the possible range of the potential subject's facial orientation in the yaw direction is "-30 degrees to +30 degrees." This is determined by the administrator or the like based on the resolution of the imaging device 2, etc.

[0032] Even when the imaging device 2 images the intended subject at a reference position and a reference orientation, the range within which the imaging device 2 can image the intended subject, more specifically, the size of the measurement area imaged by the imaging device 2, varies depending on the facial orientation of the intended subject. When the size of the measurement area changes, the amount of pulse wave components contained in the signal indicating luminance changes extracted from the measurement area also changes. The administrator or the like pre-sets a measurement area from which a luminance signal containing pulse wave components sufficient to estimate the intended subject's pulse wave is extracted, depending on the facial orientation of the intended subject (face orientation relative to the camera). In the first embodiment, the reference position of the imaging device 2 is specifically a position in real space where the horizontal and vertical coordinates indicating the position of the imaging device 2 are the same as the horizontal and vertical coordinates indicating the position of the intended subject. Note that in the first embodiment, the position of the subject is indicated by the center of the subject's face. The real space is represented by three-dimensional coordinate axes, with the x-axis being an axis parallel to the width direction of the vehicle, the y-axis being an axis parallel to the height direction of the vehicle, and the z-axis being an axis parallel to the length direction of the vehicle, in other words, the direction of travel of the vehicle. In the first embodiment, "parallel" does not necessarily mean strictly parallel, but also includes approximately parallel. That is, the x-coordinate and y-coordinate of the coordinates indicating the reference position of the image capture device 2 are the same as the x-coordinate and y-coordinate of the coordinates indicating the position of the intended subject, respectively.

[0033] For example, if the intended subject's face orientation in the yaw direction is "-30 degrees," as shown in FIG. 2B , the intended subject's left cheek is barely captured in the captured image. That is, the skin area of ​​the intended subject's left cheek is not sufficiently captured in the captured image. In this case, the size of the measurement areas (1) to (12) set in the skin area corresponding to the intended subject's left cheek in the captured image is small, and almost no area is captured. As described above, the measurement area is an area that can be used to extract a pulse wave source signal indicating a change in brightness. If the measurement area is small, the components contained in the pulse wave source signal extracted from the measurement area will contain a large amount of noise, and it may be impossible to extract a pulse wave source signal containing a pulse wave component. The same applies when the measurement area is in a so-called shadow. If a pulse wave source signal containing a pulse wave component is not extracted, the accuracy of estimating the subject's pulse wave based on the pulse wave source signal will decrease. In other words, in order to extract a pulse wave source signal without reducing the accuracy of estimating the subject's pulse wave, it is necessary that the measurement region from which the pulse wave source signal is extracted is a region from which it is assumed that a pulse wave source signal containing the subject's pulse wave component can be extracted.

[0034] For example, in the example shown in FIG. 2B , if the expected subject's facial orientation in the yaw direction is "-30 degrees," the measurement areas with sufficient size are the measurement areas with measurement area numbers (13) to (24). Therefore, the administrator or the like determines that when the imaging device 2 captures an image of the expected subject at the reference position and reference orientation, if the subject's facial orientation in the yaw direction is "-30 degrees," the measurement areas with measurement area numbers (13) to (24) will be set. Here, the administrator or the like determines the measurement area based on the size of the measurement area, for example, whether the size of the measurement area is equal to or greater than a predetermined threshold. However, this is merely an example. For example, the administrator or the like may determine the measurement area based on the brightness of the measurement area, for example, whether the brightness of the measurement area is equal to or greater than a predetermined threshold. The administrator or the like may determine the measurement area from which a pulse wave source signal containing the subject's pulse wave component can be extracted, depending on the subject's facial orientation in the yaw direction. When the expected subject's face orientation in the yaw direction is "-20 degrees to +30 degrees," the administrator, etc., determines a measurement area with a sufficient size as the measurement area for that face orientation, in the same manner as when the expected subject's face orientation in the yaw direction is "-30 degrees." As a result, the administrator, etc., sets a total of 24 measurement areas, excluding overlaps, as described with reference to FIG. 2A, for the expected subject's face orientation in the yaw direction of "-30 degrees to +30 degrees."

[0035] Then, when the administrator or the like sets the measurement area according to the facial orientation in the yaw direction of the assumed subject, the administrator or the like generates reference measurement area information and stores it in a storage unit or the like. In the above example, the administrator or the like measures the facial orientation of the subject in the yaw direction as "-30 degrees" and measurement area numbers (13) to (24), the facial orientation of the subject in the yaw direction as "-20 degrees or less and greater than -30 degrees" and measurement area numbers (1), (7), (14) to (18), (20) to (24), the facial orientation of the subject in the yaw direction as "-10 degrees or less and greater than -20 degrees" and measurement area numbers (1) to (2), (7) to (8), (15) to (18), (21) to (24), and the facial orientation of the subject in the yaw direction as "greater than -10 degrees and less than +10 degrees." Reference measurement area information is generated in which area numbers (1) to (3), (7) to (9), (16) to (18), (22) to (24), the subject's facial orientation in the yaw direction of "+10 degrees or more but less than +20 degrees" correspond to measurement area numbers (1) to (4), (7) to (10), (17) to (18), (23) to (24), the subject's facial orientation in the yaw direction of "+20 degrees or more but less than +30 degrees" correspond to measurement area numbers (1) to (5), (7) to (11), (18) to (19), and the subject's facial orientation in the yaw direction of "+30 degrees" correspond to measurement area numbers (1) to (12).

[0036] Assuming that the above-mentioned reference measurement area information, information indicating the target face direction range, and layout information have been generated in advance, the measurement area setting unit 14 determines the measurement area to be set based on the layout information, information indicating the target face direction range, and reference measurement area information, and then sets the determined multiple measurement areas in the image area on frame Im(k) corresponding to the skin area indicated by the skin area information S(k) based on the reference measurement area information, frame Im(k) of the captured image acquired by the captured image acquisition unit 11, and skin area information S(k) output by the skin area detection unit 12.

[0037] Specifically, the measurement area setting unit 14 sets the measurement area associated with the target face direction range in the reference measurement area information as the measurement area to be set in the image area on frame Im(k) corresponding to the skin area indicated by the skin area information S(k). At this time, the measurement area setting unit 14 offsets the face direction in the yaw direction of the assumed subject defined in the reference measurement area information (camera-based face direction) based on the layout information, in other words, aligns the face direction in the yaw direction of the assumed subject (subject-based face direction) when imaged at the current position and orientation of the image capture device 2 relative to the assumed subject with the face direction in the yaw direction of the assumed subject imaged at the reference position and reference angle of the image capture device 2 (camera-based face direction), and then determines the measurement area to be set.

[0038] Here, several examples will be described in which the measurement area setting unit 14 determines the measurement area by offsetting the facial orientation of the assumed subject defined in the reference measurement area information. Note that the content of the reference measurement area information is assumed to be the content described using FIG. 2B.

[0039] For example, suppose the relationship between the position and orientation of the imaging device 2 and the position and orientation of the intended subject is as shown in FIG. 3 . Here, the intended subject is a subject facing forward, as described above. In FIG. 3 , "R" indicates the imaging range of the imaging device 2. Specifically, as shown in FIG. 3A , the vertical coordinate indicating the position of the imaging device 2 is the same as the vertical coordinate indicating the position of the intended subject (indicated by "D" in FIG. 3A ). Also, as shown in FIG. 3B , the horizontal coordinate indicating the position of the imaging device 2 is the same as the horizontal coordinate indicating the position of the intended subject (indicated by "D" in FIG. 3B ). That is, the x- and y-coordinates of the coordinates indicating the position of the imaging device 2 are the same as the x- and y-coordinates of the coordinates indicating the position of the intended subject. Furthermore, the optical axis of the imaging device 2 overlaps with a line indicating the forward direction of the intended subject. There is no difference in the relationship between the forward direction of the intended subject and the orientation of the optical axis of the imaging device 2. More specifically, the yaw angle formed by the optical axis of the image capture device 2 and a line parallel to the line indicating the front direction of the assumed subject is 0 degrees. In other words, the current position and orientation of the image capture device 2 with respect to the assumed subject are the reference position and reference orientation. Also, for example, it is assumed that the target face direction range is currently determined to be "-20 degrees to +20 degrees."

[0040] In this case, the measurement area setting unit 14 determines, based on the layout information, that the position and orientation of the image capture device 2 are not displaced from the reference position and orientation. The measurement area setting unit 14 determines that there is no need to offset the facial orientation of the intended subject (the facial orientation relative to the camera) defined in the reference measurement area information. The measurement area setting unit 14 determines, as the measurement area to be set, the measurement area indicated by the measurement area number associated with a facial orientation in the yaw direction of "-20 degrees to +20 degrees" in the reference measurement area information. For example, for the captured images and corresponding measurement areas for each facial orientation in the yaw direction of the subject shown in FIG. 2B, the measurement area corresponding to "-20 degrees to +20 degrees" is determined as the measurement area to be set (see FIGS. 4A and 4B; information about measurement areas that were not determined is shown shaded). In other words, based on the reference measurement area information, the measurement area setting unit 14 determines all 20 measurement areas, with measurement area numbers (1) to (5), (7) to (11), (14) to (18), and (20) to (24), as the measurement areas to be set in the image area corresponding to the skin area.

[0041] Also, for example, suppose that the relationship between the position and orientation of the imaging device 2 and the position and orientation of the assumed subject is as shown in FIG. 5 . Note that in FIG. 5 , "R" indicates the imaging range of the imaging device 2. Specifically, as shown in FIG. 5A , in real space, the vertical coordinate indicating the position of the imaging device 2 is the same as the vertical coordinate indicating the position of the assumed subject (indicated by "D" in FIG. 5A ), but as shown in FIG. 5B , in real space, the horizontal coordinate indicating the position of the imaging device 2 is different from the horizontal coordinate indicating the position of the assumed subject (indicated by "D" in FIG. 5B ). Here, the imaging device 2 will image the assumed subject facing forward from a position at a yaw angle of "+20 degrees" to the right of the assumed subject, and with regard to the relationship between the forward direction of the assumed subject and the direction of the optical axis of the imaging device 2, the angle formed by the optical axis of the imaging device 2 and a line parallel to the line indicating the forward direction of the assumed subject is "+20 degrees." In other words, when the intended subject turns to the right by a yaw angle of 20 degrees, the image capture device 2 captures the image of the intended subject from the front. Also, for example, suppose that the administrator or the like has determined the target face direction range to be "-10 degrees to +10 degrees."

[0042] In this case, the measurement area setting unit 14 determines, based on the layout information, that the position and orientation of the imaging device 2 are deviated from the reference position and orientation. The measurement area setting unit 14 must offset the facial orientation of the intended subject (camera-based facial orientation) defined in the reference measurement area information so that the positional relationship between the intended subject and the imaging device 2 when the imaging device 2 captures an image of the intended subject from the front coincides with the positional relationship assumed in the reference measurement area information. Here, the measurement area setting unit 14 increments the facial orientation of the intended subject in the yaw direction defined in the reference measurement area information by "+20 degrees." As a result, the facial orientation of the intended subject in the yaw direction, "-30 degrees to +30 degrees" defined in the reference measurement area information, can be interpreted as "-10 degrees to +50 degrees" when adjusted to the actual position and orientation of the imaging device 2. In other words, the subject-based facial orientation can be interpreted as "-10 degrees to +50 degrees." Then, the measurement area setting unit 14 determines the measurement area corresponding to the target face direction range "-10 degrees to +10 degrees" as the measurement area to be set. Here, the measurement area corresponding to the face direction in the yaw direction "-30 degrees to -10 degrees" in the reference measurement area information is determined as the measurement area to be set (see FIGS. 6A and 6B. Information about measurement areas that were not determined is shown shaded). In other words, based on the reference measurement area information, the measurement area setting unit 14 determines all 16 measurement areas, with measurement area numbers (1) to (2), (7) to (8), and (13) to (24), as the measurement areas to be set in the image area corresponding to the skin area.

[0043] In the above specific example, the subject changes his / her facial orientation only in the yaw direction, and the reference measurement area information is information in which the facial orientation in the yaw direction is associated with information indicating the measurement area, but the reference measurement area is also generated for the pitch and roll directions in the same manner as for the yaw direction.The measurement area setting unit 14 determines the measurement area to be set by offsetting the facial orientation of the assumed subject defined in the reference measurement area information (the facial orientation based on the camera) based on the deviation in the pitch or roll direction of the position and orientation of the imaging device 2 with respect to the assumed subject from the reference position and reference orientation, in the same manner as in the yaw direction described above.

[0044] As described above, the measurement area setting unit 14 determines the multiple measurement areas to be set based on the multiple measurement areas defined in the reference measurement area information, based on the layout information and information indicating the target face direction range, and then sets the multiple measurement areas determined based on the layout information in the image area on the frame Im(k) corresponding to the skin area indicated by the skin area information S(k), based on the frame Im(k) of the captured image acquired by the captured image acquisition unit 11 and the skin area information S(k) output by the skin area detection unit 12. Note that the measurement area setting unit 14 may acquire the captured image acquired by the captured image acquisition unit 11 via the skin area detection unit 12.

[0045] 7A, 7B, and 7C are diagrams illustrating an example of a method for setting a measurement region by the measurement region setting unit 14 in the pulse wave estimation device 1 according to embodiment 1. An example of a method for the measurement region setting unit 14 to set the measurement region ri(k) will be described using FIG. 7. First, as shown in FIGS. 7A and 7B, the measurement region setting unit 14 detects Ln (a positive integer) landmarks of facial features, such as the corners and corners of the eyes, the nose, and the mouth, in the skin region sr indicated by the skin region information S(k). In FIGS. 7A and 7B, the landmarks are indicated by circles. The measurement region setting unit 14 defines a vector storing the coordinate values ​​of the detected landmarks as L(k). The measurement region setting unit 14 may detect facial features using a known method, such as a model called a Constrained Local Model (CLM).

[0046] Next, the measurement area setting unit 14 sets the vertex coordinates of the quadrangle of the measurement area ri(k) based on the detected landmarks. For example, the measurement area setting unit 14 sets the vertex coordinates of a quadrangle as shown in FIG. 7C and sets Rn measurement areas ri(k).

[0047] Taking an example in which the measurement region setting unit 14 sets the measurement region ri(k) in a portion of the skin region sr corresponding to the cheek, the measurement region setting unit 14 selects a landmark LA1 on the facial contour and a landmark LA2 on the nose. The measurement region setting unit 14 first selects the landmark LA2 on the nose, and then selects the landmark LA1 on the facial contour that is closest to the landmark LA2 on the nose. The measurement region setting unit 14 then sets auxiliary landmarks a1, a2, and a3 so as to divide the line segment between the landmark LA1 and the landmark LA2 into four equal parts. Similarly, the measurement region setting unit 14 selects a landmark LB1 on the facial contour and a landmark LB2 on the nose. The measurement region setting unit 14 also sets auxiliary landmarks b1, b2, and b3 so as to divide the line segment between the landmark LB1 and the landmark LB2 into four equal parts. Note that the landmarks LB1 and LB2 may be selected, for example, from the landmarks on the facial contour or the nose that are adjacent to the landmarks LA1 and LA2, respectively. The measurement area setting unit 14 sets a quadrilateral area surrounded by the auxiliary landmarks a1, b1, b2, and a2 as one measurement area R1. The auxiliary landmarks a1, b1, b2, and a2 each have vertex coordinates corresponding to the measurement area R1. Similarly, the measurement area setting unit 14 sets one measurement area R2 surrounded by the auxiliary landmarks a2, b2, b3, and a3 and the vertex coordinates of the measurement area R2.

[0048] In the explanation using Fig. 7, the concept of the method for setting the measurement area ri(k) was simply explained, but in embodiment 1, as described above, the measurement area setting unit 14 determines the measurement area ri(k) based on multiple measurement areas defined in the reference measurement area information, based on the layout information and information indicating the target face direction range, and sets the determined measurement area ri(k). The measurement area setting unit 14 sets the measurement area ri(k) determined based on the layout information, information indicating the target face direction range, and the reference measurement area information, using the method described using Fig. 7. Note that, based on the reference measurement area information, the measurement area setting unit 14 can identify which points will be the landmarks or auxiliary landmarks that will become the vertices of the measurement area ri(k) to be set.

[0049] Although an example of setting the measurement region ri(k) in a portion corresponding to the cheek has been described here, the measurement region setting unit 14 can similarly set the measurement region and the vertex coordinates of the measurement region ri(k) for, for example, other portions of the cheek and the skin region sr in a portion corresponding to the chin. Although not shown in Fig. 7C, the measurement region setting unit 14 can also set the measurement region ri(k) in a portion of the skin region sr of the subject corresponding to the forehead or the tip of the nose.

[0050] For example, the measurement area setting unit 14 may set the measurement area ri(k) using a method other than CLM. For example, the measurement area setting unit 14 may set the measurement area ri(k) using a tracking technique such as a Kanade-Lucas-Tomasi (KLT) tracker. Specifically, the measurement area setting unit 14 may use CLM to detect the coordinates of facial organ points for the skin area of ​​the first frame Im(1) of a series of frames Im(k-Tp+1) to Im(k), and then track the facial organ points using a KLT tracker for the skin area of ​​the next frame Im(2) and subsequent frames, thereby calculating the facial organ points for the skin area of ​​each frame Im(k). In this case, because detection errors due to tracking accumulate, the measurement area setting unit 14 may execute CLM once every few frames and perform a reset process, such as resetting the coordinate positions of the facial organ points.

[0051] When the measurement region setting unit 14 sets multiple measurement regions ri(k), it generates measurement region information R(k) that indicates the multiple measurement regions ri(k) that have been set. The measurement region information R(k) includes information that indicates the position and size of Rn (positive integer) measurement regions ri(k) on the captured image. Each measurement region is defined as measurement region ri(k) (i = 1, 2, ..., Rn). In the first embodiment, the measurement region ri(k) is defined as a quadrangle, and the position and size of the measurement region ri(k) are defined as the coordinate values ​​of the four vertices of the quadrangle on the captured image. The measurement region setting unit 14 outputs the generated measurement region information R(k) to the luminance signal extraction unit 15.

[0052] The luminance signal extraction unit 15 sets a measurement region ri(k) (hereinafter referred to as a "used measurement region") ri(k) to be used to extract a pulse wave source signal from among the measurement regions ri(k) set by the measurement region setting unit 14. Based on the frame Im(k) of the captured image acquired by the captured image acquisition unit 11 and the measurement region information R(k) output from the measurement region setting unit 14, the luminance signal extraction unit 15 extracts a pulse wave source signal indicating a change in luminance during a pulse wave estimation target period, in other words, a period corresponding to the number of frames Tp, from each of the set used measurement regions ri(k) from among the multiple measurement regions ri(k) on frame Im(k) indicated by the measurement region information R(k). The pulse wave source signal is a signal that serves as the source of a pulse wave. The pulse wave estimation device 1 estimates the subject's pulse wave using the pulse wave source signal. The subject's pulse wave is estimated by a pulse wave estimation unit 17. Details of the pulse wave estimation unit 17 will be described later. The luminance signal extraction unit 15 may acquire the captured image acquired by the captured image acquisition unit 11 via the skin region detection unit 12 and the measurement region setting unit 14 .

[0053] Here, an example of a method in which the luminance signal extraction unit 15 sets the use measurement region ri(k) in embodiment 1 will be described. The luminance signal extraction unit 15 sets the use measurement region ri(k), for example, based on the face direction information F(k) output from the face direction estimation unit 13 and the use measurement region setting information. The use measurement region setting information is information in which the face direction of the subject is associated with the measurement region number. For example, an administrator or the like generates the use measurement region setting information in advance and stores it in a location, such as a storage unit, that can be referenced by the pulse wave estimation device 1.

[0054] For example, the luminance signal extraction unit 15 compares a frame Im(k) of the captured image acquired by the captured image acquisition unit 11 with a frame Im(k) of the captured image included in the facial direction information F(k) to determine the facial direction of the subject corresponding to the frame Im(k), more specifically, the facial direction of the subject estimated by the facial direction estimation unit 13 based on the frame Im(k). The luminance signal extraction unit 15 may determine the facial direction of the subject by comparing the identification number of the frame Im(k) of the captured image. An identification number is assigned to the frame Im(k) of the captured image. The imaging device 2 assigns an identification number to each frame Im(k) and outputs the captured image. Then, the luminance signal extraction unit 15 references the information for setting the used measurement area and sets the measurement area ri(k) having the measurement area number corresponding to the determined facial direction as the used measurement area ri(k). The luminance signal extraction unit 15 extracts a pulse wave source signal during a pulse wave estimation target period from each of the set usage measurement regions ri(k) on the frame Im(k) of the captured image acquired by the captured image acquisition unit 11.

[0055] For example, suppose the face orientation in the yaw direction of the subject indicated in the face orientation information F(k) including frame Im(k) of the captured image is "-10 degrees." Furthermore, suppose that measurement region numbers (1) to (2), (7) to (8), (15) to (18), and (21) to (24) are associated with the face orientation in the yaw direction of the subject in the information for setting the used measurement region. In this case, the luminance signal extraction unit 15 sets, as the used measurement region ri(k), the measurement regions ri(k) with measurement region numbers (1) to (2), (7) to (8), (15) to (18), and (21) to (24) among the measurement regions ri(k) set in frame Im(k) of the captured image acquired by the captured image acquisition unit 11. The luminance signal extraction unit 15 then extracts a pulse wave source signal indicating a change in luminance during the pulse wave estimation period from each of the used measurement regions ri(k) having measurement region numbers (1) to (2), (7) to (8), (15) to (18), and (21) to (24). After extracting the pulse wave source signal, the luminance signal extraction unit 15 generates pulse wave source signal information W(t) indicating the extracted pulse wave source signal.

[0056] The pulse wave raw signal information W(t) includes information indicating the pulse wave raw signal wi(t) extracted from the used measurement region ri(k). The pulse wave raw signal wi(t) is time-series data for Tp, and is extracted, for example, based on frames Im(k-Tp+1), Im(k-Tp+2), ..., Im(k) for the past Tp, and measurement region information R(k-Tp+1), R(k-Tp+2), ..., R(k). In extracting the pulse wave raw signal wi(t), the luminance signal extraction unit 15 calculates the difference Gi(j) (j=k-Tp+1, k-Tp+2, ..., k) in the luminance feature amount of each used measurement region ri(k) for each frame Im(k) of the captured image, relative to the previous frame Im(k-1). The luminance feature is a value calculated for each used measurement region ri(j) based on the luminance values ​​of the frame Im(j) of the captured image. The luminance feature is, for example, the average or variance of the luminance values ​​of the pixels included in the used measurement region ri(j). In the first embodiment, as an example, the luminance feature is the average of the luminance values ​​of the pixels included in the used measurement region ri(j). The luminance signal extraction unit 15 arranges Gi(j) calculated for each frame Im(k) of the captured image acquired during the pulse wave estimation target period in chronological order to obtain the pulse wave original signal wi(t). That is, the luminance signal extraction unit 15 sets the pulse wave original signal wi(t) = [Gi(k-Tp+1), Gi(k-Tp+2), ..., Gi(k)]. Note that if a corresponding used measurement region ri(k) is not set in the previous frame Im(k-1), the luminance signal extraction unit 15 does not calculate the difference of the used measurement region ri(k). That is, the luminance signal extraction unit 15 sets the difference of the used measurement region ri(k) to "0".

[0057] The luminance signal extraction unit 15 generates pulse wave original signal information W(t) indicating the pulse wave original signal wi(t) in each used measurement region ri(k). The pulse wave original signal information W(t) includes the pulse wave original signal wi(t) in each used measurement region ri(k), information indicating which used measurement region ri(k) the pulse wave original signal wi(t) was extracted from, each frame Im(k) of the captured image including the used measurement region ri(k) from which the pulse wave original signal wi(t) was extracted, and an identification number of each frame Im(k) of the captured image in the time series. The luminance signal extraction unit 15 outputs the generated pulse wave original signal information W(t) to the luminance signal selection unit 16.

[0058] Based on the pulse wave source signal information W(t) output from the luminance signal extraction unit 15 and taking into consideration the face direction of the subject estimated by the face direction estimation unit 13, the luminance signal selection unit 16 selects a time-series pulse wave source signal wi(t) to be used for estimating the subject's pulse wave (hereinafter referred to as the "estimation pulse wave source signal") wi(t) from the time-series pulse wave source signals wi(t) extracted by the luminance signal extraction unit 15 during the pulse wave estimation target period.

[0059] The method of selecting the estimation pulse wave original signal wi(t) by the luminance signal selector 16 in the first embodiment will be described. The luminance signal selector 16 calculates the distribution ratio of the subject's facial orientation corresponding to each frame Im(k) of the captured images estimated by the face direction estimator 13 during the pulse wave estimation target period. The luminance signal selector 16 then selects, as the estimation pulse wave original signal wi(t), a time-series pulse wave original signal wi(t) extracted from a commonly-defined measurement area ri(k) in each frame Im(k) of the captured images from which the subject's facial orientation, whose appearance frequency is equal to or greater than a predetermined threshold (hereinafter referred to as the "face direction determination threshold"), is estimated. For example, the face direction estimator 13 stores the facial orientation information F(k) in a time-series memory or the like. The captured images are provided with information regarding the capture date and time of the captured images. Based on the facial direction information F(k) stored in a memory unit or the like, the luminance signal selection unit 16 can calculate the distribution ratio of the subject's facial direction corresponding to each frame Im(k) of the captured image estimated by the facial direction estimation unit 13 during the pulse wave estimation target period.

[0060] 8 is a diagram showing an example of a histogram showing the distribution ratio of the subject's facial direction corresponding to each frame Im(k) of the captured image, which is estimated by the facial direction estimation unit 13 during the pulse wave estimation target period, calculated by the luminance signal selection unit 16 in Embodiment 1. It is assumed here that the subject changes their facial direction only in the yaw direction. That is, the distribution ratio of the subject's facial direction shown in FIG. 8 is the distribution ratio of the subject's facial direction in the yaw direction.

[0061] In the histogram shown in Fig. 8, the frequency of the subject's facial orientation in the yaw direction ranging from "-5 degrees to +5 degrees" is equal to or greater than the facial orientation determination threshold. Therefore, the luminance signal selector 16 selects, as the time-series estimation pulse wave original signal wi(t), the time-series pulse wave original signal wi(t) extracted from the use measurement region ri(k) commonly determined in each frame Im(k) of the captured image from which the subject's facial orientation in the yaw direction was estimated to be "-5 degrees to +5 degrees." The use measurement region ri(k) commonly determined in each frame Im(k) of the captured image from which the subject's facial orientation in the yaw direction was estimated to be "-5 degrees to +5 degrees" refers to the measurement region ri(k) determined by the luminance signal extractor 15 as the use measurement region ri(k) in each frame Im(k) of the captured image from which the subject's facial orientation in the yaw direction was estimated to be "-5 degrees to +5 degrees."

[0062] For example, the luminance signal selector 16 may identify a commonly determined used measurement region ri(k) for each frame Im(k) of the captured image from which the subject's facial orientation in the yaw direction is estimated to be "-5 degrees to +5 degrees" based on the above-described used measurement region setting information. The luminance signal selector 16 selects a time-series pulse wave source signal wi(t) extracted from the identified used measurement region ri(k) as a time-series estimation pulse wave source signal wi(t). The luminance signal selector 16 then outputs pulse wave source signal information W(t) (hereinafter referred to as "selected pulse wave source signal information") including the selected estimation pulse wave source signal wi(t) to the pulse wave estimation unit 17. Note that, as described with reference to FIG. 8 , the distribution ratio of the subject's facial orientation is determined in increments of 5 degrees. However, this is merely an example. For example, the distribution ratio of the subject's facial orientation may be determined in increments of 1 degree. Basically, it is safe to ensure that the resolution of the face direction defined in the reference measurement area information matches the resolution of the distribution ratio of the subject's face direction.

[0063] The pulse wave estimation unit 17 estimates the subject's pulse wave based on the selected pulse wave source signal information W(t) output from the luminance signal selection unit 16. The pulse wave estimation unit 17 may estimate the subject's pulse wave based on a pulse wave source signal using a known method. Here, an example of a method for estimating the subject's pulse wave using the pulse wave estimation unit 17 will be described. For example, the pulse wave estimation unit 17 first generates a signal (hereinafter referred to as a "separated signal") representing multiple signal components (hereinafter referred to as "principal components") based on the estimated pulse wave source signal wi(t). Specifically, the pulse wave estimation unit 17 analyzes the multiple principal components using a common signal separation technique such as PCA or ICA, and generates a separated signal representing the analyzed multiple principal components. By analyzing the signal components using a common signal separation technique such as PCA or ICA, the pulse wave estimation unit 17 separates components that appear to be pulse wave components and components that appear to be noise components from the multiple estimated pulse wave source signals wi(t).

[0064] The pulse wave estimation unit 17 restores an estimated pulse wave source signal wi(t) for each used measurement region ri(k) based on the generated separated signals, specifically, the generated separated signal information Sep(t) relating to the generated separated signals indicating the multiple principal components. Note that each separated signal included in the separated signal information Sep(t) includes an estimated pulse wave source signal wi(t) for each used measurement region ri(k). The pulse wave estimation unit 17 can restore an estimated pulse wave source signal wi(t) for each used measurement region ri(k) from the multiple separated signals included in the separated signal information Sep(t).

[0065] After restoring the estimated pulse wave original signal wi(t) for each used measurement region ri(k), the pulse wave estimation unit 17 generates restored estimated pulse wave original signal information RW(t) indicating the restored estimated pulse wave original signal wi(t) for each used measurement region ri(k). The restored estimated pulse wave original signal information RW(t) includes the restored estimated pulse wave original signal wi(t) for each used measurement region ri(k).

[0066] The pulse wave estimation unit 17 estimates the subject's pulse wave based on the generated restored estimation pulse wave source signal information RW(t). For example, the pulse wave estimation unit 17 calculates the S / N ratio of the restored estimation pulse wave source signal for each used measurement region ri(k). The pulse wave estimation unit 17 weights the restored estimation pulse wave source signal for each used measurement region ri(k) based on the calculated S / N ratio, and then calculates composite estimation pulse wave signal information D(t) by adding up the restored estimation pulse wave source signals corresponding to each used measurement region ri(k). In other words, the pulse wave estimation unit 17 calculates one composite estimation pulse wave signal information D(t) for all used measurement regions ri(k). Because weighting based on the S / N ratio has been performed, the composite estimation pulse wave signal information D(t) is assumed to be a signal that resembles a pulse wave component, with noise components removed. Pulse wave estimation unit 17 then performs a Fourier transform on the composite estimated pulse wave signal information D(t) and calculates the peak frequency in the frequency power spectrum within a predetermined frequency range as the pulse rate. The predetermined frequency range is set taking into account the range of human heart rates. Pulse wave estimation unit 17 outputs pulse wave estimation result P(t), which is pulse wave information indicating the estimated pulse wave, to output unit 18. The pulse wave information may be, for example, time-series data of the subject's pulse wave estimated by pulse wave estimation unit 17, the subject's pulse rate, or the subject's pulse interval.

[0067] Output unit 18 outputs the pulse wave estimation result P(t) output from pulse wave estimation unit 17 to, for example, an arousal level estimation device or an abnormality detection device. The function of output unit 18 may be provided in pulse wave estimation unit 17. If the function of output unit 18 is provided in pulse wave estimation unit 17, output unit 18 is not an essential component of pulse wave estimation device 1.

[0068] The operation of the pulse wave estimation device 1 according to embodiment 1 will now be described. Fig. 9 is a flowchart for explaining the operation of the pulse wave estimation device 1 according to embodiment 1. For example, when the power supply of a vehicle is turned on, the pulse wave estimation device 1 repeats the processing shown in the flowchart of Fig. 9 until the power supply of the vehicle is turned off.

[0069] The captured image acquisition unit 11 acquires a captured image of the subject (step ST1), and outputs the acquired captured image to the skin region detection unit 12.

[0070] The skin region detection unit 12 detects the subject's skin region from the frame Im(k) included in the captured image acquired by the captured image acquisition unit 11 in step ST1 (step ST2). The skin region detection unit 12 generates skin region information S(k) indicating the detected skin region. The skin region detection unit 12 outputs the generated skin region information S(k) to the measurement region setting unit 14.

[0071] Based on the frame Im(k) of the captured image acquired by the captured image acquisition unit 11 in step ST1 and the skin area information S(k) output by the skin area detection unit 12 in step ST2, the measurement area setting unit 14 sets a plurality of measurement areas ri(k) that can be used to extract a pulse wave original signal indicating a luminance change in an image area on the frame Im(k) corresponding to the skin area indicated by the skin area information S(k) (step ST3). Specifically, based on the layout information and the information indicating the target face direction range, the measurement area setting unit 14 determines a plurality of measurement areas ri(k) to be set based on the plurality of measurement areas defined in the reference measurement area information, and sets the determined plurality of measurement areas ri(k) in the image area on the frame Im(k) corresponding to the skin area indicated by the skin area information S(k) based on the frame Im(k) of the captured image acquired by the captured image acquisition unit 11 and the skin area information S(k) output by the skin area detection unit 12. After setting the plurality of measurement regions ri(k), the measurement region setting unit 14 generates measurement region information R(k) indicating the set plurality of measurement regions ri(k). The measurement region setting unit 14 outputs the generated measurement region information R(k) to the luminance signal extraction unit 15.

[0072] The face direction estimation unit 13 estimates the face direction of the subject (face direction based on the subject) on a frame-by-frame basis based on the frame Im(k) of the captured image acquired by the captured image acquisition unit 11 in step ST1 (step ST4). The face direction estimation unit 13 outputs face direction information F(k) relating to the estimated face direction of the subject (face direction based on the subject) to the luminance signal extraction unit 15.

[0073] Based on the frame Im(k) of the captured image acquired by the captured image acquisition unit 11 in step ST1 and the measurement region information R(k) output from the measurement region setting unit 14 in step ST3, the luminance signal extraction unit 15 extracts a time-series luminance signal indicating a change in luminance during a pulse wave estimation period from each of the use measurement regions ri(k) among the multiple measurement regions ri(k) indicated by the measurement region information R(k) on the frame Im(k), in other words, a time-series pulse wave source signal wi(t) (step ST5). The luminance signal extraction unit 15 generates pulse wave source signal information W(t) indicating the extracted time-series pulse wave source signal wi(t). The luminance signal extraction unit 15 outputs the generated pulse wave source signal information W(t) to the luminance signal selection unit 16.

[0074] Based on the pulse wave source signal information W(t) output from the luminance signal extraction unit 15 in step ST5 and taking into consideration the subject's facial orientation estimated by the facial orientation estimation unit 13 in step ST4, the luminance signal selection unit 16 selects a time-series luminance signal, in other words, a time-series estimated pulse wave source signal wi(t) (step ST6). The luminance signal selection unit 16 outputs the selected pulse wave source signal information W(t) to the pulse wave estimation unit 17.

[0075] The pulse wave estimation unit 17 estimates the subject's pulse wave based on the selected pulse wave original signal information W(t) output from the luminance signal selection unit 16 in step ST6 (step ST7). The pulse wave estimation unit 17 outputs the pulse wave estimation result P(t), which is pulse wave information indicating the estimated pulse wave, to the output unit 18. The output unit 18 outputs the pulse wave estimation result P(t) output from the pulse wave estimation unit 17 to, for example, an arousal level estimation device or an abnormality detection device.

[0076] 9, the process of step ST4 is performed after the process of step ST3, but this is merely an example. For example, the order of the processes of step ST4 and step ST3 may be reversed, or the processes of steps ST2 to ST3 and step ST4 may be performed in parallel. The process of step ST4 may be performed after the process of step ST1 and before the process of step ST5 is performed.

[0077] Fig. 10 is a flowchart for explaining the details of the processing of step ST5 in Fig. 9. The luminance signal extraction unit 15 compares frame Im(k) of the captured image acquired by the captured image acquisition unit 11 in step ST1 in Fig. 9 with frame Im(k) of the captured image included in the face direction information F(k) output in step ST4 in Fig. 9, and determines the face direction of the subject corresponding to frame Im(k). Then, the luminance signal extraction unit 15 refers to the information for setting the used measurement area, and sets the measurement area ri(k) having the measurement area number corresponding to the determined face direction as the used measurement area ri(k) (step ST501). Based on frame Im(k) of the captured image acquired by the captured image acquisition unit 11 in step ST1 of Fig. 9 and measurement region information R(k) output from the measurement region setting unit 14 in step ST3 of Fig. 9, the luminance signal extraction unit 15 extracts time-series pulse wave source signals wi(t) during the pulse wave estimation target period from each of the used measurement regions ri(k) in frame Im(k) (step ST502). After extracting the time-series pulse wave source signals wi(t), the luminance signal extraction unit 15 generates pulse wave source signal information W(t) indicating the extracted time-series pulse wave source signals wi(t). The luminance signal extraction unit 15 outputs the generated pulse wave source signal information W(t) to the luminance signal selection unit 16.

[0078] 11 is a flowchart illustrating the details of the process of step ST6 in FIG. The luminance signal selection unit 16 calculates the distribution ratio of the subject's facial orientation corresponding to each frame Im(k) of captured images estimated by the facial orientation estimation unit 13 during the pulse wave estimation period (step ST601). The luminance signal selection unit 16 selects, as a time-series estimated pulse wave original signal wi(t), a time-series estimated pulse wave original signal wi(t) extracted from a common used measurement region ri(k) for each frame Im(k) of captured images from which the subject's facial orientation, whose appearance frequency is equal to or greater than the facial orientation determination threshold, is estimated (step ST602). The luminance signal selection unit 16 outputs the selected pulse wave original signal information W(t) to the pulse wave estimation unit 17.

[0079] In this way, pulse wave estimation device 1 detects the subject's skin area from the captured image and sets, in the area corresponding to the skin area on the captured image, measurement area ri(k) that can be used to extract pulse wave source signal wi(t) that indicates a change in brightness and that includes the subject's pulse wave component. Specifically, pulse wave estimation device 1 determines measurement area ri(k) to be set in the area corresponding to the skin area on the captured image based on information indicating the target face direction range, layout information, and reference measurement area information, and sets the determined measurement area ri(k). The pulse wave estimation device 1 sets a usage measurement region ri(k) from among the measurement regions ri(k), extracts a time-series pulse wave source signal wi(t) based on luminance changes in the usage measurement region ri(k), and then selects a time-series estimation pulse wave source signal wi(t) to be used to estimate the subject's pulse wave from the time-series pulse wave source signals wi(t) extracted during the pulse wave estimation target period, taking into account the subject's facial orientation estimated frame by frame based on the captured images. The pulse wave estimation device 1 estimates the subject's pulse wave based on the selected time-series estimation pulse wave source signal wi(t). Specifically, the pulse wave estimation device 1 extracts the time-series pulse wave source signal wi(t) and selects the time-series estimation pulse wave source signal wi(t) by using, as the used measurement area ri(k), a measurement area ri(k) from the set measurement area ri(k) that corresponds to the subject's facial orientation estimated based on the captured image in which the measurement area ri(k) is set. The pulse wave estimation device 1 then calculates the distribution ratio of the subject's facial orientation for each frame estimated during the pulse wave estimation period, and selects, as the time-series estimation pulse wave source signal wi(t), the time-series pulse wave source signal wi(t) extracted from the used measurement area ri(k) that is set in common in the captured images from which the subject's facial orientation is estimated, and whose occurrence frequency is equal to or greater than a facial orientation determination threshold.

[0080] This allows pulse wave estimation device 1 to extract a luminance signal containing a sufficient pulse wave component for estimating the subject's pulse wave, in other words, a pulse wave source signal wi(t), from the skin region of the captured image. As a result, pulse wave estimation device 1 can prevent a decrease in the accuracy of estimating the subject's pulse wave due to, for example, the skin region of the person whose pulse wave is to be estimated (i.e., the subject) not being captured in the captured image or the skin region being shadowed.

[0081] 12A and 12B are diagrams illustrating an example of the hardware configuration of pulse wave estimation device 1 according to embodiment 1. In embodiment 1, the functions of captured image acquisition unit 11, skin region detection unit 12, face orientation estimation unit 13, measurement region setting unit 14, luminance signal extraction unit 15, luminance signal selection unit 16, pulse wave estimation unit 17, and output unit 18 are realized by processing circuit 101. That is, pulse wave estimation device 1 includes processing circuit 101 for extracting, from the skin region of a captured image, a luminance signal containing a pulse wave component sufficient to estimate a person's pulse wave, and for controlling estimation of the subject's pulse wave from the extracted luminance signal. Processing circuit 101 may be dedicated hardware as shown in FIG. 12A, or a processor 104 that executes a program stored in memory as shown in FIG. 12B.

[0082] When processing circuitry 101 is dedicated hardware, processing circuitry 101 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof.

[0083] When the processing circuit is the processor 104, the functions of the captured image acquisition unit 11, skin area detection unit 12, face direction estimation unit 13, measurement area setting unit 14, luminance signal extraction unit 15, luminance signal selection unit 16, pulse wave estimation unit 17, and output unit 18 are realized by software, firmware, or a combination of software and firmware. The software or firmware is written as a program and stored in the memory 105. The processor 104 reads and executes the program stored in the memory 105 to execute the functions of the captured image acquisition unit 11, skin area detection unit 12, face direction estimation unit 13, measurement area setting unit 14, luminance signal extraction unit 15, luminance signal selection unit 16, pulse wave estimation unit 17, and output unit 18. In other words, the pulse wave estimation device 1 includes the memory 105 for storing a program that, when executed by the processor 104, results in the execution of steps ST1 to ST7 of FIG. 9 described above. In addition, the program stored in memory 105 can also be said to cause the computer to execute the processing procedures or methods of the captured image acquisition unit 11, skin area detection unit 12, face direction estimation unit 13, measurement area setting unit 14, luminance signal extraction unit 15, luminance signal selection unit 16, pulse wave estimation unit 17, and output unit 18. Here, the memory 105 may be, for example, a non-volatile or volatile semiconductor memory such as a RAM, a ROM (Read Only Memory), a flash memory, an EPROM (Erasable Programmable Read Only Memory), or an EEPROM (Electrically Erasable Programmable Read-Only Memory), a magnetic disk, a flexible disk, an optical disk, a compact disk, a mini disk, or a DVD (Digital Versatile Disc).

[0084] Note that the functions of the captured image acquisition unit 11, skin area detection unit 12, face direction estimation unit 13, measurement area setting unit 14, luminance signal extraction unit 15, luminance signal selection unit 16, pulse wave estimation unit 17, and output unit 18 may be partially implemented by dedicated hardware and partially implemented by software or firmware. For example, the functions of the captured image acquisition unit 11 and output unit 18 may be implemented by a processing circuit 101 as dedicated hardware, while the functions of the skin area detection unit 12, face direction estimation unit 13, measurement area setting unit 14, luminance signal extraction unit 15, luminance signal selection unit 16, and pulse wave estimation unit 17 may be implemented by a processor 104 reading and executing a program stored in a memory 105. The storage unit (not shown) may be, for example, the memory 105. The pulse wave estimation device 1 also includes an input interface device 102 and an output interface device 103 that communicate with devices such as the image capture device 2 via wired or wireless communication.

[0085] In the first embodiment, the subject is a driver of a vehicle, but this is merely an example. The subject may be a passenger other than the driver of the vehicle.

[0086] In the first embodiment described above, pulse wave estimation device 1 is an in-vehicle device, and captured image acquisition unit 11, skin area detection unit 12, face direction estimation unit 13, measurement area setting unit 14, luminance signal extraction unit 15, luminance signal selection unit 16, pulse wave estimation unit 17, and output unit 18 are provided in the in-vehicle device. However, the present invention is not limited to this. Some of captured image acquisition unit 11, skin area detection unit 12, face direction estimation unit 13, measurement area setting unit 14, luminance signal extraction unit 15, luminance signal selection unit 16, pulse wave estimation unit 17, and output unit 18 may be mounted in the in-vehicle device of the vehicle, and the rest may be provided in a server connected to the in-vehicle device via a network, so that the in-vehicle device and the server form a system. In addition, the captured image acquisition unit 11, skin area detection unit 12, face direction estimation unit 13, measurement area setting unit 14, luminance signal extraction unit 15, luminance signal selection unit 16, pulse wave estimation unit 17, and output unit 18 may all be provided on the server.

[0087] Furthermore, the pulse wave estimation device 1 according to the first embodiment described above is not limited to an in-vehicle device mounted on a vehicle, but can also be applied to, for example, moving objects other than vehicles or home appliances. Furthermore, the subject is not limited to a vehicle occupant, but can be various other people. As a specific example, the pulse wave estimation device 1 may be mounted on a television set installed in the living room of a residence. In this case, the subject is a user, such as a resident of the residence. The pulse wave estimation device 1 estimates the user's pulse wave based on an image captured by an imaging device 2 mounted on the television.

[0088] In the first embodiment described above, the measurement region setting unit 14 in the pulse wave estimation device 1 determines the measurement region ri(k) to be set in the region corresponding to the skin region on the captured image based on information indicating the target face direction range, layout information, and reference measurement region information. This is merely an example. For example, if the target face direction range is the range of all possible face directions that the subject can take within the range that the imaging device 2 can capture, and it is assumed that the imaging device 2 always captures the subject at a reference position and reference direction, the measurement region setting unit 14 does not need to consider the target face direction range and layout information. In this case, for example, an administrator or the like may determine in advance, based on the captured image of the potential subject captured by the imaging device 2, the measurement region ri(k) that can be used to extract the pulse wave original signal wi(t) containing the subject's pulse wave component, and generate information indicating the determined measurement region ri(k) as reference measurement region information and store it in a storage unit or the like. The measurement area setting unit 14 does not take into account the information indicating the target face direction range and the layout information, and simply sets the measurement area ri(k) defined in the reference measurement area information to an area corresponding to the skin area on the captured image.

[0089] As described above, the pulse wave estimation device 1 according to the first embodiment includes a captured image acquisition unit 11 that acquires captured images of a person (subject) frame by frame, a skin region detection unit 12 that detects the person's skin region from the captured image, a measurement region setting unit 14 that sets a measurement region ri(k) that can be used to extract a pulse wave original signal wi(t) that indicates a luminance change and that includes a pulse wave component of the person in an area corresponding to the skin region on the captured image, a face direction estimation unit 13 that estimates the face direction of the person frame by frame based on the captured image, and a measurement region setting unit 14 that detects the pulse wave original signal wi(t) from the measurement region ri(k) set by the measurement region setting unit 14. The pulse wave estimation device 1 is configured to include a luminance signal extraction unit 15 that sets a used measurement region ri(k) used to extract wi(t) and extracts a time-series pulse wave original signal wi(t) based on luminance changes in the set used measurement region ri(k), a luminance signal selection unit 16 that selects a time-series estimation pulse wave original signal wi(t) to be used for estimating the person's pulse wave from the time-series pulse wave original signals wi(t) extracted by the luminance signal extraction unit 15 during a pulse wave estimation target period, taking into account the person's facial direction estimated by the facial direction estimation unit 13, and a pulse wave estimation unit 17 that estimates the person's pulse wave based on the time-series estimation pulse wave original signal wi(t) selected by the luminance signal selection unit 16. Therefore, the pulse wave estimation device 1 can prevent a decrease in the accuracy of estimating the subject's pulse wave due to, for example, the skin region of the person whose pulse wave is to be estimated, i.e., the subject, not being captured in the captured image or the skin region being overshadowed.

[0090] In detail, in the pulse wave estimation device 1, the luminance signal extraction unit 15 extracts pulse wave source signals wi(t) from measurement regions ri(k) set by the measurement region setting unit 14, the measurement region ri(k) corresponding to the facial direction of the person (subject) estimated by the facial direction estimation unit 13 based on the captured image in which the measurement region ri(k) is set, as the used measurement region ri(k). The luminance signal selection unit 16 calculates the distribution rate of the facial direction of the person estimated by the facial direction estimation unit 13 for each frame during the pulse wave estimation target period, and selects, as the time-series pulse wave source signal wi(t), the time-series pulse wave source signal wi(t) extracted from the used measurement region ri(k) set in common in the captured images from which the facial direction of the person whose appearance frequency is equal to or greater than the facial direction determination threshold is estimated. Therefore, the pulse wave estimation device 1 can prevent a decrease in the accuracy of estimating the subject's pulse wave due to the skin area of ​​the person whose pulse wave is to be estimated, i.e., the subject, not being captured in the captured image, or the occurrence of so-called shadowing on the skin area.

[0091] Embodiment 2 In Embodiment 1, the pulse wave estimation device selected a time-series pulse wave source signal for estimation based on the distribution ratio of the subject's facial orientation for each frame. In Embodiment 2, an embodiment will be described in which a time-series pulse wave source signal for estimation is selected using a method different from that of Embodiment 1. Note that in the following Embodiment 2, as in Embodiment 1, it is assumed that the pulse wave estimation device is mounted on a vehicle and the subject is the driver of the vehicle.

[0092] 13 is a diagram showing an example of the configuration of pulse wave estimation device 1a according to embodiment 2. In the configuration of pulse wave estimation device 1a according to embodiment 2, the same components as those of pulse wave estimation device 1 described in embodiment 1 using FIG. 1 are assigned the same reference numerals, and redundant description will be omitted. In pulse wave estimation device 1a according to embodiment 2, the specific operation of luminance signal selection unit 16a differs from the specific operation of luminance signal selection unit 16 in pulse wave estimation device 1 according to embodiment 1.

[0093] Based on the pulse wave source signal information W(t) output from the luminance signal extraction unit 15 and taking into consideration the face direction of the subject estimated by the face direction estimation unit 13, the luminance signal selection unit 16a selects a time-series estimation pulse wave source signal wi(t) to be used for estimating the subject's pulse wave from among the time-series pulse wave source signals wi(t) extracted by the luminance signal extraction unit 15 during the pulse wave estimation target period.

[0094] The method of selecting the estimation pulse wave original signal wi(t) by the luminance signal selector 16a will now be described. In the second embodiment, the luminance signal selector 16a calculates the distribution ratio of the used measurement region ri(k) from which the luminance signal extractor 15 extracted the pulse wave original signal wi(t) during the pulse wave estimation target period. The luminance signal selector 16a then selects the time-series pulse wave original signal wi(t) extracted from the used measurement region ri(k) whose occurrence frequency is equal to or greater than a predetermined threshold (hereinafter referred to as the "region determination threshold") as the estimation pulse wave original signal wi(t). Note that, for example, the luminance signal selector 16a stores the pulse wave original signal information W(t) in a time-series memory unit (not shown). Based on the pulse wave source signal information W(t) stored in the memory unit, the luminance signal selection unit 16a can calculate the distribution ratio of the used measurement region ri(k) from which the luminance signal extraction unit 15 extracted the pulse wave source signal wi(t) during the pulse wave estimation target period.

[0095] FIG. 14 illustrates an example of a histogram showing the distribution ratio of the used measurement regions ri(k) from which the pulse wave source signals wi(t) extracted by the luminance signal extracting unit 15 were extracted during the pulse wave estimation target period, as calculated by the luminance signal selecting unit 16a in the second embodiment. In FIG. 14 , as an example, the luminance signal extracting unit 15 sets all 42 measurement regions ri(k) with measurement region numbers (1) to (42) as the used measurement regions ri(k) during the pulse wave estimation target period, and extracts the pulse wave source signals wi(t). In the histogram shown in FIG. 14 , the frequency of the used measurement regions ri(k) with measurement region numbers (1) to (26) is equal to or greater than the region determination threshold. Therefore, the luminance signal selecting unit 16a selects the time-series pulse wave source signals wi(t) extracted from the used measurement regions ri(k) with measurement region numbers (1) to (26) as the time-series estimated pulse wave source signals wi(t). The luminance signal selector 16a outputs pulse wave source signal information W(t) including the selected time-series estimation pulse wave source signal wi(t) to the pulse wave estimation unit 17 as selected pulse wave source signal information W(t).

[0096] The operation of pulse wave estimation device 1a according to embodiment 2 will now be described. Fig. 15 is a flowchart for explaining the operation of pulse wave estimation device 1a according to embodiment 2. For example, when the vehicle power is turned on, pulse wave estimation device 1a repeats the processing shown in the flowchart of Fig. 15 until the vehicle power is turned off.

[0097] The specific operations of steps ST1 to ST5 and step ST7 performed by pulse wave estimation device 1a are similar to the specific operations of steps ST1 to ST5 and step ST7 already explained using the flowchart of FIG. 9 in embodiment 1, and therefore the same step numbers are used and redundant explanations will be omitted.

[0098] Based on the pulse wave source signal information W(t) output from the luminance signal extraction unit 15 in step ST5, the luminance signal selection unit 16 selects a time-series estimated pulse wave source signal wi(t) (step ST6a) taking into consideration the subject's facial orientation estimated by the facial orientation estimation unit 13 in step ST4. The luminance signal selection unit 16 outputs the selected pulse wave source signal information W(t) to the pulse wave estimation unit 17.

[0099] 15, the process of step ST4 is performed after the process of step ST3, but this is merely an example. For example, the order of the processes of step ST4 and step ST3 may be reversed, or the processes of steps ST2 to ST3 and step ST4 may be performed in parallel. The process of step ST4 may be performed after the process of step ST1 and before the process of step ST5 is performed.

[0100] 16 is a flowchart for explaining the details of step ST6a in FIG. The luminance signal selector 16a calculates the distribution ratio of the used measurement region ri(k) from which the luminance signal extractor 15 extracted the pulse wave source signal wi(t) during the pulse wave estimation period (step ST611). The luminance signal selector 16a selects the time-series pulse wave source signal wi(t) extracted from the used measurement region ri(k) whose occurrence frequency is equal to or greater than the region-determination threshold as the time-series estimated pulse wave source signal wi(t) (step ST612). The luminance signal selector 16a outputs the selected pulse wave source signal information W(t) to the pulse wave estimation unit 17.

[0101] In this way, pulse wave estimation device 1a detects the subject's skin area from the captured image and sets, in the area corresponding to the skin area on the captured image, measurement area ri(k) that can be used to extract pulse wave original signal wi(t) that indicates a change in brightness and that includes the subject's pulse wave component. Specifically, pulse wave estimation device 1a determines measurement area ri(k) to be set in the area corresponding to the skin area on the captured image based on information indicating the target face direction range, layout information, and reference measurement area information, and sets the determined measurement area ri(k). The pulse wave estimation device 1a sets a usage measurement area ri(k) from among the measurement areas ri(k), extracts a time-series pulse wave source signal based on luminance changes in the usage measurement area ri(k), and then selects a time-series estimation pulse wave source signal wi(t) to be used to estimate the subject's pulse wave from the time-series pulse wave source signals wi(t) extracted during the pulse wave estimation target period, taking into account the subject's facial orientation estimated frame by frame based on the captured images. The pulse wave estimation device 1a estimates the subject's pulse wave based on the selected time-series estimation pulse wave source signal wi(t). In detail, when extracting the time-series pulse wave source signal wi(t) and selecting the time-series estimated pulse wave source signal wi(t), the pulse wave estimation device 1a calculates the distribution ratio of the used measurement region ri(k) from which the pulse wave source signal wi(t) was extracted during the pulse wave estimation target period, and selects the time-series pulse wave source signal wi(t) extracted from the used measurement region ri(k) whose occurrence frequency is equal to or greater than the region determination threshold as the time-series estimated pulse wave source signal wi(t).

[0102] This allows pulse wave estimation device 1a to extract a luminance signal containing a sufficient pulse wave component for estimating the subject's pulse wave (i.e., pulse wave source signal wi(t)) from the skin region of the captured image. As a result, pulse wave estimation device 1a can prevent a decrease in the accuracy of estimating the subject's pulse wave due to, for example, the skin region of the person whose pulse wave is to be estimated (i.e., the subject) not being captured in the captured image or the skin region being shadowed.

[0103] The hardware configuration of pulse wave estimation device 1a according to embodiment 2 is the same as the hardware configuration of pulse wave estimation device 1 described in embodiment 1 using Figures 12A and 12B, and therefore is not shown in the drawings. In embodiment 2, the functions of captured image acquisition unit 11, skin area detection unit 12, face direction estimation unit 13, measurement area setting unit 14, luminance signal extraction unit 15, luminance signal selection unit 16a, pulse wave estimation unit 17, and output unit 18 are realized by processing circuit 101. That is, pulse wave estimation device 1a includes processing circuit 101 for extracting, from the skin area of ​​a captured image, a luminance signal containing a pulse wave component sufficient to estimate a person's pulse wave, and for controlling estimation of the subject's pulse wave from the extracted luminance signal.

[0104] Processing circuit 101 reads and executes programs stored in memory 105 to perform the functions of captured image acquisition unit 11, skin area detection unit 12, facial direction estimation unit 13, measurement area setting unit 14, luminance signal extraction unit 15, luminance signal selection unit 16a, pulse wave estimation unit 17, and output unit 18. That is, pulse wave estimation device 1a includes memory 105 for storing a program that, when executed by processing circuit 101, results in the execution of steps ST1 to ST7 of FIG. 15 . The program stored in memory 105 can also be said to cause a computer to execute the processing procedures or methods of captured image acquisition unit 11, skin area detection unit 12, facial direction estimation unit 13, measurement area setting unit 14, luminance signal extraction unit 15, luminance signal selection unit 16a, pulse wave estimation unit 17, and output unit 18. A storage unit (not shown) is, for example, configured with memory 105. The pulse wave estimation device 1a also includes devices such as an image capture device 2, an input interface device 102 for wired or wireless communication, and an output interface device 103.

[0105] In the second embodiment, the subject is a driver of a vehicle, but this is merely an example. The subject may be a passenger other than the driver of the vehicle.

[0106] In the second embodiment, pulse wave estimation device 1a is an in-vehicle device, and captured image acquisition unit 11, skin area detection unit 12, face direction estimation unit 13, measurement area setting unit 14, luminance signal extraction unit 15, luminance signal selection unit 16a, pulse wave estimation unit 17, and output unit 18 are provided in the in-vehicle device. However, the present invention is not limited to this. Some of captured image acquisition unit 11, skin area detection unit 12, face direction estimation unit 13, measurement area setting unit 14, luminance signal extraction unit 15, luminance signal selection unit 16a, pulse wave estimation unit 17, and output unit 18 may be mounted in the in-vehicle device of the vehicle, and the rest may be provided in a server connected to the in-vehicle device via a network, so that the in-vehicle device and the server form a system. In addition, the captured image acquisition unit 11, skin area detection unit 12, face direction estimation unit 13, measurement area setting unit 14, luminance signal extraction unit 15, luminance signal selection unit 16a, pulse wave estimation unit 17, and output unit 18 may all be provided in the server.

[0107] The pulse wave estimation device 1a according to the second embodiment is not limited to an in-vehicle device mounted on a vehicle, but can also be applied to, for example, a moving body other than a vehicle or a home appliance. The subject is not limited to a vehicle occupant, but can be various people.

[0108] In the second embodiment described above, the measurement region setting unit 14 in the pulse wave estimation device 1a determines the measurement region ri(k) to be set in the region corresponding to the skin region on the captured image based on information indicating the target face direction range, layout information, and reference measurement region information. However, this is merely an example. For example, if the target face direction range is the range of all possible face directions of the subject within the range that the imaging device 2 can capture, and it is assumed that the imaging device 2 always captures the subject at a reference position and reference direction, the measurement region setting unit 14 does not need to consider the target face direction range and layout information. In this case, for example, an administrator or the like may determine in advance, based on the captured image of the potential subject captured by the imaging device 2, a measurement region ri(k) that can be used to extract a pulse wave original signal wi(t) containing the subject's pulse wave component, and generate information indicating the determined measurement region ri(k) as reference measurement region information and store it in a storage unit or the like. The measurement area setting unit 14 does not take into account the information indicating the target face direction range and the layout information, and simply sets the measurement area ri(k) defined in the reference measurement area information to an area corresponding to the skin area on the captured image.

[0109] As described above, according to the second embodiment, the pulse wave estimation device 1 a includes a captured image acquisition unit 11 that acquires captured images of a person (subject) frame by frame, a skin region detection unit 12 that detects the person's skin region from the captured image, a measurement region setting unit 14 that sets a measurement region ri(k) that can be used to extract a pulse wave source signal wi(t) that indicates a change in brightness in a region corresponding to the skin region on the captured image and that contains a pulse wave component of the person, a face direction estimation unit 13 that estimates the face direction of the person frame by frame based on the captured image, and a pulse wave source signal ri(k) that can be used to extract a pulse wave source signal wi(t) from the measurement region ri(k) set by the measurement region setting unit 14. The pulse wave estimation device 1a is configured to include a luminance signal extraction unit 15 that sets a used measurement region ri(k) used to extract wi(t) and extracts a time-series pulse wave original signal wi(t) based on luminance changes in the set used measurement region ri(k), a luminance signal selection unit 16a that selects a time-series estimation pulse wave original signal wi(t) to be used for estimating the person's pulse wave from the time-series pulse wave original signals wi(t) extracted by the luminance signal extraction unit 15 during a pulse wave estimation target period, taking into account the person's facial orientation estimated by the facial orientation estimation unit 13, and a pulse wave estimation unit 17 that estimates the person's pulse wave based on the time-series estimation pulse wave original signal wi(t) selected by the luminance signal selection unit 16a. Therefore, the pulse wave estimation device 1a can prevent a decrease in the accuracy of estimating the subject's pulse wave due to, for example, the skin region of the person whose pulse wave is to be estimated, i.e., the subject, not being captured in the captured image or the skin region being overshadowed.

[0110] More specifically, in pulse wave estimation device 1a, luminance signal extraction unit 15 extracts pulse wave original signals wi(t) from measurement areas ri(k) set by measurement area setting unit 14, the measurement area ri(k) corresponding to the facial orientation of the person (subject) estimated by facial orientation estimation unit 13 based on the captured image in which measurement area ri(k) is set, using this measurement area as the used measurement area ri(k). Luminance signal selection unit 16a calculates the distribution rate of the used measurement area ri(k) from which luminance signal extraction unit 15 extracted pulse wave original signals wi(t) during the pulse wave estimation target period, and selects, as the time-series pulse wave original signals wi(t), the time-series pulse wave original signals wi(t) extracted from the used measurement area ri(k) whose appearance frequency is equal to or greater than the area determination threshold, as the time-series estimated pulse wave original signals wi(t). Therefore, the pulse wave estimation device 1a can prevent a decrease in the accuracy of estimating the pulse wave of a subject due to, for example, the skin area of ​​the person whose pulse wave is to be estimated, i.e., the subject, not being captured in the captured image, or the occurrence of so-called shadowing on that skin area.

[0111] Embodiment 3 In this embodiment, a time-series pulse wave source signal for estimation is selected using a method different from that of embodiments 1 and 2. In the following embodiment 3, as in embodiments 1 and 2, it is assumed that the pulse wave estimation device is mounted on a vehicle and the subject is the driver of the vehicle.

[0112] FIG. 17 is a diagram showing an example configuration of a pulse wave estimation device 1b according to embodiment 3. Regarding the configuration of pulse wave estimation device 1b according to embodiment 3, the same components as those of pulse wave estimation device 1 described in embodiment 1 using FIG. 1 are denoted by the same reference numerals, and redundant description will be omitted. Pulse wave estimation device 1b according to embodiment 3 differs from pulse wave estimation device 1 according to embodiment 1 in that it includes a weight setting unit 19. Furthermore, in pulse wave estimation device 1b according to embodiment 3, the specific operations of luminance signal extraction unit 15a and luminance signal selection unit 16b differ from the specific operations of luminance signal extraction unit 15 and luminance signal selection unit 16, respectively, in pulse wave estimation device 1 according to embodiment 1.

[0113] The weight setting unit 19 sets a weighting coefficient for each measurement region ri(k) set by the measurement region setting unit 14 in the captured image in which the face direction of the subject has been estimated, based on the face direction of the subject estimated on a frame-by-frame basis by the face direction estimation unit 13. Note that in the third embodiment, the face direction estimation unit 13 outputs face direction information F(k) to the weight setting unit 19. Furthermore, the measurement region setting unit 14 outputs the generated measurement region information R(k) to the luminance signal extraction unit 15 and the weight setting unit 19.

[0114] FIG. 18 is a diagram illustrating an example of a weighting coefficient for each measurement region ri(k) that the weighting unit 19 sets based on the face direction of the subject estimated by the face direction estimation unit 13 on a frame-by-frame basis in the third embodiment. The weighting unit 19 sets the weighting coefficient based on the face direction of the subject, for example, so that the weight of the measurement region ri(k) closer to the image capture device 2 is increased. In the example shown in FIG. 18 , the weighting unit 19 sets the weighting coefficient for each measurement region ri(k) so that "the weight of the measurement region ri(k) indicated by W1 > the weight of the measurement region ri(k) indicated by W2 > the weight of the measurement region ri(k) indicated by W3." The weighting unit 19 can determine the positional relationship between the measurement region ri(k) and the image capture device 2 based on the layout information and face direction information F(k). In other words, the weighting unit 19 can select the measurement region ri(k) closer to the image capture device 2 and assign a weighting coefficient based on the layout information and face direction information F(k).

[0115] The weight setting unit 19 outputs information about the weight coefficient set for each measurement region ri(k) (hereinafter referred to as "weight information") to the luminance signal selecting unit 16b. The weight information is information that associates each frame Im(k) of the captured image, the identification number of each frame Im(k) of the captured image, and the weight coefficient for each measurement region ri(k).

[0116] The luminance signal extraction unit 15a sets a use measurement region ri(k) from among the measurement regions ri(k) set by the measurement region setting unit 14. In the third embodiment, the luminance signal extraction unit 15a sets the measurement region ri(k) set by the measurement region setting unit 14 as the use measurement region ri(k). Then, based on the frame Im(k) of the captured image acquired by the captured image acquisition unit 11 and the measurement region information R(k) output from the measurement region setting unit 14, the luminance signal extraction unit 15a extracts a pulse wave original signal wi(t) indicating a change in luminance during a pulse wave estimation target period, in other words, a period corresponding to the number of frames Tp, from each of the set use measurement regions ri(k) from among the multiple measurement regions ri(k) on frame Im(k) indicated by the measurement region information R(k). The method by which the luminance signal extracting unit 15a extracts the pulse wave original signal wi(t) indicating the change in luminance during the pulse wave estimation period from each of the set used measurement regions ri(k) is similar to the method by which the luminance signal extracting unit 15 extracts the pulse wave original signal wi(t) indicating the change in luminance during the pulse wave estimation period from each of the set used measurement regions ri(k), as already described in embodiment 1, and therefore will not be described again. The luminance signal extracting unit 15a generates pulse wave original signal information W(t) indicating the pulse wave original signal wi(t) for each used measurement region ri(k) and outputs the generated pulse wave original signal information W(t) to the luminance signal selecting unit 16b.

[0117] Based on the pulse wave source signal information W(t) output from the luminance signal extraction unit 15a and taking into consideration the face direction of the subject estimated by the face direction estimation unit 13, the luminance signal selection unit 16b selects a time-series estimation pulse wave source signal wi(t) to be used for estimating the subject's pulse wave from among the time-series pulse wave source signals wi(t) extracted by the luminance signal extraction unit 15a during the pulse wave estimation target period.

[0118] The method of selecting the estimated pulse wave original signal wi(t) by the luminance signal selector 16b will now be described. In embodiment 3, the luminance signal selector 16b optimizes the pulse wave original signal wi(t) extracted by the luminance signal extractor 15a based on the weighting coefficient set by the weighting unit 19 for the used measurement region ri(k) from which the pulse wave original signal wi(t) was extracted. Note that the luminance signal selector 16b can identify the weighting coefficient set by the weighting unit 19 for the used measurement region ri(k) from which the pulse wave original signal wi(t) was extracted by matching the pulse wave original signal information W(t) with the weighting information using, for example, the identification number of the frame Im(k) of the captured image and the used measurement region ri(k) as keys.

[0119] For example, when the weighting coefficient for the used measurement region ri(k) from which the pulse wave source signal wi(t) extracted by the luminance signal extraction unit 15a was extracted is small, the luminance signal selection unit 16b performs the optimization by correcting the pulse wave source signal wi(t) to be smaller. The amount by which the pulse wave source signal wi(t) is reduced depending on the weighting coefficient is predetermined. Furthermore, for example, when the weighting coefficient for the used measurement region ri(k) from which the pulse wave source signal wi(t) was extracted by the luminance signal extraction unit 15a was extracted is small, the luminance signal selection unit 16b performs the optimization by selecting the pulse wave source signal wi(t) to be blank (set to zero). The weighting coefficient for blanking the pulse wave source signal wi(t) is predetermined.

[0120] The luminance signal selector 16b then selects the optimized time-series pulse wave source signal wi(t) as the time-series estimated pulse wave source signal wi(t). The luminance signal selector 16b outputs the selected time-series pulse wave source signal information W(t) to the pulse wave estimator 17.

[0121] The operation of pulse wave estimation device 1b according to embodiment 3 will now be described. Fig. 19 is a flowchart for explaining the operation of pulse wave estimation device 1b according to embodiment 3. For example, when the vehicle power is turned on, pulse wave estimation device 1b repeats the processing shown in the flowchart of Fig. 19 until the vehicle power is turned off.

[0122] The specific operations of steps ST1 to ST4 and step ST7 performed by pulse wave estimation device 1b are similar to the specific operations of steps ST1 to ST4 and step ST7 already explained using the flowchart of FIG. 9 in embodiment 1, and therefore the same step numbers are used and redundant explanations will be omitted.

[0123] The weight setting unit 19 sets a weighting coefficient for each measurement region r i (k) set by the measurement region setting unit 14 in step ST3 in the captured image in which the face direction of the subject has been estimated, based on the face direction of the subject estimated on a frame-by-frame basis by the face direction estimation unit 13 in step ST4 (step ST41). The weight setting unit 19 outputs the weight information to the luminance signal selection unit 16 b.

[0124] Based on the frame Im(k) of the captured image acquired by the captured image acquisition unit 11 in step ST1 and the measurement region information R(k) output from the measurement region setting unit 14 in step ST3, the luminance signal extraction unit 15a extracts a luminance signal indicating a change in luminance during a pulse wave estimation target period, in other words, a pulse wave source signal wi(t), from each of the multiple measurement regions ri(k) in the frame Im(k) indicated by the measurement region information R(k) (step ST5a). The luminance signal extraction unit 15a generates pulse wave source signal information W(t) indicating the extracted pulse wave source signal wi(t). The luminance signal extraction unit 15a outputs the generated pulse wave source signal information W(t) to the luminance signal selection unit 16.

[0125] Based on the pulse wave original signal information W(t) output from the luminance signal extraction unit 15a in step ST5, the luminance signal selection unit 16b selects an estimated pulse wave original signal wi(t) in consideration of the subject's facial orientation estimated by the facial orientation estimation unit 13 in step ST4 (step ST6b). The luminance signal selection unit 16b outputs the selected pulse wave original signal information W(t) to the pulse wave estimation unit 17.

[0126] 19, the processing of steps ST4 to ST41 is performed after the processing of step ST3, but this is merely an example. For example, the order of the processing of step ST4 and the processing of step ST3 may be reversed, or the processing of steps ST2 to ST3 and the processing of step ST4 may be performed in parallel. The processing of step ST4 may be performed after the processing of step ST1 and before the processing of step ST41 is performed.

[0127] 20 is a flowchart for explaining the details of step ST5a in FIG. 19 . The luminance signal extraction unit 15a sets the measurement region ri(k) set by the measurement region setting unit 14 in step ST3 in FIG. 19 as the used measurement region ri(k) (step ST521). Then, based on the frame Im(k) of the captured image acquired by the captured image acquisition unit 11 in step ST1 in FIG. 9 and the measurement region information R(k) output from the measurement region setting unit 14 in step ST3 in FIG. 9 , the luminance signal extraction unit 15a extracts a time-series pulse wave source signal wi(t) during the pulse wave estimation target period from each used measurement region ri(k) in frame Im(k) (step ST522). After extracting the pulse wave source signal wi(t), the luminance signal extraction unit 15a generates pulse wave source signal information W(t) indicating the extracted time-series pulse wave source signal wi(t). The luminance signal extractor 15a outputs the generated pulse wave source signal information W(t) to the luminance signal selector 16b.

[0128] 21 is a flowchart for explaining the details of step ST6b in FIG. The luminance signal selector 16b optimizes the time-series pulse wave source signal wi(t) extracted by the luminance signal extractor 15a during the pulse wave estimation period based on the weighting coefficient set by the weight setting unit 19 for the used measurement region ri(k) from which the time-series pulse wave source signal wi(t) was extracted (step ST621). The luminance signal selector 16b selects the optimized time-series pulse wave source signal wi(t) as the time-series estimated pulse wave source signal wi(t) (step ST622). The luminance signal selector 16b outputs the selected pulse wave source signal information W(t) to the pulse wave estimation unit 17.

[0129] In this way, pulse wave estimation device 1b sets measurement region ri(k) in an area corresponding to the skin area on the captured image that can be used to extract pulse wave source signal wi(t) that indicates a change in brightness and that includes the subject's pulse wave component. Specifically, pulse wave estimation device 1a determines measurement region ri(k) to be set in the area corresponding to the skin area on the captured image based on information indicating the target face direction range, layout information, and reference measurement area information, and sets the determined measurement region ri(k). Pulse wave estimation device 1b sets a usage measurement region ri(k) from among measurement regions ri(k), extracts a time-series pulse wave source signal wi(t) based on luminance changes in usage measurement region ri(k), and then selects a time-series estimation pulse wave source signal wi(t) to be used to estimate the subject's pulse wave from the time-series pulse wave source signals wi(t) extracted during a pulse wave estimation target period, taking into account the subject's facial orientation estimated frame by frame based on the captured images. Pulse wave estimation device 1b estimates the subject's pulse wave based on the selected time-series estimation pulse wave source signal wi(t). In detail, pulse wave estimation device 1b includes a weight setting unit 19 that sets a weighting factor for each measurement region ri(k) set by measurement region setting unit 14 in an image in which the facial direction of a person (subject) has been estimated, based on the facial direction of the person estimated by facial direction estimation unit 13 on a frame-by-frame basis. Luminance signal extraction unit 15a extracts a pulse wave original signal wi(t) using the measurement region ri(k) set by measurement region setting unit 14 as the used measurement region ri(k). Luminance signal selection unit 16b optimizes the pulse wave original signal wi(t) extracted by luminance signal extraction unit 15a based on the weighting factor set by weight setting unit 19 for the used measurement region ri(k) from which pulse wave original signal wi(t) was extracted, and selects this time-series pulse wave original signal wi(t) as the time-series estimated pulse wave original signal wi(t). This allows pulse wave estimation device 1b to extract a luminance signal containing sufficient pulse wave components to estimate a person's pulse wave, in other words, a pulse wave source signal wi(t), from the skin area on the captured image.As a result, the pulse wave estimation device 1b can prevent a decrease in the accuracy of estimating the pulse wave of a subject due to, for example, the skin area of ​​the person whose pulse wave is to be estimated, i.e., the subject, not being captured in the captured image, or the occurrence of so-called shadow cast on that skin area.

[0130] The hardware configuration of pulse wave estimation device 1b according to embodiment 3 is the same as the hardware configuration of pulse wave estimation device 1 described in embodiment 1 using Figures 12A and 12B, and therefore is not shown in the drawings. In embodiment 3, the functions of captured image acquisition unit 11, skin area detection unit 12, face direction estimation unit 13, measurement area setting unit 14, luminance signal extraction unit 15a, luminance signal selection unit 16b, pulse wave estimation unit 17, output unit 18, and weight setting unit 19 are realized by processing circuit 101. That is, pulse wave estimation device 1b includes processing circuit 101 for extracting, from the skin area of ​​a captured image, a luminance signal containing a pulse wave component sufficient to estimate a person's pulse wave, and for controlling estimation of the subject's pulse wave from the extracted luminance signal.

[0131] Processing circuit 101 reads and executes programs stored in memory 105 to perform the functions of captured image acquisition unit 11, skin region detection unit 12, face direction estimation unit 13, measurement region setting unit 14, luminance signal extraction unit 15a, luminance signal selection unit 16b, pulse wave estimation unit 17, output unit 18, and weight setting unit 19. In other words, pulse wave estimation device 1b includes memory 105 for storing programs that, when executed by processing circuit 101, result in the execution of steps ST1 to ST7 in FIG. 19 described above. It can also be said that the programs stored in memory 105 cause a computer to execute the processing procedures or methods of captured image acquisition unit 11, skin region detection unit 12, face direction estimation unit 13, measurement region setting unit 14, luminance signal extraction unit 15a, luminance signal selection unit 16b, pulse wave estimation unit 17, output unit 18, and weight setting unit 19. The storage unit (not shown) is, for example, a memory 105. Pulse wave estimation device 1b also includes an input interface device 102 and an output interface device 103 that communicate with devices such as image capture device 2 via wired or wireless communication.

[0132] In the third embodiment, the subject is a driver of a vehicle, but this is merely an example. The subject may be a passenger other than the driver of the vehicle.

[0133] In the third embodiment, pulse wave estimation device 1b is an in-vehicle device, and captured image acquisition unit 11, skin area detection unit 12, face direction estimation unit 13, measurement area setting unit 14, luminance signal extraction unit 15a, luminance signal selection unit 16b, pulse wave estimation unit 17, output unit 18, and weight setting unit 19 are provided in the in-vehicle device. However, the present invention is not limited to this. Some of captured image acquisition unit 11, skin area detection unit 12, face direction estimation unit 13, measurement area setting unit 14, luminance signal extraction unit 15a, luminance signal selection unit 16b, pulse wave estimation unit 17, output unit 18, and weight setting unit 19 may be mounted in the in-vehicle device of the vehicle, and the rest may be provided in a server connected to the in-vehicle device via a network, so that the in-vehicle device and the server form a system. In addition, the captured image acquisition unit 11, skin area detection unit 12, face direction estimation unit 13, measurement area setting unit 14, luminance signal extraction unit 15a, luminance signal selection unit 16b, pulse wave estimation unit 17, output unit 18, and weight setting unit 19 may all be provided on the server.

[0134] The pulse wave estimation device 1b according to the third embodiment is not limited to an in-vehicle device mounted on a vehicle, but can also be applied to, for example, a moving body other than a vehicle or a home appliance. The subject is not limited to a vehicle occupant, but can be various people.

[0135] In the above-described third embodiment, the measurement region setting unit 14 in the pulse wave estimation device 1b determines the measurement region ri(k) to be set in the region corresponding to the skin region on the captured image based on information indicating the target face direction range, layout information, and reference measurement region information. However, this is merely an example. For example, if the target face direction range is the range of all possible face directions of the subject within the range that the imaging device 2 can capture, and it is assumed that the imaging device 2 always captures the subject at a reference position and reference direction, the measurement region setting unit 14 does not need to consider the target face direction range and layout information. In this case, for example, an administrator or the like may determine in advance, based on the captured image of the potential subject captured by the imaging device 2, a measurement region ri(k) that can be used to extract a pulse wave original signal wi(t) containing the subject's pulse wave component, and generate information indicating the determined measurement region ri(k) as reference measurement region information and store it in a storage unit or the like. The measurement area setting unit 14 does not take into account the information indicating the target face direction range and the layout information, and simply sets the measurement area ri(k) defined in the reference measurement area information to an area corresponding to the skin area on the captured image.

[0136] As described above, according to the third embodiment, the pulse wave estimation device 1b includes a captured image acquisition unit 11 that acquires captured images of a person (subject) frame by frame, a skin region detection unit 12 that detects the person's skin region from the captured image, a measurement region setting unit 14 that sets a measurement region ri(k) that can be used to extract a pulse wave source signal wi(t) that indicates a luminance change and that contains a pulse wave component of the person in an area corresponding to the skin region on the captured image, a face direction estimation unit 13 that estimates the face direction of the person frame by frame based on the captured image, and a skin region detection unit 12 that detects the skin region of the person from the captured image. The pulse wave estimation device 1b is configured to include a luminance signal extraction unit 15a that sets a used measurement region ri(k) to be used to extract i(t) and extracts a time-series pulse wave original signal wi(t) based on luminance changes in the set used measurement region ri(k), a luminance signal selection unit 16b that selects a time-series estimation pulse wave original signal wi(t) to be used for estimating the person's pulse wave from the time-series pulse wave original signals wi(t) extracted by the luminance signal extraction unit 15a during a pulse wave estimation target period, taking into account the person's facial direction estimated by the facial direction estimation unit 13, and a pulse wave estimation unit 17 that estimates the person's pulse wave based on the time-series estimation pulse wave original signal wi(t) selected by the luminance signal selection unit 16b. Therefore, the pulse wave estimation device 1b can prevent a decrease in the accuracy of estimating the subject's pulse wave due to, for example, the skin region of the person whose pulse wave is to be estimated, i.e., the subject, not being captured in the captured image or the skin region being overshadowed.

[0137] In detail, pulse wave estimation device 1b includes a weight setting unit 19 that sets a weighting factor for each measurement region ri(k) set by measurement region setting unit 14 in an image in which the face direction of a person (subject) has been estimated, based on the face direction of the person estimated by face direction estimation unit 13 on a frame-by-frame basis. Luminance signal extraction unit 15a extracts a pulse wave original signal wi(t) using the measurement region set by measurement region setting unit 14 as a used measurement region ri(k). Luminance signal selection unit 16b optimizes the pulse wave original signal wi(t) extracted by luminance signal extraction unit 15a based on the weighting factor set by weight setting unit 19 for the used measurement region ri(k) from which pulse wave original signal wi(t) was extracted, and selects the resulting time-series pulse wave original signal wi(t) as an estimated time-series pulse wave original signal wi(t). Therefore, the pulse wave estimation device 1b can prevent a decrease in the accuracy of estimating the pulse wave of a subject due to, for example, the skin area of ​​the person whose pulse wave is to be estimated, i.e., the subject, not being captured in the captured image, or the occurrence of so-called shadowing on that skin area.

[0138] In the first to third embodiments, the pulse wave estimation unit 17 analyzes multiple principal components using a common signal separation technique such as PCA or ICA, generates separated signals representing the analyzed multiple principal components, and estimates the subject's pulse wave based on the separated signals. This is merely an example, and the pulse wave estimation unit 17 may estimate the subject's pulse wave using other methods. For example, the pulse wave estimation unit 17 may perform a Fourier transform on the selected pulse wave original signal information W(t) and calculate the peak frequency in the frequency power spectrum as the pulse rate.

[0139] In the above-described first to third embodiments, it is assumed that the measurement region setting unit 14 sets multiple measurement regions ri(k), and the luminance signal extraction unit 15, 15a extracts the time-series pulse wave source signal wi(t) from the multiple used measurement regions ri(k). However, this is merely an example. In the above-described first to third embodiments, the time-series pulse wave source signal wi(t) may be extracted from a single used measurement region ri(k). However, if the time-series pulse wave source signal wi(t) is extracted from a single used measurement region ri(k), the pulse wave estimation unit 17 estimates the subject's pulse wave using a method that does not use a general signal separation technique, such as PCA or ICA (for example, a method that performs a Fourier transform on the above-described selected pulse wave source signal information W(t)).

[0140] The pulse wave estimation device according to the present disclosure can prevent a situation in which, during noise removal, even the pulse wave signal contained in the luminance signal of the subject's skin area is deemed a noise component and removed, making it impossible to extract the luminance signal of the subject's skin area that should be used to estimate the subject's pulse wave.

[0141] 1, 1a, 1b Pulse wave estimation device, 11 Image acquisition unit, 12 Skin area detection unit, 13 Face direction estimation unit, 14 Measurement area setting unit, 15, 15a Luminance signal extraction unit, 16, 16a, 16b Luminance signal selection unit, 17 Pulse wave estimation unit, 18 Output unit, 19 Weight setting unit, 2 Imaging device, 101 Processing circuit, 102 Input interface device, 103 Output interface device, 104 Processor, 105 Memory.

Claims

1. a captured image acquisition unit that acquires captured images of a person frame by frame; a skin area detection unit that detects a skin area of ​​the person from the captured image; a measurement region setting unit that sets a measurement region that can be used to extract a pulse wave source signal that indicates a luminance change and that includes a pulse wave component of the person, in an area corresponding to the skin region on the captured image; and a face direction estimation unit that estimates a face direction of the person for each frame based on the captured image; a luminance signal extracting unit that sets a used measurement area to be used for extracting the pulse wave source signal from the measurement areas set by the measurement area setting unit, and extracts the pulse wave source signal in time series based on the luminance change in the set used measurement area; a luminance signal selection unit that selects a time-series estimation pulse wave source signal to be used for estimating the pulse wave of the person from the time-series pulse wave source signals extracted by the luminance signal extraction unit during a pulse wave estimation target period, taking into consideration the face direction of the person estimated by the face direction estimation unit; and a pulse wave estimation unit that estimates the pulse wave of the person based on the time-series pulse wave source signal selected by the luminance signal selection unit; A pulse wave estimation device comprising:

2. The measurement area setting unit The measurement area to be set in an area corresponding to the skin area on the captured image is determined based on information indicating a target face direction range, which is a range of face directions of the person for which a pulse wave is to be estimated, layout information indicating a positional relationship between an imaging device that captures the captured image and the assumed person, and reference measurement area information that associates the measurement area from which the pulse wave original signal containing the pulse wave component of the person can be extracted when the imaging device captures the assumed person at a reference position and reference direction with the person's face direction.

2. The pulse wave estimation device according to claim 1.

3. the luminance signal extraction unit extracts the pulse wave original signal from the measurement area set by the measurement area setting unit, using as the used measurement area a measurement area corresponding to the face direction of the person estimated by the face direction estimation unit based on the captured image in which the measurement area is set; The luminance signal selection unit calculates a distribution ratio of the face direction of the person for each frame estimated by the face direction estimation unit during the pulse wave estimation target period, and selects, as the time-series pulse wave source signal for estimation, the time-series pulse wave source signal extracted from the used measurement area commonly set in the captured images from which the face direction of the person whose appearance frequency is equal to or greater than a face direction determination threshold is estimated.

3. The pulse wave estimation device according to claim 1 or 2.

4. the luminance signal extraction unit extracts the pulse wave original signal from the measurement area set by the measurement area setting unit, using as the used measurement area a measurement area corresponding to the face direction of the person estimated by the face direction estimation unit based on the captured image in which the measurement area is set; The luminance signal selection unit calculates a distribution ratio of the used measurement area from which the luminance signal extraction unit extracted the pulse wave source signal during the pulse wave estimation target period, and selects the time-series pulse wave source signal extracted from the used measurement area whose appearance frequency is equal to or greater than a threshold for area determination as the time-series pulse wave source signal for estimation.

3. The pulse wave estimation device according to claim 1 or 2.

5. a weight setting unit that sets a weighting coefficient for each of the measurement areas set by the measurement area setting unit in the captured image in which the face direction of the person has been estimated based on the face direction of the person estimated on a frame-by-frame basis by the face direction estimation unit, the luminance signal extraction unit extracts the pulse wave original signal using the measurement region set by the measurement region setting unit as the used measurement region; The luminance signal selection unit selects, as the estimated pulse wave source signal in time series, the pulse wave source signal extracted by the luminance signal extraction unit after optimization based on the weighting coefficient set by the weight setting unit for the used measurement region from which the pulse wave source signal was extracted.

3. The pulse wave estimation device according to claim 1 or 2.

6. The person is a driver of a vehicle.

3. The pulse wave estimation device according to claim 1 or 2.

7. A captured image acquisition unit acquires a captured image of a person on a frame-by-frame basis; a step of detecting a skin region of the person from the captured image by a skin region detection unit; a measurement region setting unit setting a measurement region in an area corresponding to the skin region on the captured image, the measurement region being usable for extracting a pulse wave source signal indicating a luminance change, the pulse wave source signal including a pulse wave component of the person; a face direction estimation unit estimating a face direction of the person for each frame based on the captured image; a luminance signal extraction unit setting a use measurement area to be used for extracting the pulse wave source signal from the measurement areas set by the measurement area setting unit, and extracting the time-series pulse wave source signal based on the luminance change in the set use measurement area; a luminance signal selecting unit selecting a time-series pulse wave source signal to be used for estimating the pulse wave of the person from the time-series pulse wave source signals extracted by the luminance signal extracting unit during a pulse wave estimation target period, taking into consideration the face direction of the person estimated by the face direction estimating unit; a step in which a pulse wave estimation unit estimates the pulse wave of the person based on the time-series pulse wave source signal for estimation selected by the luminance signal selection unit; A pulse wave estimation method comprising: