Pulse wave estimation device and pulse wave estimation method

Abstract

This pulse wave inference device comprises: a captured image acquisition unit (11) that acquires a captured image; a skin region detection unit (12) that detects a skin region of a person from the captured image; a measurement region setting unit (14) that sets, in a region corresponding to the skin region on the captured image, a measurement region that can be used for extracting a pulse wave source signal including a pulse wave component of the person; a face direction inference unit (13) that infers the direction of the face of the person on the basis of the captured image; a luminance signal extraction unit (15, 15a) that sets a use measurement region and extracts time series pulse wave source signals on the basis of a luminance change in the set use measurement region; a luminance signal selection unit (16, 16a, 16b) that selects a time series inference pulse wave source signal to be used for inferring the pulse wave of the person while taking into consideration the direction of the face of the person inferred by the face direction inference unit (13); and a pulse wave inference unit (17) that infers the pulse wave of the person on the basis of the time series inference pulse wave source signal selected by the luminance signal selection unit (16, 16a, 16b).

Description

Technical Field

[0001] The present disclosure relates to a pulse wave estimation device and a pulse wave estimation method.

Background Art

[0002] Conventionally, based on time-series luminance signals extracted from a plurality of regions (hereinafter referred to as "measurement regions") set in a region including a person's skin (hereinafter referred to as "skin region") in a captured image captured by an imaging device, a technique for estimating a person's pulse wave from minute luminance changes on the surface of the person's skin is known (for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, conventional techniques related to pulse wave estimation, typified by the technique disclosed in Patent Document 1, do not consider the possibility that the skin region may not be captured in the captured image or that a so-called shadow drop may occur where the skin region is shaded, depending on the orientation of the person's face whose pulse wave is to be estimated. If the skin region is not captured in the captured image or a so-called shadow drop occurs, the luminance signal extracted from the skin region may not contain sufficient pulse wave components for estimating the person's pulse wave. Therefore, in the conventional technique, there is a problem that the estimation accuracy of the pulse wave may decrease due to the fact that the skin region of the person whose pulse wave is to be estimated is not captured in the captured image or a so-called shadow drop occurs in the skin region.

[0005] This disclosure was made to solve the above-mentioned problems, and aims to provide a pulse wave estimation device that prevents a decrease in the accuracy of estimating a person's pulse wave due to the skin area of ​​the person to be estimated not being captured in the captured image, or due to so-called shadowing occurring in the skin area. [Means for solving the problem]

[0006] The pulse wave estimation device according to this disclosure comprises: an image acquisition unit that acquires captured images of a person on a frame-by-frame basis; a skin region detection unit that detects skin regions of a person from the captured images; a measurement region setting unit that sets a measurement region in the region corresponding to the skin region on the captured image that can be used to extract a pulse wave source signal that shows a change in brightness and contains a component of a person's pulse wave; a face orientation estimation unit that estimates the orientation of a person's face on a frame-by-frame basis based on the captured images; a brightness signal extraction unit that sets a measurement region to be used to extract a pulse wave source signal from among the measurement regions set by the measurement region setting unit and extracts a time-series pulse wave source signal based on the change in brightness in the set measurement region; a brightness signal selection unit that selects a time-series estimated pulse wave source signal to be used for estimating a person's pulse wave from among the time-series pulse wave source signals extracted by the brightness signal extraction unit during the pulse wave estimation period, taking into account the face orientation of a person estimated by the face orientation estimation unit; and a pulse wave estimation unit that estimates a person's pulse wave based on the time-series estimated pulse wave source signal selected by the brightness signal selection unit. [Effects of the Invention]

[0007] According to this disclosure, it is possible to prevent a decrease in the accuracy of estimating a person's pulse wave due to the skin area of ​​the person whose pulse wave is to be estimated not being captured in the captured image, or due to shadows occurring in that skin area. [Brief explanation of the drawing]

[0008] [Figure 1] This figure shows an example configuration of a pulse wave estimation device according to Embodiment 1. [Figure 2] Figures 2A and 2B are diagrams illustrating an example of the content of reference measurement area information in Embodiment 1. [Figure 3] Figures 3A and 3B illustrate an example of the relationship between the position and orientation of the imaging device and the position and orientation of the assumed subject in Embodiment 1. [Figure 4] Figures 4A and 4B illustrate an example of a setting area determined by the measurement area setting unit in Embodiment 1. [Figure 5] Figures 5A and 5B illustrate another example of the relationship between the position and orientation of the imaging device and the position and orientation of the assumed subject in Embodiment 1. [Figure 6] Figures 6A and 6B illustrate another example of a setting area determined by the measurement area setting unit in Embodiment 1. [Figure 7] Figures 7A, 7B, and 7C illustrate an example of how the measurement area is set by the measurement area setting unit in the pulse wave estimation device according to Embodiment 1. [Figure 8] This figure shows an example of a histogram in Embodiment 1, which shows the distribution ratio of the face orientation in the yaw direction of the subject corresponding to each frame of the captured image, estimated by the face orientation estimation unit during the pulse wave estimation period, calculated by the luminance signal selection unit. [Figure 9] This is a flowchart illustrating the operation of the pulse wave estimation device according to Embodiment 1. [Figure 10] This is a flowchart illustrating the details of the process in step ST5 of Figure 9. [Figure 11] This is a flowchart illustrating the details of the process in step ST6 of Figure 9. [Figure 12] Figures 12A and 12B show an example of the hardware configuration of the pulse wave estimation device according to Embodiment 1. [Figure 13] This figure shows an example configuration of a pulse wave estimation device according to Embodiment 2. [Figure 14]In Embodiment 2, it is a diagram showing an example of a histogram indicating the distribution ratio of the usage measurement area that is the extraction source of the pulse wave original signal extracted by the luminance signal extraction unit in the pulse wave estimation target period calculated by the luminance signal selection unit. [Figure 15] It is a flowchart for explaining the operation of the pulse wave estimation device according to Embodiment 2. [Figure 16] It is a flowchart for explaining the details of step ST6a in FIG. 15. [Figure 17] It is a diagram showing a configuration example of the pulse wave estimation device according to Embodiment 3. [Figure 18] In Embodiment 3, it is a diagram for explaining an example of the weight coefficient for each measurement area set by the weight setting unit based on the face orientation of the subject estimated by the face orientation estimation unit for each frame. [Figure 19] It is a flowchart for explaining the operation of the pulse wave estimation device according to Embodiment 3. [Figure 20] It is a flowchart for explaining the details of step ST5a in FIG. 19. [Figure 21] It is a flowchart for explaining the details of step ST6b in FIG. 19.

Modes for Carrying Out the Invention

[0009] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Embodiment 1. FIG. 1 is a diagram showing a configuration example of a pulse wave estimation device 1 according to Embodiment 1.

[0010] The pulse wave estimation device 1 estimates the pulse wave of a person based on a captured image captured by the imaging device 2 of the person. In the following Embodiment 1, a person whose pulse wave is estimated by the pulse wave estimation device 1 is referred to as a "subject". The pulse wave estimation device 1 acquires an image consisting of a series of frames Im(k) at a predetermined frame rate Fr, capturing at least the area where the skin region, which includes the subject's skin, should exist (hereinafter referred to as the "skin region"). Here, k represents the frame number assigned to each frame. For example, the frame given at the timing following frame Im(k) is frame Im(k+1). In Embodiment 1, the skin area is defined as the area corresponding to the subject's face. However, this is merely an example, and the skin area may be an area other than the subject's face. For example, the skin area may be an area corresponding to parts of the face, such as the subject's eyes, eyebrows, nose, mouth, forehead, cheeks, or chin. Alternatively, the skin area may be an area corresponding to a body part other than the face, such as the subject's head, shoulders, hands, neck, or feet. The skin area may consist of multiple areas.

[0011] The pulse wave estimation device 1 then estimates the subject's pulse wave from a series of frames Im(k-Tp+1) to Im(k) at every specific number of frames Tp, and outputs a pulse wave estimation result P(t), which is information indicating the estimated pulse wave (hereinafter referred to as "pulse wave information"). Specifically, the pulse wave estimation device 1 estimates the subject's pulse wave from a brightness signal based on the brightness change of the subject's skin region in a series of frames Im(k-Tp+1) to Im(k). In Embodiment 1, the series of frames Im(k-Tp+1) to Im(k) are assumed to be captured images acquired from the imaging device 2 during a predetermined period (hereinafter referred to as the "pulse wave estimation target period"). The pulse wave estimation target period is set, for example, by an administrator, as the period during which one pulse wave estimation is performed.

[0012] Here, t represents the output number assigned for every specific number of frames Tp. For example, the pulse wave estimation result given at the next timing after pulse wave estimation result P(t) is pulse wave estimation result P(t+1). The frame number k and output number t are integers greater than or equal to 1. The number of frames Tp is an integer greater than or equal to 2.

[0013] Depending on the orientation of the subject's face, it is possible that the subject's skin area may not be sufficiently captured in the image captured by the imaging device 2. This means that, for example, the subject's skin area may not be captured at all in the image captured by the imaging device 2, or a shadow may fall over that skin area, a phenomenon known as "shadow falloff." For example, if the imaging device 2 is capturing an image of a subject from the front, and the subject's face is facing 90 degrees to the left or right relative to the optical axis of the imaging device 2, the right or left side of the subject's face may not be captured or may be shadowed. In this case, all or part of the subject's skin area may not be detected, and the luminance signal based on the luminance change extracted from that skin area may not sufficiently contain the subject's pulse wave component. As a result, the accuracy of estimating a person's pulse wave may decrease. Therefore, the pulse wave estimation device 1 selects, considering the orientation of the subject's face, which luminance signal based on luminance changes that can be extracted from the subject's skin region will be used to estimate the subject's pulse wave. The pulse wave estimation device 1 selects, considering the orientation of the subject's face, the luminance signal extracted from a region from which it is assumed that a luminance signal containing a sufficient amount of the subject's pulse wave component can be extracted, as the luminance signal to be used to estimate the subject's pulse wave. Then, the pulse wave estimation device 1 estimates the subject's pulse wave from the luminance signal selected considering the orientation of the subject's face.

[0014] The number of subjects (people) included in the captured image may be one or multiple. For the sake of simplicity, in the following Embodiment 1, we will assume that the number of subjects included in the captured image is one.

[0015] The pulse wave estimation result P(t) is output from the pulse wave estimation device 1 to, for example, an alertness estimation device that estimates a person's level of alertness, or an abnormality detection device that detects abnormalities in a person's physical condition. Note that the alertness estimation device and the abnormality detection device are not shown in Figure 1. The alertness level estimation device estimates the decrease in the subject's alertness level based on the pulse wave estimation result P(t) output from the pulse wave estimation device 1. For example, the alertness level estimation device estimates that the subject's alertness level is decreasing when the subject's pulse rate tends to decrease slowly. If the alertness level estimation device estimates that the subject's alertness level is decreasing, it will warn the subject or those around them about the decrease in the subject's alertness level, for example. The abnormality detection device detects abnormal physical conditions in the subject based on the pulse wave estimation result P(t) output from the pulse wave estimation device 1. Examples of abnormal physical conditions include epilepsy or heart disease. For example, the abnormality detection device detects that the subject is in an abnormal state if the subject's pulse wave is rapidly increasing. If the abnormality detection device detects an abnormal physical condition in the subject, it warns the subject or those around them that the subject is in an abnormal state.

[0016] The imaging device 2 comprises an imaging unit (not shown) and an illumination unit (not shown). The illumination unit is composed of, for example, an LED (Light Emitting Diode). The illumination unit illuminates the imaging range of the imaging unit with light. The imaging unit images the imaging range illuminated by the light emitted by the illumination unit. The imaging device 2 may be provided with one illumination unit or multiple illumination units. The imaging device 2 is positioned to capture images of the subject's skin area. In other words, in this case, the imaging device 2 is positioned to capture images of the driver's skin area.

[0017] In the following Embodiment 1, as an example, the pulse wave estimation device 1, imaging device 2, alertness level estimation device (not shown), and abnormality detection device (not shown) are mounted on a vehicle (not shown), and the subject is the vehicle's driver. In other words, the pulse wave estimation device 1 estimates the pulse wave of the vehicle's driver.

[0018] A detailed explanation of the configuration example of the pulse wave estimation device 1 shown in Figure 1 will be provided. As shown in Figure 1, the pulse wave estimation device 1 comprises an image acquisition unit 11, a skin area detection unit 12, a face orientation estimation unit 13, a measurement area setting unit 14, a brightness signal extraction unit 15, a brightness signal selection unit 16, a pulse wave estimation unit 17, and an output unit 18.

[0019] The image acquisition unit 11 acquires the captured image of the subject. More specifically, the image acquisition unit 11 acquires the captured image of the vehicle driver taken by the imaging device 2. The image acquisition unit 11 outputs the acquired image to the skin region detection unit 12 and the face orientation estimation unit 13.

[0020] The skin region detection unit 12 detects the subject's skin region from the frame Im(k) included in the captured image acquired by the image acquisition unit 11. The skin region detection unit 12 can detect the skin region using known means. For example, the skin region detection unit 12 can detect the skin region using a cascade-type face detector that uses Haar-like features. The skin region detection unit 12 generates skin region information S(k) indicating the detected skin region. The skin region information S(k) may include information indicating whether or not a skin region has been detected, and information indicating the position and size of the detected skin region on the captured image. In Embodiment 1, the skin region is represented by a rectangular region on the captured image, and the skin region information S(k) includes information indicating the position and size of the rectangular region on the captured image. Specifically, if the skin region corresponds to the subject's face, the skin region information S(k) indicates, for example, whether or not the subject's face was detected, the center coordinates Fc(Fcx,Fcy) of the rectangle surrounding the subject's face on the captured image, and the width Fcw and height Fch of this rectangle. Whether or not the subject's face was detected is represented, for example, "1" if detected and "0" if not detected. The center coordinates of the rectangle surrounding the face are expressed in the coordinate system of frame Im(k). The upper left corner of frame Im(k) is the origin, the rightward direction of frame Im(k) is the positive x-axis, and the downward direction of frame Im(k) is the positive y-axis. The skin region detection unit 12 outputs the generated skin region information S(k) to the measurement region setting unit 14.

[0021] The face orientation estimation unit 13 estimates the subject's face orientation on a frame-by-frame basis based on the frame Im(k) of the captured image acquired by the image acquisition unit 11. In Embodiment 1, the facial orientation of the subject estimated by the facial orientation estimation unit 13 is an angle calculated with the frontal position of the subject, i.e., the driver in this case, as the reference (0 degrees), regardless of the installation position of the imaging device 2. In Embodiment 1, "front" does not mean strictly frontal, but includes approximately frontal. In Embodiment 1, the facial orientation of the subject estimated by the facial orientation estimation unit 13, with the frontal position of the subject as the reference, is also called the "subject-referenced facial orientation." The face orientation estimation unit 13 can estimate the orientation of the subject's face in three directions: yaw [deg], pitch [deg], and roll [deg]. The face orientation estimation unit 13 can estimate the subject's face orientation, i.e., the subject's reference face orientation, using a method that utilizes a known, pre-trained model (hereinafter referred to as a "machine learning model"), such as Hope-Net. The face orientation estimation unit 13 generates information (hereinafter referred to as "face orientation information") F(k) regarding the estimated face orientation of the subject (face orientation based on the subject). Facial orientation information F(k) is information that associates the estimated subject-based facial orientation with the frame Im(k) of the captured image from which the facial orientation was estimated. The face orientation estimation unit 13 outputs the generated face orientation information F(k) to the brightness signal extraction unit 15.

[0022] The measurement area setting unit 14 sets a plurality of measurement areas in the image area on frame Im(k) corresponding to the skin area indicated by the skin area information S(k) on frame Im(k), based on the frame Im(k) of the image acquired by the image acquisition unit 11 and the skin area information S(k) output by the skin area detection unit 12, which can be used to extract a luminance signal that indicates a change in luminance and includes the pulse wave component of the subject. In the following explanation, the luminance signal that shows a change in brightness, extracted from multiple measurement areas, is also referred to as the "pulse wave source signal." The area on the image area corresponding to the skin region that will be used as the measurement area is determined based on the range of the subject's face orientation for pulse wave estimation (hereinafter referred to as the "target face orientation range") and the positional relationship between the imaging device 2 and the assumed subject. The target face orientation range is determined as appropriate by the administrator, etc. The administrator, etc. determines the target face orientation range according to the intended use of the pulse wave estimation device 1, etc. The administrator, etc. stores information indicating the target face orientation range in a location accessible to the measurement area setting unit 14, such as a storage unit not shown in the figure.

[0023] For example, information (hereinafter referred to as "reference measurement area information") is generated in advance by an administrator or the like, defining multiple measurement areas from which it is assumed that a pulse wave source signal containing the pulse wave component of the subject can be extracted when the imaging device 2 images the subject at a reference position (hereinafter referred to as "reference position") and orientation (hereinafter referred to as "reference orientation"), and this information is stored in a memory unit or other location accessible to the pulse wave estimation device 1. The reference measurement area information is a table-formatted information that associates the orientation of the subject's face in the captured image with information indicating the measurement area, when the imaging device 2 captures the subject at a reference position and orientation. The information indicating the measurement area is, for example, a number that can identify the measurement area (hereinafter referred to as the "measurement area number"). Each measurement area is assigned a measurement area number. The subjects described above are assumed to be subjects of standard build facing forward in a standard position (hereinafter referred to as "assumed subjects"). Here, the assumed subjects are, for example, drivers of standard build who are seated facing forward in a standard position in a standard driver's seat, without compromising their posture. In Embodiment 1, when a subject (hypothetical subject) is imaged at the reference position and reference orientation defined in the reference measurement area information, the orientation of the subject's (hypothetical subject's) face in the captured image is an angle calculated with the front of the imaging device 2 as the reference (0 degrees). In Embodiment 1, the orientation of the subject's (hypothetical subject's) face with respect to the front of the imaging device 2, as defined in the reference measurement area, is also called the "camera-referenced face orientation."

[0024] Furthermore, information indicating the positional relationship between the actually installed imaging device 2 and the assumed subject (hereinafter referred to as "layout information") is generated in advance by the administrator or other relevant personnel and stored in the memory unit. For example, when installing the imaging device 2, the administrator or other relevant personnel generates the layout information considering the position of the assumed subject and stores it in the memory unit or other relevant personnel.

[0025] In other words, the information indicating the target face orientation range, the reference measurement area information, and the layout information are generated in advance by an administrator or the like and stored in a memory unit or the like.

[0026] The measurement area setting unit 14 sets multiple measurement areas based on multiple measurement areas defined in the reference measurement area information, based on the layout information and information indicating the target face orientation range. Specifically, the measurement area setting unit 14 corrects the camera-based face orientation defined in the reference measurement area information to the subject-based face orientation based on the layout information, and sets multiple measurement areas based on the multiple measurement areas defined in the reference measurement area information, based on the target face orientation range. More specifically, the measurement area setting unit 14 calculates the difference between the position and orientation of the imaging device 2 and the reference position and orientation based on the layout information. Based on the calculated difference, it offsets the assumed subject's face orientation (camera-referenced face orientation) in the reference measurement area information to match the assumed subject's face orientation (camera-referenced face orientation) in frame Im(k) of the image captured at the reference position and orientation of the imaging device 2 assumed in the reference measurement area information to the assumed subject's face orientation (subject-referenced face orientation) in frame Im(k) of the image captured at the current position and orientation of the imaging device 2. Then, based on the target face orientation range, the measurement area setting unit 14 determines multiple measurement areas to be set from among the multiple measurement areas defined in the reference measurement area information.

[0027] Here, we will explain with specific examples how the measurement area setting unit 14 determines multiple measurement areas to be set based on layout information, information indicating the target face orientation range, and reference measurement area information. For convenience, in the following examples, the subject will be assumed to change their face orientation only in the yaw direction.

[0028] First, we will explain the details of the prerequisite reference measurement area information using diagrams. As mentioned above, it is assumed that the subject changes their face orientation only in the yaw direction, and the reference measurement area information is information that associates the subject's face orientation in the yaw direction with information indicating the measurement area. Figure 2 is a diagram illustrating an example of the content of reference measurement area information in Embodiment 1. Figure 2 illustrates an example of the content of reference measurement area information when the subject's face orientation in the yaw direction from "-30 degrees to +30 degrees" is associated with information indicating the measurement area. In Embodiment 1, the yaw direction of the subject's face is defined as "0 degrees" when the subject's face is facing forward, with negative angles indicating the angle as the face is turned to the left from the front, and positive angles indicating the angle as the face is turned to the right from the front.

[0029] For example, as shown in Figure 2A, the reference measurement area information includes information indicating all 24 measurement areas, numbered (1) to (24). As mentioned above, in the reference measurement area information, the measurement area number is associated with the subject's face orientation (camera-referenced face orientation), but the information indicating face orientation is omitted in Figure 2A. In Figure 2A, for clarity, all 24 measurement regions set in the reference measurement region information are shown in a way that shows their positional relationship on the captured image. Of the 24 measurement areas set in the reference measurement area information shown in Figure 2A, measurement areas (1) to (12) represent the measurement areas set in the skin area corresponding to the left cheek. Also, of the 24 measurement areas set in the reference measurement area information shown in Figure 2A, measurement areas (13) to (24) represent the measurement areas set in the skin area corresponding to the right cheek. When the imaging device 2 captures an image of the subject's face, the subject's left cheek is captured on the right side of the image, and the subject's right cheek is captured on the left side of the image.

[0030] Although not shown in Figure 2A, the reference measurement area information includes information indicating each measurement area, and more specifically, each measurement area number is assigned information that allows identification of which area of ​​the skin region the measurement area is set in. This information that allows identification of which area of ​​the skin region the measurement area is set in includes, for example, information indicating the four vertices of the measurement area. In Embodiment 1, the measurement area is assumed to be a quadrilateral. The information indicating the four vertices is represented, for example, by landmarks of facial organs such as the outer corner of the eye, the inner corner of the eye, the nose, and the mouth, or by auxiliary landmarks. In the reference measurement area information, the information indicating the four vertices of the measurement area includes, for example, information indicating which facial organ landmark it is, or which auxiliary landmark it is on the line segment between which facial organ landmarks. Details of facial organ landmarks and auxiliary landmarks will be described later.

[0031] Figure 2B shows the correspondence between the frames of images of the hypothetical subject captured by the imaging device 2 (indicated as "I" in Figure 2B) and the measurement area determined by the administrator, etc. (indicated as "M" in Figure 2B), for every 10 degrees of the hypothetical subject's face orientation in the yaw direction. Here, administrators shall set the measurement area at 10-degree intervals in the yaw direction of the subject's face. Note that this is merely an example, and administrators may set the measurement area in appropriate units, such as 1-degree increments. Furthermore, in Figure 2B, the range of possible yaw-direction facial orientations for the hypothetical subject is "-30 degrees to +30 degrees." This is determined by the administrator or other relevant personnel based on the resolution of the imaging device 2.

[0032] Even when the imaging device 2 images a hypothetical subject at a reference position and orientation, the range in which the imaging device 2 can image the hypothetical subject, or more specifically, the size of the measurement area imaged by the imaging device 2, changes depending on the orientation of the hypothetical subject's face. When the size of the measurement area changes, the amount of pulse wave components included in the signal indicating brightness changes extracted from that measurement area also changes. The administrator or other relevant person sets in advance, according to the facial orientation of the assumed subject (facial orientation relative to the camera), a measurement area in which a brightness signal containing sufficient pulse wave components to estimate the pulse wave of the assumed subject is extracted when the imaging device 2 images the assumed subject from a reference position and in a reference orientation. In Embodiment 1, the reference position of the imaging device 2 is specifically the position in real space where the horizontal and vertical coordinates indicating the position of the imaging device 2 are the same as the horizontal and vertical coordinates indicating the position of the assumed subject. In Embodiment 1, the subject's position is indicated by the position of the center of the subject's face. Real space is represented by a three-dimensional coordinate system with the x-axis parallel to the vehicle's width direction, the y-axis parallel to the vehicle's height direction, and the z-axis parallel to the vehicle's length direction, or in other words, the direction of travel of the vehicle. In Embodiment 1, "parallel" is not limited to strictly parallel, but includes approximately parallel. That is, the x-coordinate and y-coordinate of the coordinate system indicating the reference position of the imaging device 2 are the same as the x-coordinate and y-coordinate of the coordinate system indicating the position of the assumed subject.

[0033] For example, if the hypothetical subject's face orientation in the yaw direction is "-30 degrees", then, as shown in Figure 2B, the left cheek of the hypothetical subject will be almost completely obscured in the captured image. In other words, the skin area of ​​the left cheek of the hypothetical subject will not be sufficiently captured in the captured image. In this case, the size of the measurement areas numbered (1) to (12) set in the skin area corresponding to the left cheek of the assumed subject on the captured image will be reduced, and almost no area will be secured. As mentioned above, the measurement area is the region that can be used to extract the pulse wave source signal that shows the change in brightness. If the measurement area is small, the components included in the pulse wave source signal extracted from that measurement area will contain many noise components, and it may not be possible to extract the pulse wave source signal that contains the pulse wave component. The same applies if the measurement area is in a so-called shadow. If the pulse wave source signal containing the pulse wave component is not extracted, the accuracy of estimating the subject's pulse wave based on that source signal will decrease. In other words, in order to ensure that the pulse wave source signal is extracted without causing a decrease in the accuracy of the subject's pulse wave estimation, the measurement region from which the pulse wave source signal is extracted must be a region from which it is assumed that a pulse wave source signal containing the subject's pulse wave component can be extracted.

[0034] For example, in the example shown in Figure 2B, if the assumed subject's face orientation in the yaw direction is "-30 degrees", the measurement areas with sufficient size are measurement areas numbered (13) to (24). Therefore, the administrator decides that if the imaging device 2 images the assumed subject at a reference position and reference orientation, and the subject's face orientation in the yaw direction is "-30 degrees", then measurement areas (13) to (24) will be set. Here, the administrator shall determine the measurement area based on its size, for example, that the size of the measurement area is above a predetermined threshold. However, this is only one example. For example, the administrator may determine the measurement area based on the brightness of the measurement area, for example, that the brightness of the measurement area is above a predetermined threshold. The administrator should determine the measurement area in which it is assumed that the pulse wave source signal containing the subject's pulse wave component can be extracted, according to the subject's face orientation in the yaw direction. When the yaw direction of the assumed subject's face is between -20 and +30 degrees, the administrators will determine the measurement area for that face direction in the same way as when the yaw direction of the assumed subject's face is between -30 degrees, by selecting a measurement area that is sufficiently large. As a result, the administrators will set a total of 24 measurement areas for the yaw direction of the assumed subject's face between -30 and +30 degrees, excluding duplicates, as explained using Figure 2A.

[0035] Then, the administrator sets the measurement area according to the yaw direction of the assumed subject's face, generates reference measurement area information, and stores it in a memory unit or similar. In the example above, the administrator would measure the subject's yaw direction face orientation at "-30 degrees" with measurement area numbers (13) to (24), the subject's yaw direction face orientation at "-20 degrees or less and greater than -30 degrees" with measurement area numbers (1), (7), (14) to (18), (20) to (24), the subject's yaw direction face orientation at "-10 degrees or less and greater than -20 degrees" with measurement area numbers (1) to (2), (7) to (8), (15) to (18), (21) to (24), and the subject's yaw direction face orientation at "greater than -10 degrees and less than +10 degrees". Reference measurement area information is generated by associating the subject's yaw direction face orientation "+10 degrees or more and less than +20 degrees" with measurement area numbers (1) to (4), (7) to (10), (17) to (18), (23) to (24), the subject's yaw direction face orientation "+20 degrees or more and less than +30 degrees" with measurement area numbers (1) to (5), (7) to (11), (18) to (19), the subject's yaw direction face orientation "+30 degrees" with measurement area numbers (1) to (12).

[0036] Assuming that the above-mentioned reference measurement area information, information indicating the target face orientation range, and layout information have been generated in advance, the measurement area setting unit 14 determines the measurement areas to be set based on the layout information, the information indicating the target face orientation range, and the reference measurement area information. Based on the reference measurement area information, the frame Im(k) of the captured image acquired by the image acquisition unit 11, and the skin area information S(k) output by the skin area detection unit 12, it sets the determined multiple measurement areas in the image areas on frame Im(k) that correspond to the skin areas indicated by the skin area information S(k).

[0037] In detail, the measurement area setting unit 14 sets the measurement area that is associated with the target face orientation range in the reference measurement area information as the image area on frame Im(k) that corresponds to the skin area indicated by the skin area information S(k). At this time, the measurement area setting unit 14, based on the layout information, offsets the yaw direction of the assumed subject's face orientation (camera-referenced face orientation) as defined in the reference measurement area information. In other words, it adjusts the yaw direction of the assumed subject's face orientation (subject-referenced face orientation) when imaged at the current position and orientation of the imaging device 2 relative to the assumed subject, to match the yaw direction of the assumed subject's face orientation (camera-referenced face orientation) when imaged at the reference position and reference angle of the imaging device 2, and then determines the measurement area to be set.

[0038] Here, we will explain with some examples the case in which the measurement area setting unit 14 determines the measurement area by offsetting the face orientation of the assumed subject as defined in the reference measurement area information. The content of the reference measurement area information is assumed to be as described using Figure 2B.

[0039] For example, suppose the relationship between the position and orientation of imaging device 2 and the position and orientation of the hypothetical subject is as shown in Figure 3. Here, the hypothetical subject is a subject facing forward, as described above. In Figure 3, "R" indicates the imaging range of imaging device 2. Specifically, as shown in Figure 3A, in real space, the vertical coordinates indicating the position of the imaging device 2 are the same as the vertical coordinates indicating the position of the assumed subject (indicated as "D" in Figure 3A), and as shown in Figure 3B, in real space, the horizontal coordinates indicating the position of the imaging device 2 are the same as the horizontal coordinates indicating the position of the assumed subject (indicated as "D" in Figure 3B). That is, the x and y coordinates of the coordinates indicating the position of the imaging device 2 are the same as the x and y coordinates of the coordinates indicating the position of the assumed subject. Also, the optical axis of the imaging device 2 coincides with the line indicating the direction in front of the assumed subject. There is no difference in the relationship between the direction in front of the assumed subject and the orientation of the optical axis of the imaging device 2. More specifically, the yaw angle formed by the optical axis of the imaging device 2 and the line parallel to the line indicating the direction in front of the assumed subject is 0 degrees. In other words, the position and orientation of imaging device 2 relative to the assumed subject are now the reference position and reference orientation. For example, let's assume that the target face orientation range is currently defined as "-20 degrees to +20 degrees".

[0040] In this case, the measurement area setting unit 14 determines, based on the layout information, that there is no deviation in the position and orientation of the imaging device 2 from the reference position and reference orientation. The measurement area setting unit 14 determines that it is not necessary to offset the face orientation of the assumed subject (face orientation relative to the camera) as defined in the reference measurement area information. The measurement area setting unit 14 determines the measurement area to be set as the measurement area indicated by the measurement area number corresponding to the yaw direction face orientation "-20 degrees to +20 degrees" in the reference measurement area information. For example, in Figure 2B, for each captured image of the subject's face orientation in the yaw direction and the corresponding measurement area, the measurement area corresponding to "-20 degrees to +20 degrees" will be determined as the measurement area to be set (see Figures 4A and 4B. Information regarding measurement areas that have not been determined is shown with shading). In other words, the measurement area setting unit 14 determines, based on the reference measurement area information, that all 20 measurement areas, numbered (1) to (5), (7) to (11), (14) to (18), and (20) to (24), are the measurement areas to be set as the image areas corresponding to the skin area.

[0041] Furthermore, for example, suppose the relationship between the position and orientation of imaging device 2 and the assumed position and orientation of the subject is as shown in Figure 5. In Figure 5, "R" indicates the imaging range of imaging device 2. Specifically, as shown in Figure 5A, in real space, the vertical coordinates indicating the position of the imaging device 2 and the vertical coordinates indicating the position of the hypothetical subject (indicated as "D" in Figure 5A) are the same. However, as shown in Figure 5B, in real space, the horizontal coordinates indicating the position of the imaging device 2 and the horizontal coordinates indicating the position of the hypothetical subject (indicated as "D" in Figure 5B) are different. Here, the imaging device 2 will image the hypothetical subject, who is facing forward, from a position at a yaw angle of "+20 degrees" to the right of the hypothetical subject's front. Regarding the relationship between the hypothetical subject's front direction and the direction of the optical axis of the imaging device 2, the angle between the optical axis of the imaging device 2 and the line parallel to the line indicating the hypothetical subject's front direction is assumed to be "+20 degrees". In other words, the imaging device 2 will image the hypothetical subject from the front when the hypothetical subject is facing 20 degrees to the right. For example, let's assume that the administrator has now determined the target face orientation range to be "-10 degrees to +10 degrees".

[0042] In this case, the measurement area setting unit 14 determines, based on the layout information, that there is a deviation in the position and orientation of the imaging device 2 from the reference position and reference orientation. The measurement area setting unit 14 needs to offset the face orientation of the assumed subject (camera-referenced face orientation) defined in the reference measurement area information so that the positional relationship with the assumed subject when the imaging device 2 images the assumed subject from the front matches the positional relationship assumed in the reference measurement area information. Here, the measurement area setting unit 14 adds "+20 degrees" to the face orientation in the yaw direction of the assumed subject as defined in the reference measurement area information. As a result, the face orientation in the yaw direction of the assumed subject defined in the reference measurement area information, which is "-30 degrees to +30 degrees", can be considered as "-10 degrees to +50 degrees" when aligned with the actual position and orientation of the imaging device 2. In other words, the face orientation based on the subject can be considered as "-10 degrees to +50 degrees". Then, the measurement area setting unit 14 determines the measurement area to be set to correspond to the target face orientation range "-10 degrees to +10 degrees". In this case, the measurement area corresponding to the yaw direction face orientation "-30 degrees to -10 degrees" in the reference measurement area information is determined to be set as the measurement area to be set (see Figures 6A and 6B. Information regarding measurement areas that were not determined is shown with shading). In other words, the measurement area setting unit 14 determines, based on the reference measurement area information, that all 16 measurement areas, numbered (1) to (2), (7) to (8), and (13) to (24), are to be set as the measurement areas to be set for the image areas corresponding to the skin area. ru.

[0043] In the above specific example, it was assumed that the subject only changes their face orientation in the yaw direction, and that the reference measurement area information is information that associates the face orientation in the yaw direction with information indicating the measurement area. However, the reference measurement area is also generated in the pitch and roll directions in the same way as in the yaw direction. The measurement area setting unit 14 should determine the measurement area to be set by offsetting the face orientation of the assumed subject (camera-referenced face orientation) as defined in the reference measurement area information, based on the deviation in the pitch or roll direction of the position and orientation of the imaging device 2 relative to the assumed subject from the reference position and reference orientation, in the same way as in the case of the yaw direction described above.

[0044] As described above, the measurement area setting unit 14 determines multiple measurement areas to be set based on the multiple measurement areas defined in the reference measurement area information, using the layout information and the information indicating the target face orientation range. Then, based on the frame Im(k) of the captured image acquired by the image acquisition unit 11 and the skin area information S(k) output by the skin area detection unit 12, it sets the multiple measurement areas determined based on the layout information in the image area on frame Im(k) that corresponds to the skin area indicated by the skin area information S(k). The measurement area setting unit 14 can acquire the captured image acquired by the image acquisition unit 11 via the skin area detection unit 12.

[0045] Here, Figures 7A, 7B, and 7C are diagrams illustrating an example of how the measurement area is set by the measurement area setting unit 14 in the pulse wave estimation device 1 according to Embodiment 1. An example of how the measurement area setting unit 14 sets the measurement area ri(k) will be explained using Figure 7. First, as shown in Figures 7A and 7B, the measurement area setting unit 14 detects Ln (a positive integer) landmarks of facial organs such as the outer corners of the eyes, inner corners of the eyes, nose, and mouth in the skin area sr indicated by the skin area information S(k). In Figures 7A and 7B, the landmarks are shown as circles. The measurement area setting unit 14 sets L(k) to be a vector containing the coordinate values ​​of the detected landmarks. The measurement area setting unit 14 can detect facial features using known methods, such as employing a model called a Constrained Local Model (CLM).

[0046] Next, the measurement area setting unit 14 sets the vertex coordinates of the quadrilateral of the measurement area ri(k) based on the detected landmark. For example, the measurement area setting unit 14 sets the vertex coordinates of the quadrilateral as shown in Figure 7C and sets Rn measurement areas ri(k).

[0047] To illustrate with an example where the measurement area setting unit 14 sets the measurement area ri(k) to the part of the skin area sr corresponding to the cheek, the measurement area setting unit 14 selects the landmark LA1 of the face contour and the landmark LA2 of the nose. The measurement area setting unit 14 should first select the landmark LA2 of the nose, and then select the landmark LA1 of the face contour that is closest to the landmark LA2 of the nose. The measurement area setting unit 14 then sets auxiliary landmarks a1, a2, and a3 so as to divide the line segment between landmark LA1 and landmark LA2 into four equal parts. Similarly, the measurement area setting unit 14 selects landmark LB1 of the face contour and landmark LB2 of the nose. The measurement area setting unit 14 also sets auxiliary landmarks b1, b2, and b3 so as to divide the line segment between landmark LB1 and landmark LB2 into four equal parts. Landmarks LB1 and LB2 can be selected from, for example, face contour or nose landmarks adjacent to landmarks LA1 and LA2, respectively. The measurement area setting unit 14 sets the quadrilateral region enclosed by auxiliary landmarks a1, b1, b2, and a2 as a single measurement area R1. Auxiliary landmarks a1, b1, b2, and a2 each represent the vertex coordinates corresponding to the measurement area R1. Similarly, the measurement area setting unit 14 sets a measurement area R2 enclosed by auxiliary landmarks a2, b2, b3, and a3, and the vertex coordinates of said measurement area R2.

[0048] In the explanation using Figure 7, the concept of how to set the measurement area ri(k) was simply explained. However, in Embodiment 1, as described above, the measurement area setting unit 14 determines the measurement area ri(k) based on a plurality of measurement areas defined in the reference measurement area information, based on the layout information and information indicating the target face orientation range, and sets the determined measurement area ri(k). The measurement area setting unit 14 sets the measurement area ri(k) determined based on the layout information, the information indicating the target face orientation range, and the reference measurement area information, using the method described with reference to Figure 7. The measurement area setting unit 14 can identify which point will be the landmark or auxiliary landmark that will be the vertex of the measurement area ri(k) to be set, based on the reference measurement area information.

[0049] Furthermore, while this example describes setting the measurement area ri(k) to a corresponding part of the cheek, the measurement area setting unit 14 can similarly set the measurement area and the vertex coordinates of the measurement area ri(k) for other parts of the cheek and the skin area sr corresponding to the chin. Although not shown in Figure 7C, the measurement area setting unit 14 can also set the measurement area ri(k) to the part of the subject's skin area sr corresponding to the forehead or the tip of the nose.

[0050] For example, the measurement area setting unit 14 may set the measurement area ri(k) using a method other than CLM. For example, the measurement area setting unit 14 may set the measurement area ri(k) using tracking technology such as a Kanade-Lucas-Tomasi (KLT) tracker. Specifically, the measurement area setting unit 14 may detect the coordinates of facial organ points for the skin area of ​​the first frame Im(1) in a series of frames Im(k-Tp+1)~Im(k) using CLM, and then track the facial organ points using a KLT tracker for the skin area of ​​the next frame Im(2) and subsequent frames, and calculate the facial organ points for the skin area of ​​each frame Im(k). In this case, since detection errors due to tracking accumulate, the measurement area setting unit 14 may perform reset processing such as executing CLM once every few frames to reset the coordinate positions of the facial organ points.

[0051] When the measurement area setting unit 14 sets multiple measurement areas ri(k), it generates measurement area information R(k) that indicates the set multiple measurement areas ri(k). The measurement area information R(k) includes information indicating the position and size of Rn (positive integer) measurement areas ri(k) on the captured image. Each measurement area is defined as measurement area ri(k) (i=1,2,···,Rn). In Embodiment 1, the measurement area ri(k) is a quadrilateral, and the position and size of the measurement area ri(k) are the coordinate values ​​of the four vertices of the quadrilateral on the captured image. The measurement area setting unit 14 outputs the generated measurement area information R(k) to the luminance signal extraction unit 15.

[0052] The luminance signal extraction unit 15 sets a measurement area ri(k) to be used to extract the pulse wave source signal (hereinafter referred to as the "used measurement area") from among the measurement areas ri(k) set by the measurement area setting unit 14. The luminance signal extraction unit 15 then extracts a pulse wave source signal from each of the set measurement areas ri(k) on frame Im(k) indicated by the measurement area information R(k) on frame Im(k), based on the frame Im(k) of the image acquisition unit 11 and the measurement area information R(k) output from the measurement area setting unit 14. This signal represents the change in luminance during the pulse wave estimation period, or in other words, the period corresponding to the number of frames Tp. The pulse wave source signal is the signal that forms the basis of the pulse wave. The pulse wave estimation device 1 uses the pulse wave source signal to estimate the subject's pulse wave. The pulse wave estimation unit 17 performs the estimation of the subject's pulse wave. Details of the pulse wave estimation unit 17 will be described later. The luminance signal extraction unit 15 can acquire the captured image acquired by the image acquisition unit 11 via the skin area detection unit 12 and the measurement area setting unit 14.

[0053] Here, we will describe an example of how the luminance signal extraction unit 15 sets the measurement area ri(k) used in Embodiment 1. The luminance signal extraction unit 15 sets the measurement area ri(k) to be used based, for example, on the face orientation information F(k) output from the face orientation estimation unit 13 and the information for setting the measurement area to be used. The information for setting the measurement area to be used is information that associates the subject's face orientation with the measurement area number. For example, an administrator or the like may generate the information for setting the measurement area to be used in advance and store it in a location accessible to the pulse wave estimation device 1, such as a memory unit.

[0054] The luminance signal extraction unit 15, for example, compares the frame Im(k) of the captured image acquired by the image acquisition unit 11 with the frame Im(k) of the captured image included in the face orientation information F(k), and determines the face orientation of the subject corresponding to the frame Im(k), or more specifically, the face orientation of the subject estimated by the face orientation estimation unit 13 based on the frame Im(k). The luminance signal extraction unit 15 may also determine the face orientation of the subject by matching the identification numbers of the frame Im(k) of the captured image. Each frame Im(k) of the captured image is assigned an identification number. The imaging device 2 assigns an identification number to each frame Im(k) and outputs the captured image. Then, the luminance signal extraction unit 15 refers to the information for setting the measurement area to be used and sets the measurement area ri(k) corresponding to the measurement area number associated with the determined face orientation as the measurement area ri(k) to be used. The luminance signal extraction unit 15 extracts the pulse wave source signal for the pulse wave estimation period from each of the set measurement areas ri(k) used on the frame Im(k) of the captured image acquired by the image acquisition unit 11.

[0055] For example, suppose the subject's yaw direction face orientation, as indicated by face orientation information F(k) including the frame Im(k) of the captured image, is "-10 degrees". Also, suppose that in the information for setting the measurement area to be used, the subject's yaw direction face orientation of "-10 degrees" is associated with measurement area numbers (1)~(2), (7)~(8), (15)~(18), and (21)~(24). In this case, the luminance signal extraction unit 15 sets the measurement areas ri(k) numbered (1) to (2), (7) to (8), (15) to (18), and (21) to (24) from the measurement areas ri(k) set in frame Im(k) of the captured image acquired by the image acquisition unit 11 as the measurement areas ri(k) to be used. The luminance signal extraction unit 15 then extracts pulse wave source signals that show the luminance change during the pulse wave estimation period from each of the measurement areas ri(k) used, numbered (1) to (2), (7) to (8), (15) to (18), and (21) to (24). When the luminance signal extraction unit 15 extracts a pulse wave source signal, it generates pulse wave source signal information W(t) that indicates the extracted pulse wave source signal.

[0056] The pulse wave source signal information W(t) includes information indicating the pulse wave source signal wi(t) extracted in the measurement area ri(k). The pulse wave source signal wi(t) is time-series data for Tp minutes and is extracted, for example, based on past Tp minutes of frames Im(k-Tp+1), Im(k-Tp+2), ..., Im(k) and measurement area information R(k-Tp+1), R(k-Tp+2), ..., R(k). In extracting the pulse wave source signal wi(t), the luminance signal extraction unit 15 calculates the difference Gi(j) (j=k-Tp+1,k-Tp+2,···,k) of the luminance features of each measurement area ri(k) used between each frame Im(k) of the captured image and the previous frame Im(k-1). The luminance features are values ​​calculated for each measurement area ri(j) based on the luminance values ​​on frame Im(j) of the captured image. The luminance features are the average or variance of the luminance values ​​of the pixels included in the measurement area ri(j). In Embodiment 1, as an example, the luminance features are the average of the luminance values ​​of the pixels included in the measurement area ri(j). The luminance signal extraction unit 15 arranges the Gi(j) calculated for each frame Im(k) of the captured image acquired during the pulse wave estimation period in a time series to obtain the pulse wave source signal wi(t). That is, the luminance signal extraction unit 15 defines the pulse wave source signal wi(t) as [Gi(k-Tp+1),Gi(k-Tp+2),···,Gi(k)]. Furthermore, if the luminance signal extraction unit 15 does not have a corresponding measurement area ri(k) set in the previous frame Im(k-1), it does not calculate the difference in the measurement area ri(k). In other words, the luminance signal extraction unit 15 sets the difference in the measurement area ri(k) to "0".

[0057] The luminance signal extraction unit 15 generates pulse wave source signal information W(t) indicating the pulse wave source signal wi(t) in each measurement area ri(k) used. The pulse wave source signal information W(t) includes the pulse wave source signal wi(t) in each measurement area ri(k) used, information indicating which measurement area ri(k) the pulse wave source signal wi(t) was extracted from, each frame Im(k) of the captured image containing the measurement area ri(k) from which the pulse wave source signal wi(t) was extracted, and the identification number of each frame Im(k) of the captured image in that time series. The luminance signal extraction unit 15 outputs the generated pulse wave source signal information W(t) to the luminance signal selection unit 16.

[0058] Based on the pulse wave source signal information W(t) output from the luminance signal extraction unit 15, the luminance signal selection unit 16 selects a time-series pulse wave source signal (hereinafter referred to as the "estimated pulse wave source signal") wi(t) to be used for estimating the subject's pulse wave from the time-series pulse wave source signals wi(t) extracted by the luminance signal extraction unit 15 during the pulse wave estimation period, taking into consideration the subject's face orientation estimated by the face orientation estimation unit 13.

[0059] The method for selecting the estimated pulse wave source signal wi(t) by the luminance signal selection unit 16 in Embodiment 1 will be described below. The luminance signal selection unit 16 calculates the distribution ratio of the subject's face orientation corresponding to each frame Im(k) of the captured image estimated by the face orientation estimation unit 13 during the pulse wave estimation period. The luminance signal selection unit 16 then selects the time-series pulse wave source signal wi(t) extracted from the measurement area ri(k) commonly set for each frame Im(k) of the captured image from which the subject's face orientation was estimated, whose frequency of appearance is above a predetermined threshold (hereinafter referred to as the "face orientation determination threshold"), as the time-series pulse wave source signal wi(t) for estimation. For example, the face orientation estimation unit 13 stores face orientation information F(k) in a time series in a memory unit or the like. The captured image is also associated with information about the date and time the image was captured. Based on the face orientation information F(k) stored in the memory unit or the like, the luminance signal selection unit 16 can calculate the distribution ratio of the subject's face orientation corresponding to each frame Im(k) of the captured image estimated by the face orientation estimation unit 13 during the pulse wave estimation period.

[0060] Here, Figure 8 is an example of a histogram showing the distribution ratio of the subject's face orientation corresponding to each frame Im(k) of the captured image, as estimated by the face orientation estimation unit 13 during the pulse wave estimation period, calculated by the luminance signal selection unit 16 in Embodiment 1. Here, it is assumed that the subject changes their face orientation only in the yaw direction. That is, the distribution ratio of the subject's face orientation shown in Figure 8 is the distribution ratio of the subject's face orientation in the yaw direction.

[0061] In the histogram shown in Figure 8, the frequency of the subjects' yaw-direction face orientations between "-5 degrees and +5 degrees" is above the threshold for determining face orientation. Therefore, the luminance signal selection unit 16 selects the time-series pulse wave source signal wi(t) extracted from the measurement area ri(k) used, which was commonly determined in each frame Im(k) of the captured image from which the subject's face orientation in the yaw direction was estimated to be "-5 degrees to +5 degrees", as the time-series pulse wave source signal wi(t) for estimation. The measurement region ri(k) used, which is commonly determined in each frame Im(k) of the captured image from which the subject's yaw direction face orientation was estimated to be "-5 degrees to +5 degrees", refers to the measurement region ri(k) determined as the measurement region ri(k) used by the luminance signal extraction unit 15 in each frame Im(k) of the captured image from which the subject's yaw direction face orientation was estimated to be "-5 degrees to +5 degrees".

[0062] The luminance signal selection unit 16 can, for example, identify the measurement area ri(k) that was commonly determined in each frame Im(k) of the captured image from which the subject's face orientation in the yaw direction was estimated to be "-5 degrees to +5 degrees", based on the above-mentioned information for setting the measurement area to be used. The luminance signal selection unit 16 selects the time-series pulse wave source signal wi(t) extracted from the specified measurement area ri(k) as the time-series pulse wave source signal wi(t) for estimation. The luminance signal selection unit 16 then outputs pulse wave source signal information (hereinafter referred to as "selected pulse wave source signal information") W(t), which includes the selected estimated pulse wave source signal wi(t), to the pulse wave estimation unit 17. In this example, as explained using Figure 8, the distribution ratio of the subjects' face orientations is determined in 5-degree increments, but this is just one example. For example, the distribution ratio of the subjects' face orientations may be determined in 1-degree increments. Basically, it is best to ensure that the resolution of the face orientation defined in the reference measurement area information matches the resolution of the distribution ratio of the subjects' face orientations.

[0063] The pulse wave estimation unit 17 estimates the subject's pulse wave based on the selected pulse wave source signal information W(t) output from the luminance signal selection unit 16. The pulse wave estimation unit 17 can estimate the subject's pulse wave using a known method that estimates the subject's pulse wave based on the pulse wave source signal. Here, we will describe an example of how the pulse wave estimation unit 17 estimates the subject's pulse wave. For example, the pulse wave estimation unit 17 first generates a signal (hereinafter referred to as "separated signal") that shows multiple signal components (hereinafter referred to as "principal components") based on the estimated pulse wave source signal wi(t). Specifically, the pulse wave estimation unit 17 analyzes the multiple principal components using general signal separation techniques such as PCA or ICA, and generates a separated signal that shows the analyzed multiple principal components. By analyzing the signal components using general signal separation techniques such as PCA or ICA, the pulse wave estimation unit 17 separates components that are likely to be pulse wave components from components that are likely to be noise components from the multiple estimated pulse wave source signals wi(t).

[0064] The pulse wave estimation unit 17 reconstructs the estimated pulse wave source signal wi(t) for each measurement area ri(k) used, based on the generated multiple separated signals, specifically the separated signal information Sep(t) relating to the multiple separated signals representing the multiple principal components of the generated signals. Furthermore, each separated signal included in the separated signal information Sep(t) contains the estimated pulse wave source signal wi(t) for each measurement area ri(k) used. The pulse wave estimation unit 17 can reconstruct the estimated pulse wave source signal wi(t) for each measurement area ri(k) used from the multiple separated signals included in the separated signal information Sep(t).

[0065] The pulse wave estimation unit 17 reconstructs the estimated pulse wave source signal wi(t) for each measurement area ri(k) used, and then generates reconstructed estimated pulse wave source signal information RW(t) that shows the reconstructed estimated pulse wave source signal wi(t) for each measurement area ri(k) used. The reconstructed estimated pulse wave source signal information RW(t) includes the estimated pulse wave source signal wi(t) for each reconstructed measurement area ri(k).

[0066] The pulse wave estimation unit 17 estimates the subject's pulse wave based on the generated post-reconstruction estimated pulse wave source signal information RW(t). The pulse wave estimation unit 17, for example, determines the signal-to-noise ratio (S / N ratio) of the reconstructed pulse wave source signals for estimation for each measurement area ri(k). The pulse wave estimation unit 17 then weights the reconstructed pulse wave source signals for estimation for each measurement area ri(k) based on the obtained S / N ratio, and calculates a composite estimated pulse wave signal information D(t) by summing the reconstructed pulse wave source signals corresponding to each measurement area ri(k). In other words, the pulse wave estimation unit 17 calculates one composite estimated pulse wave signal information D(t) for all measurement areas ri(k). Because weighting is performed based on the S / N ratio, the composite estimated pulse wave signal information D(t) is assumed to be a signal that resembles a pulse wave component, with noise components removed. The pulse wave estimation unit 17 then performs a Fourier transform on the synthesized estimation pulse wave signal information D(t) and calculates the pulse rate as the peak frequency in the frequency power spectrum within a predetermined frequency range. The predetermined frequency range is set considering the range of human heart rates. The pulse wave estimation unit 17 outputs pulse wave estimation result P(t), which is pulse wave information indicating the estimated pulse wave, to the output unit 18. The pulse wave information may be, for example, time-series data of the subject's pulse wave estimated by the pulse wave estimation unit 17, the subject's pulse rate, or the subject's pulse interval.

[0067] The output unit 18 outputs the pulse wave estimation result P(t) output from the pulse wave estimation unit 17 to, for example, an alertness level estimation device or an abnormality detection device. The functions of the output unit 18 may also be provided in the pulse wave estimation unit 17. If the functions of the output unit 18 are provided in the pulse wave estimation unit 17, the pulse wave estimation device 1 does not require the output unit 18 as a necessary component.

[0068] The operation of the pulse wave estimation device 1 according to Embodiment 1 will be described below. Figure 9 is a flowchart illustrating the operation of the pulse wave estimation device 1 according to Embodiment 1. The pulse wave estimation device 1, for example, when the vehicle's power is turned on, repeats the process shown in the flowchart of Figure 9 until the vehicle's power is turned off.

[0069] The image acquisition unit 11 acquires an image of the subject (step ST1). The image acquisition unit 11 outputs the acquired image to the skin region detection unit 12.

[0070] The skin region detection unit 12 detects the subject's skin region from the frame Im(k) included in the captured image acquired by the image acquisition unit 11 in step ST1 (step ST2). The skin region detection unit 12 generates skin region information S(k) indicating the detected skin region. The skin region detection unit 12 outputs the generated skin region information S(k) to the measurement region setting unit 14.

[0071] The measurement area setting unit 14 sets multiple measurement areas ri(k) that can be used to extract pulse wave source signals indicating brightness changes in the image area on frame Im(k) corresponding to the skin area indicated by the skin area information S(k) on frame Im(k) (step ST3), based on the frame Im(k) of the image acquired by the image acquisition unit 11 in step ST1 and the skin area information S(k) output by the skin area detection unit 12 in step ST2 (step ST3). In detail, the measurement area setting unit 14 determines multiple measurement areas ri(k) to be set based on multiple measurement areas defined in the reference measurement area information, using layout information and information indicating the target face orientation range. Based on the frame Im(k) of the captured image acquired by the image acquisition unit 11 and the skin area information S(k) output by the skin area detection unit 12, the determined multiple measurement areas ri(k) are set in the image areas on frame Im(k) that correspond to the skin areas indicated by the skin area information S(k). When the measurement area setting unit 14 sets multiple measurement areas ri(k), it generates measurement area information R(k) that indicates the set multiple measurement areas ri(k). The measurement area setting unit 14 outputs the generated measurement area information R(k) to the luminance signal extraction unit 15.

[0072] The face orientation estimation unit 13 estimates the subject's face orientation (subject-referenced face orientation) on a frame-by-frame basis based on the frame Im(k) of the captured image acquired by the image acquisition unit 11 in step ST1 (step ST4). The face orientation estimation unit 13 outputs face orientation information F(k) related to the estimated face orientation of the subject (face orientation relative to the subject) to the brightness signal extraction unit 15.

[0073] The luminance signal extraction unit 15 extracts a time-series luminance signal, or in other words, a time-series pulse wave source signal wi(t), from each of the measurement areas ri(k) used, from among the multiple measurement areas ri(k) indicated by the measurement area information R(k) on the frame Im(k) indicated by the measurement area information R(k) (step ST5). The luminance signal extraction unit 15 generates pulse wave source signal information W(t) that shows the extracted time-series pulse wave source signal wi(t). The luminance signal extraction unit 15 outputs the generated pulse wave source signal information W(t) to the luminance signal selection unit 16.

[0074] The luminance signal selection unit 16 selects a time-series luminance signal, or in other words, a time-series estimation pulse wave source signal wi(t), based on the pulse wave source signal information W(t) output from the luminance signal extraction unit 15 in step ST5, and taking into account the subject's face orientation estimated by the face orientation estimation unit 13 in step ST4 (step ST6). The luminance signal selection unit 16 outputs the selected pulse wave source signal information W(t) to the pulse wave estimation unit 17.

[0075] The pulse wave estimation unit 17 estimates the subject's pulse wave based on the selected pulse wave source signal information W(t) output from the luminance signal selection unit 16 in step ST6 (step ST7). The pulse wave estimation unit 17 outputs pulse wave estimation result P(t), which is pulse wave information indicating the estimated pulse wave, to the output unit 18. The output unit 18 outputs the pulse wave estimation result P(t) output from the pulse wave estimation unit 17 to, for example, an alertness level estimation device or an abnormality detection device.

[0076] Note that in the flowchart shown in Figure 9, step ST4 is assumed to occur after step ST3, but this is only one example. For example, the order of steps ST4 and ST3 may be reversed, or steps ST2 and ST3 may be performed in parallel with step ST4. Step ST4 only needs to be performed after step ST1 and before step ST5.

[0077] Figure 10 is a flowchart illustrating the details of the process in step ST5 of Figure 9. The luminance signal extraction unit 15 compares the frame Im(k) of the captured image acquired by the image acquisition unit 11 in step ST1 of Figure 9 with the frame Im(k) of the captured image included in the face orientation information F(k) output in step ST4 of Figure 9, and determines the face orientation of the subject corresponding to the frame Im(k). Then, the luminance signal extraction unit 15 refers to the information for setting the measurement area to be used and sets the measurement area ri(k) corresponding to the measurement area number associated with the determined face orientation as the measurement area to be used (step ST501). The luminance signal extraction unit 15 extracts the time-series pulse wave source signal wi(t) during the pulse wave estimation period from each of the measurement areas ri(k) used on the frame Im(k) in step ST1 of Figure 9, based on the frame Im(k) of the image acquisition unit 11 acquired by the image acquisition unit 11 and the measurement area information R(k) output from the measurement area setting unit 14 in step ST3 of Figure 9 (step ST502). The luminance signal extraction unit 15 extracts a time-series pulse wave source signal wi(t) and generates pulse wave source signal information W(t) that represents the extracted time-series pulse wave source signal wi(t). The luminance signal extraction unit 15 outputs the generated pulse wave source signal information W(t) to the luminance signal selection unit 16.

[0078] Figure 11 is a flowchart illustrating the details of the process in step ST6 of Figure 9. The luminance signal selection unit 16 calculates the distribution ratio of the subject's face orientation corresponding to each frame Im(k) of the captured image estimated by the face orientation estimation unit 13 during the pulse wave estimation period (step ST601). The luminance signal selection unit 16 selects a time-series pulse wave source signal wi(t) extracted from the measurement area ri(k) commonly set in each frame Im(k) of the captured image from which the facial orientation of a subject whose frequency of appearance is equal to or greater than the threshold for facial orientation determination is estimated, as the time-series pulse wave source signal wi(t) for estimation (step ST602). The luminance signal selection unit 16 outputs the selected pulse wave source signal information W(t) to the pulse wave estimation unit 17.

[0079] In this way, the pulse wave estimation device 1 detects the subject's skin region from the captured image and sets a measurement region ri(k) in the region corresponding to the skin region on the captured image, which can be used to extract a pulse wave source signal wi(t) that shows a change in brightness and contains the subject's pulse wave component. Specifically, the pulse wave estimation device 1 determines the measurement region ri(k) to be set in the region corresponding to the skin region on the captured image based on information indicating the target face orientation range, layout information, and reference measurement region information, and sets the determined measurement region ri(k). The pulse wave estimation device 1 sets a measurement area ri(k) to be used from among the measurement areas ri(k), extracts a time-series pulse wave source signal wi(t) based on the brightness change in the measurement area ri(k), and, taking into account the subject's face orientation estimated frame by frame based on the captured image, selects a time-series pulse wave source signal wi(t) to be used for estimating the subject's pulse wave from among the time-series pulse wave source signals wi(t) extracted during the pulse wave estimation period. The pulse wave estimation device 1 estimates the subject's pulse wave based on the selected time-series pulse wave source signal wi(t). In detail, the pulse wave estimation device 1 extracts the time-series pulse wave source signal wi(t) and selects the time-series pulse wave source signal wi(t) for estimation. From the set measurement areas ri(k), the device extracts the time-series pulse wave source signal wi(t) using the measurement area ri(k) that corresponds to the subject's face orientation estimated based on the captured image in which the measurement area ri(k) is set. The pulse wave estimation device 1 then calculates the distribution ratio of the subject's face orientation for each frame estimated during the pulse wave estimation period, and selects the time-series pulse wave source signal wi(t) extracted from the commonly set measurement area ri(k) in the captured image from which the subject's face orientation was estimated, whose frequency of appearance is above the threshold for face orientation determination, as the time-series pulse wave source signal wi(t) for estimation.

[0080] As a result, the pulse wave estimation device 1 can extract a luminance signal containing sufficient pulse wave components to estimate the subject's pulse wave, in other words, a pulse wave source signal wi(t), from the skin area of ​​the captured image. Consequently, the pulse wave estimation device 1 can prevent a decrease in the accuracy of estimating the subject's pulse wave due to the skin area of ​​the person whose pulse wave is to be estimated, i.e., the subject, not being captured in the captured image, or due to so-called shadowing occurring in that skin area.

[0081] Figures 12A and 12B show an example of the hardware configuration of the pulse wave estimation device 1 according to Embodiment 1. In Embodiment 1, the functions of the image acquisition unit 11, skin area detection unit 12, face orientation estimation unit 13, measurement area setting unit 14, brightness signal extraction unit 15, brightness signal selection unit 16, pulse wave estimation unit 17, and output unit 18 are realized by the processing circuit 101. That is, the pulse wave estimation device 1 includes a processing circuit 101 for extracting a brightness signal containing sufficient pulse wave components to estimate a person's pulse wave from the skin area on the captured image, and for controlling the estimation of the subject's pulse wave from the extracted brightness signal. The processing circuit 101 may be dedicated hardware as shown in Figure 12A, or it may be a processor 104 that executes a program stored in memory as shown in Figure 12B.

[0082] If the processing circuit 101 is dedicated hardware, it may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof.

[0083] When the processing circuit is a processor 104, the functions of the image acquisition unit 11, skin region detection unit 12, face orientation estimation unit 13, measurement region setting unit 14, brightness signal extraction unit 15, brightness signal selection unit 16, pulse wave estimation unit 17, and output unit 18 are realized by software, firmware, or a combination of software and firmware. The software or firmware is written as a program and stored in memory 105. The processor 104 reads and executes the program stored in memory 105 to perform the functions of the image acquisition unit 11, skin region detection unit 12, face orientation estimation unit 13, measurement region setting unit 14, brightness signal extraction unit 15, brightness signal selection unit 16, pulse wave estimation unit 17, and output unit 18. In other words, the pulse wave estimation device 1 includes memory 105 for storing a program that, when executed by the processor 104, will result in the execution of steps ST1 to ST7 in Figure 9 described above. Furthermore, the program stored in memory 105 can be said to cause the computer to execute the processing procedures or methods of the image acquisition unit 11, the skin area detection unit 12, the face orientation estimation unit 13, the measurement area setting unit 14, the brightness signal extraction unit 15, the brightness signal selection unit 16, the pulse wave estimation unit 17, and the output unit 18. Here, memory 105 refers to non-volatile or volatile semiconductor memory such as RAM, ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), magnetic disks, flexible disks, optical disks, compact disks, minidiscs, DVDs (Digital Versatile Discs), etc.

[0084] Furthermore, the functions of the image acquisition unit 11, skin area detection unit 12, face orientation estimation unit 13, measurement area setting unit 14, brightness signal extraction unit 15, brightness signal selection unit 16, pulse wave estimation unit 17, and output unit 18 may be partially implemented by dedicated hardware and partially by software or firmware. For example, the image acquisition unit 11 and output unit 18 can be implemented by a processing circuit 101 as dedicated hardware, while the functions of the skin area detection unit 12, face orientation estimation unit 13, measurement area setting unit 14, brightness signal extraction unit 15, brightness signal selection unit 16, and pulse wave estimation unit 17 can be implemented by the processor 104 reading and executing a program stored in memory 105. The storage unit, which is not shown in the diagram, is composed of, for example, memory 105. Furthermore, the pulse wave estimation device 1 includes devices such as an imaging device 2, and an input interface device 102 and an output interface device 103 for wired or wireless communication.

[0085] In the above embodiment 1, the subject was the vehicle driver, but this is merely an example. The subject may be a passenger other than the vehicle driver.

[0086] Furthermore, in the above embodiment 1, the pulse wave estimation device 1 was an in-vehicle device, and the image acquisition unit 11, skin area detection unit 12, face orientation estimation unit 13, measurement area setting unit 14, brightness signal extraction unit 15, brightness signal selection unit 16, pulse wave estimation unit 17, and output unit 18 were provided in the in-vehicle device. The system is not limited to this, however, some of the components of the image acquisition unit 11, skin area detection unit 12, face orientation estimation unit 13, measurement area setting unit 14, brightness signal extraction unit 15, brightness signal selection unit 16, pulse wave estimation unit 17, and output unit 18 may be mounted on an in-vehicle device, while the others are provided on a server connected to the in-vehicle device via a network, thereby configuring the system with the in-vehicle device and the server. Alternatively, the image acquisition unit 11, skin area detection unit 12, face orientation estimation unit 13, measurement area setting unit 14, brightness signal extraction unit 15, brightness signal selection unit 16, pulse wave estimation unit 17, and output unit 18 may all be provided on the server.

[0087] Furthermore, the pulse wave estimation device 1 according to Embodiment 1 described above is not limited to in-vehicle devices mounted on vehicles, but can also be applied to other mobile devices or home appliances, for example. Also, the subjects are not limited to vehicle occupants, but can be various people. For example, the pulse wave estimation device 1 may be installed on a television set in the living room of a residence. In this case, the subject is a user, such as a resident of the residence. The pulse wave estimation device 1 estimates the user's pulse wave based on the image captured by the imaging device 2 installed on the television set.

[0088] Furthermore, in the first embodiment described above, the pulse wave estimation device 1, the measurement area setting unit 14 determines a measurement area ri(k) to be set in the area corresponding to the skin area on the captured image, based on information indicating the target face orientation range, layout information, and reference measurement area information. This is just one example. For instance, if the target face orientation range is defined as the range of all possible face orientations that the subject can take within the imaging range of the imaging device 2, and if it is assumed that the imaging device 2 always images the subject at a reference position and reference orientation, then the measurement area setting unit 14 does not need to consider the target face orientation range and layout information. In this case, for example, an administrator or the like may determine in advance, based on the image captured by the imaging device 2 of the assumed subject, a measurement region ri(k) that can be used to extract the pulse wave source signal wi(t) containing the subject's pulse wave component, generate information indicating the determined measurement region ri(k) as reference measurement region information, and store it in a memory unit or the like. The measurement area setting unit 14 does not take into account the information indicating the target face orientation range or the layout information, and simply sets the measurement area ri(k) defined in the reference measurement area information to the area corresponding to the skin area on the captured image.

[0089] As described above, the pulse wave estimation device 1 according to Embodiment 1 includes an image acquisition unit 11 that acquires captured images of a person (subject) on a frame-by-frame basis, a skin region detection unit 12 that detects the skin region of a person from the captured image, and on the captured image skinA measurement area setting unit 14 sets a measurement area ri(k) that can be used to extract a pulse wave source signal wi(t) that shows brightness changes and contains a human pulse wave component in a region corresponding to the region; a face orientation estimation unit 13 estimates the face orientation of a person on a frame-by-frame basis based on the captured image; and sets a usable measurement area ri(k) to be used to extract the pulse wave source signal wi(t) from the measurement area ri(k) set by the measurement area setting unit 14, and in the set usable measurement area ri(k) The pulse wave estimation device 1 is configured to include a luminance signal extraction unit 15 that extracts a time-series pulse wave source signal wi(t) based on changes in luminance, a luminance signal selection unit 16 that selects a time-series pulse wave source signal wi(t) to be used for estimating a person's pulse wave from the time-series pulse wave source signals wi(t) extracted by the luminance signal extraction unit 15 during the pulse wave estimation period, taking into account the person's face orientation estimated by the face orientation estimation unit 13, and a pulse wave estimation unit 17 that estimates a person's pulse wave based on the time-series pulse wave source signal wi(t) selected by the luminance signal selection unit 16. Therefore, the pulse wave estimation device 1 can prevent a decrease in the accuracy of estimating a subject's pulse wave due to the skin area of ​​the person whose pulse wave is to be estimated, i.e., the subject, not being captured in the captured image, or so-called shadows occurring in the skin area.

[0090] In detail, in the pulse wave estimation device 1, the luminance signal extraction unit 15 extracts the pulse wave source signal wi(t) as the measurement area ri(k) used, which corresponds to the face orientation of the person (subject) estimated by the face orientation estimation unit 13 based on the captured image in which the measurement area ri(k) is set, from among the measurement areas ri(k) set by the measurement area setting unit 14. The luminance signal selection unit 16 calculates the distribution ratio of the person's face orientation for each frame estimated by the face orientation estimation unit 13 during the pulse wave estimation target period, and selects the time-series pulse wave source signal wi(t) extracted from the commonly set measurement area ri(k) in the captured image that was the source of the estimation of the person's face orientation whose appearance frequency is above the face orientation determination threshold, as the time-series pulse wave source signal wi(t) for estimation. Therefore, the pulse wave estimation device 1 can prevent a decrease in the accuracy of estimating the pulse wave of a subject due to the skin area of ​​the person whose pulse wave is to be estimated, i.e., the subject, not being captured in the captured image, or due to so-called shadowing occurring in that skin area.

[0091] Embodiment 2. In Embodiment 1, the pulse wave estimation device selected the time-series pulse wave source signals for estimation based on the distribution ratio of the subject's face orientation for each frame. Embodiment 2 describes an embodiment in which a pulse wave source signal for time-series estimation is selected using a method different from that of Embodiment 1. In the second embodiment described below, as in the first embodiment, the pulse wave estimation device is mounted on the vehicle, and the subject is assumed to be the vehicle's driver.

[0092] Figure 13 shows an example of the configuration of the pulse wave estimation device 1a according to Embodiment 2. Regarding the configuration of the pulse wave estimation device 1a according to Embodiment 2, the same reference numerals are used for components that are the same as those in the pulse wave estimation device 1 described with reference to Figure 1 in Embodiment 1, and redundant explanations are omitted. In the pulse wave estimation device 1a according to Embodiment 2, the specific operation of the luminance signal selection unit 16a differs from the specific operation of the luminance signal selection unit 16 in the pulse wave estimation device 1 according to Embodiment 1.

[0093] Based on the pulse wave source signal information W(t) output from the luminance signal extraction unit 15, the luminance signal selection unit 16a selects a time-series pulse wave source signal wi(t) to be used for estimating the subject's pulse wave from the time-series pulse wave source signals wi(t) extracted by the luminance signal extraction unit 15 during the pulse wave estimation period, taking into account the subject's face orientation estimated by the face orientation estimation unit 13.

[0094] The method for selecting the estimated pulse wave source signal wi(t) by the luminance signal selection unit 16a will be explained. In Embodiment 2, the luminance signal selection unit 16a calculates the distribution ratio of the measurement area ri(k) used by the luminance signal extraction unit 15 to extract the pulse wave source signal wi(t) during the pulse wave estimation period. The luminance signal selection unit 16a then selects the time-series pulse wave source signal wi(t) extracted from the measurement area ri(k) whose occurrence frequency is equal to or greater than a preset threshold (hereinafter referred to as the "area determination threshold") as the time-series pulse wave source signal wi(t) for estimation. For example, the luminance signal selection unit 16a stores the pulse wave source signal information W(t) in a memory unit that does not illustrate the information in a time series. Based on the pulse wave source signal information W(t) stored in the memory unit, the luminance signal selection unit 16a can calculate the distribution ratio of the measurement area ri(k) used by the luminance signal extraction unit 15 to extract the pulse wave source signal wi(t) during the pulse wave estimation period.

[0095] Here, Figure 14 is a diagram showing an example of a histogram in Embodiment 2, which shows the distribution ratio of the measurement area ri(k) used as the source of the pulse wave source signal wi(t) extracted by the luminance signal extraction unit 15 during the pulse wave estimation period, calculated by the luminance signal selection unit 16a. In Figure 14, as an example, the luminance signal extraction unit 15 sets all 42 measurement regions ri(k) from measurement region numbers (1) to (42) as the measurement region ri(k) used during the pulse wave estimation period and extracts the pulse wave source signal wi(t). In the histogram shown in Figure 14, the frequency of the measurement region ri(k) used for measurement region numbers (1) to (26) is above the threshold for region determination. Therefore, the luminance signal selection unit 16a selects the time-series pulse wave source signal wi(t) extracted from the measurement area ri(k) used with measurement area numbers (1) to (26) as the time-series pulse wave source signal wi(t) for estimation. The luminance signal selection unit 16a outputs pulse wave source signal information W(t), which includes the selected time-series pulse wave source signal wi(t) for estimation, to the pulse wave estimation unit 17 as selected pulse wave source signal information W(t).

[0096] The operation of the pulse wave estimation device 1a according to Embodiment 2 will be described. Figure 15 is a flowchart illustrating the operation of the pulse wave estimation device 1a according to Embodiment 2. The pulse wave estimation device 1a, for example, when the vehicle's power is turned on, repeats the process shown in the flowchart of Figure 15 until the vehicle's power is turned off.

[0097] The specific operations of steps ST1 to ST5 and step ST7 by the pulse wave estimation device 1a are the same as those of steps ST1 to ST5 and step ST7, which were explained using the flowchart in Figure 9 in Embodiment 1. Therefore, the same step numbers are used and redundant explanations are omitted.

[0098] The luminance signal selection unit 16 selects a time-series pulse wave source signal wi(t) for estimation based on the pulse wave source signal information W(t) output from the luminance signal extraction unit 15 in step ST5, and taking into account the face orientation of the subject estimated by the face orientation estimation unit 13 in step ST4 (step ST6a). The luminance signal selection unit 16 outputs the selected pulse wave source signal information W(t) to the pulse wave estimation unit 17.

[0099] Note that in the flowchart shown in Figure 15, step ST4 is assumed to occur after step ST3, but this is only one example. For example, the order of steps ST4 and ST3 may be reversed, or steps ST2 and ST3 may be performed in parallel with step ST4. Step ST4 only needs to be performed after step ST1 and before step ST5.

[0100] Figure 16 is a flowchart illustrating the details of step ST6a in Figure 15. The luminance signal selection unit 16a calculates the distribution ratio of the measurement area ri(k) used by the luminance signal extraction unit 15 to extract the pulse wave source signal wi(t) during the pulse wave estimation period (step ST611). The luminance signal selection unit 16a selects the time-series pulse wave source signal wi(t) extracted from the measurement area ri(k) whose occurrence frequency is equal to or greater than the threshold for area determination, as the time-series pulse wave source signal wi(t) for estimation (step ST612). The luminance signal selection unit 16a outputs the selected pulse wave source signal information W(t) to the pulse wave estimation unit 17.

[0101] Thus, the pulse wave estimation device 1a detects the subject's skin region from the captured image and sets a measurement region ri(k) in the region corresponding to the skin region on the captured image, which can be used to extract a pulse wave source signal wi(t) that shows a change in brightness and contains the subject's pulse wave component. Specifically, the pulse wave estimation device 1a determines the measurement region ri(k) to be set in the region corresponding to the skin region on the captured image based on information indicating the target face orientation range, layout information, and reference measurement region information, and sets the determined measurement region ri(k). The pulse wave estimation device 1a sets a measurement area ri(k) to be used from among the measurement areas ri(k), extracts a time-series pulse wave source signal based on the brightness change in the measurement area ri(k), and, taking into account the subject's face orientation estimated frame by frame based on the captured image, selects a time-series pulse wave source signal wi(t) to be used for estimating the subject's pulse wave from among the time-series pulse wave source signals wi(t) extracted during the pulse wave estimation period. The pulse wave estimation device 1a estimates the subject's pulse wave based on the selected time-series pulse wave source signal wi(t). In detail, the pulse wave estimation device 1a calculates the distribution ratio of the measurement area ri(k) used to extract the time-series pulse wave source signal wi(t) and selects the time-series pulse wave source signal wi(t) for estimation. It then selects the time-series pulse wave source signal wi(t) extracted from the measurement area ri(k) whose occurrence frequency is equal to or greater than the threshold for area determination as the time-series pulse wave source signal wi(t) for estimation.

[0102] As a result, the pulse wave estimation device 1a can extract a luminance signal containing sufficient pulse wave components to estimate the subject's pulse wave, in other words, a pulse wave source signal wi(t), from the skin region of the captured image. Consequently, the pulse wave estimation device 1a can prevent a decrease in the accuracy of estimating the subject's pulse wave due to the skin region of the person whose pulse wave is to be estimated, i.e., the subject, not being captured in the captured image, or due to so-called shadowing occurring in that skin region.

[0103] The hardware configuration of the pulse wave estimation device 1a according to Embodiment 2 is the same as the hardware configuration of the pulse wave estimation device 1 described using Figures 12A and 12B in Embodiment 1, so it is omitted from the illustration. In Embodiment 2, the functions of the image acquisition unit 11, skin region detection unit 12, face orientation estimation unit 13, measurement region setting unit 14, brightness signal extraction unit 15, brightness signal selection unit 16a, pulse wave estimation unit 17, and output unit 18 are realized by the processing circuit 101. That is, the pulse wave estimation device 1a includes a processing circuit 101 for extracting a brightness signal containing sufficient pulse wave components to estimate a person's pulse wave from the skin region on the captured image, and for controlling the estimation of the subject's pulse wave from the extracted brightness signal.

[0104] The processing circuit 101 reads and executes a program stored in the memory 105, thereby executing the functions of the image acquisition unit 11, the skin region detection unit 12, the face orientation estimation unit 13, the measurement region setting unit 14, the brightness signal extraction unit 15, the brightness signal selection unit 16a, the pulse wave estimation unit 17, and the output unit 18. In other words, the pulse wave estimation device 1a includes a memory 105 for storing a program that, when executed by the processing circuit 101, will result in the execution of steps ST1 to ST7 in Figure 15 described above. It can also be said that the program stored in the memory 105 causes the computer to execute the processing procedures or methods of the image acquisition unit 11, the skin region detection unit 12, the face orientation estimation unit 13, the measurement region setting unit 14, the brightness signal extraction unit 15, the brightness signal selection unit 16a, the pulse wave estimation unit 17, and the output unit 18. The storage unit, which is not shown in the diagram, is composed of, for example, memory 105. Furthermore, the pulse wave estimation device 1a includes devices such as an imaging device 2, and an input interface device 102 and an output interface device 103 for wired or wireless communication.

[0105] In the above embodiment 2, the subject was the vehicle driver, but this is merely an example. The subject may be a passenger other than the vehicle driver.

[0106] Furthermore, in the above embodiment 2, the pulse wave estimation device 1a was an in-vehicle device, and the image acquisition unit 11, skin area detection unit 12, face orientation estimation unit 13, measurement area setting unit 14, brightness signal extraction unit 15, brightness signal selection unit 16a, pulse wave estimation unit 17, and output unit 18 were provided in the in-vehicle device. The system is not limited to this, however, some of the following components—image acquisition unit 11, skin area detection unit 12, face orientation estimation unit 13, measurement area setting unit 14, brightness signal extraction unit 15, brightness signal selection unit 16a, pulse wave estimation unit 17, and output unit 18—may be mounted on an in-vehicle device, while the others are provided on a server connected to the in-vehicle device via a network, with the in-vehicle device and the server forming the system. Alternatively, the image acquisition unit 11, skin area detection unit 12, face orientation estimation unit 13, measurement area setting unit 14, brightness signal extraction unit 15, brightness signal selection unit 16a, pulse wave estimation unit 17, and output unit 18 may all be provided on the server.

[0107] Furthermore, the pulse wave estimation device 1a according to the second embodiment described above is not limited to in-vehicle devices mounted on vehicles, but can also be applied to other mobile devices or home appliances, for example. Also, the subjects are not limited to vehicle occupants, but can be various people.

[0108] Furthermore, in the second embodiment described above, the pulse wave estimation device 1a determined the measurement area setting unit 14 to be set to the area corresponding to the skin area on the captured image based on information indicating the target face orientation range, layout information, and reference measurement area information, but this is merely one example. For example, if the target face orientation range is defined as the range of all face orientations that the subject can take within the range that the imaging device 2 can capture, and it is assumed that the imaging device 2 always captures the subject at a reference position and reference orientation, then the measurement area setting unit 14 does not need to consider the target face orientation range and layout information. In this case, for example, an administrator or the like may determine in advance, based on the image captured by the imaging device 2 of the assumed subject, a measurement region ri(k) that can be used to extract the pulse wave source signal wi(t) containing the subject's pulse wave component, generate information indicating the determined measurement region ri(k) as reference measurement region information, and store it in a memory unit or the like. The measurement area setting unit 14 does not take into account the information indicating the target face orientation range or the layout information, and simply sets the measurement area ri(k) defined in the reference measurement area information to the area corresponding to the skin area on the captured image.

[0109] As described above, according to Embodiment 2, the pulse wave estimation device 1a includes an image acquisition unit 11 that acquires captured images of a person (subject) on a frame-by-frame basis, a skin region detection unit 12 that detects the skin region of a person from the captured image, and on the captured image skinA measurement area setting unit 14 sets a measurement area ri(k) that can be used to extract a pulse wave source signal wi(t) that shows brightness changes and contains a human pulse wave component in a region corresponding to the region; a face orientation estimation unit 13 estimates the face orientation of a person on a frame-by-frame basis based on the captured image; and sets a usable measurement area ri(k) to be used to extract the pulse wave source signal wi(t) from the measurement area ri(k) set by the measurement area setting unit 14, and in the set usable measurement area ri(k) The pulse wave estimation device 1a is configured to include a luminance signal extraction unit 15 that extracts a time-series pulse wave source signal wi(t) based on changes in luminance, a luminance signal selection unit 16a that selects a time-series pulse wave source signal wi(t) to be used for estimating a person's pulse wave from the time-series pulse wave source signals wi(t) extracted by the luminance signal extraction unit 15 during the pulse wave estimation period, taking into account the person's face orientation estimated by the face orientation estimation unit 13, and a pulse wave estimation unit 17 that estimates a person's pulse wave based on the time-series pulse wave source signal wi(t) selected by the luminance signal selection unit 16a. As a result, the pulse wave estimation device 1a can prevent a decrease in the accuracy of estimating the pulse wave of a subject due to the skin area of ​​the person whose pulse wave is to be estimated, i.e., the subject, not being captured in the captured image, or so-called shadows occurring in the skin area.

[0110] In detail, in the pulse wave estimation device 1a, the luminance signal extraction unit 15 extracts the pulse wave source signal wi(t) as the measurement area ri(k) used by the face orientation estimation unit 13, based on the captured image in which the measurement area ri(k) is set, from among the measurement areas ri(k) set by the measurement area setting unit 14. The luminance signal selection unit 16a calculates the distribution ratio of the measurement areas ri(k) used by the luminance signal extraction unit 15 to extract the pulse wave source signal wi(t) during the pulse wave estimation target period, and selects the time-series pulse wave source signal wi(t) extracted from the measurement areas ri(k) whose occurrence frequency is equal to or greater than the threshold for area determination, as the time-series estimated pulse wave source signal wi(t). Therefore, the pulse wave estimation device 1a can prevent a decrease in the accuracy of estimating the pulse wave of a subject due to the skin area of ​​the person whose pulse wave is to be estimated, i.e., the subject, not being captured in the captured image, or due to so-called shadowing occurring in that skin area.

[0111] Embodiment 3. Embodiment 3 describes an embodiment in which a pulse wave source signal for time-series estimation is selected in a different manner from Embodiments 1 and 2. In the following Embodiment 3, as in Embodiments 1 and 2, the pulse wave estimation device is mounted on the vehicle, and the subject is assumed to be the vehicle's driver.

[0112] Figure 17 shows an example of the configuration of the pulse wave estimation device 1b according to Embodiment 3. Regarding the configuration of the pulse wave estimation device 1b according to Embodiment 3, the same reference numerals are used for components that are the same as those in the pulse wave estimation device 1 described with reference to Figure 1 in Embodiment 1, and redundant explanations are omitted. The pulse wave estimation device 1b according to Embodiment 3 differs from the pulse wave estimation device 1 according to Embodiment 1 in that it is equipped with a weight setting unit 19. Furthermore, in the pulse wave estimation device 1b according to Embodiment 3, the specific operations of the luminance signal extraction unit 15a and the luminance signal selection unit 16b differ from the specific operations of the luminance signal extraction unit 15 and the luminance signal selection unit 16 in the pulse wave estimation device 1 according to Embodiment 1, respectively.

[0113] The weight setting unit 19 sets weight coefficients for each measurement region ri(k) set by the measurement region setting unit 14 in the captured image in which the face orientation of the subject was estimated, based on the face orientation of the subject estimated by the face orientation estimation unit 13 on a frame-by-frame basis. In Embodiment 3, the face orientation estimation unit 13 outputs face orientation information F(k) to the weight setting unit 19. The measurement area setting unit 14 outputs the generated measurement area information R(k) to the brightness signal extraction unit 15 and the weight setting unit 19.

[0114] Here, Figure 18 is a diagram illustrating an example of a weight coefficient for each measurement region ri(k) that the weight setting unit 19 sets based on the subject's face orientation estimated by the face orientation estimation unit 13 on a frame-by-frame basis in Embodiment 3. The weight setting unit 19 sets weight coefficients based on, for example, the orientation of the subject's face, such that the weight increases as the measurement region ri(k) is closer to the imaging device 2. In the example shown in Figure 18, the weight setting unit 19 sets weight coefficients for each measurement area ri(k) such that "weight of measurement area ri(k) indicated by W1 > weight of measurement area ri(k) indicated by W2 > weight of measurement area ri(k) indicated by W3". Furthermore, the weight setting unit 19 can determine the positional relationship between the measurement area ri(k) and the imaging device 2 based on the layout information and face orientation information F(k). In other words, the weight setting unit 19 can select the measurement area ri(k) closest to the imaging device 2 and assign weight coefficients to it based on the layout information and face orientation information F(k).

[0115] The weight setting unit 19 outputs information regarding the weight coefficient set for each measurement area ri(k) (hereinafter referred to as "weight information") to the luminance signal selection unit 16b. The weight information is information that associates each frame Im(k) of the captured image with the identification number of each frame Im(k) of the captured image and the weight coefficient for each measurement region ri(k).

[0116] The luminance signal extraction unit 15a sets the measurement area ri(k) to be used from among the measurement areas ri(k) set by the measurement area setting unit 14. In Embodiment 3, the luminance signal extraction unit 15a sets the measurement area ri(k) set by the measurement area setting unit 14 as the measurement area ri(k) to be used. Then, the luminance signal extraction unit 15a extracts a pulse wave source signal wi(t) that indicates the luminance change during the pulse wave estimation period, or in other words, the period corresponding to the number of frames Tp, from each of the set used measurement areas ri(k) among the multiple measurement areas ri(k) indicated by the measurement area information R(k) on the frame Im(k). The method by which the luminance signal extraction unit 15a extracts pulse wave source signals wi(t) indicating luminance changes during the pulse wave estimation period from each of the set measurement areas ri(k) is the same as the method by which the luminance signal extraction unit 15 extracts pulse wave source signals wi(t) indicating luminance changes during the pulse wave estimation period from each of the set measurement areas ri(k) described in Embodiment 1, so a redundant explanation will be omitted. The luminance signal extraction unit 15a generates pulse wave source signal information W(t) indicating the pulse wave source signal wi(t) in each measurement area ri(k) used, and outputs the generated pulse wave source signal information W(t) to the luminance signal selection unit 16b.

[0117] Based on the pulse wave source signal information W(t) output from the luminance signal extraction unit 15a, the luminance signal selection unit 16b selects a time-series pulse wave source signal wi(t) to be used for estimating the subject's pulse wave from among the time-series pulse wave source signals wi(t) extracted by the luminance signal extraction unit 15a during the pulse wave estimation period, taking into account the subject's face orientation estimated by the face orientation estimation unit 13.

[0118] The method for selecting the estimated pulse wave source signal wi(t) by the luminance signal selection unit 16b will be explained. In Embodiment 3, the luminance signal selection unit 16b optimizes the pulse wave source signal wi(t) extracted by the luminance signal extraction unit 15a based on the weight coefficients set by the weight setting unit 19 for the measurement area ri(k) from which the pulse wave source signal wi(t) was extracted. The luminance signal selection unit 16b can identify the weight coefficients set by the weight setting unit 19 for the measurement area ri(k) from which the pulse wave source signal wi(t) was extracted by matching the pulse wave source signal information W(t) with the weight information, for example, using the identification number of the frame Im(k) of the captured image and the measurement area ri(k) as keys.

[0119] For example, the luminance signal selection unit 16b performs the above optimization by correcting the pulse wave source signal wi(t) extracted by the luminance signal extraction unit 15a to reduce its size if the weighting coefficient for the measurement area ri(k) from which the pulse wave source signal wi(t) was extracted is small. The amount by which the pulse wave source signal wi(t) is reduced for a given weighting coefficient is predetermined. Furthermore, for example, the luminance signal selection unit 16b performs the above optimization by selecting the pulse wave source signal wi(t) extracted by the luminance signal extraction unit 15a to be blank (set to zero) if the weighting coefficient for the measurement area ri(k) from which the pulse wave source signal wi(t) was extracted is small. The weighting coefficient at which the pulse wave source signal wi(t) is blank is predetermined.

[0120] The luminance signal selection unit 16b then selects the time-series pulse wave source signal wi(t) after optimization as the time-series pulse wave source signal wi(t) for estimation. The luminance signal selection unit 16b outputs the time-series selected pulse wave source signal information W(t) to the pulse wave estimation unit 17.

[0121] The operation of the pulse wave estimation device 1b according to Embodiment 3 will be described. Figure 19 is a flowchart illustrating the operation of the pulse wave estimation device 1b according to Embodiment 3. The pulse wave estimation device 1b, for example, when the vehicle's power is turned on, repeats the process shown in the flowchart of Figure 19 until the vehicle's power is turned off.

[0122] The specific operations of steps ST1 to ST4 and step ST7 by the pulse wave estimation device 1b are the same as those of steps ST1 to ST4 and step ST7, which were explained using the flowchart in Figure 9 in Embodiment 1. Therefore, the same step numbers are used and redundant explanations are omitted.

[0123] The weight setting unit 19 sets weight coefficients for each measurement region ri(k) set by the measurement region setting unit 14 in step ST3 in the captured image in which the face orientation of the subject was estimated, based on the face orientation of the subject estimated on a frame-by-frame basis by the face orientation estimation unit 13 in step ST4 (step ST41). The weight setting unit 19 outputs weight information to the luminance signal selection unit 16b.

[0124] The luminance signal extraction unit 15a extracts a luminance signal, or in other words, a pulse wave source signal wi(t), from each of the measurement areas ri(k) used, which are among the multiple measurement areas ri(k) indicated by the measurement area information R(k) on the frame Im(k) indicated by the measurement area information R(k) (step ST5a). The luminance signal extraction unit 15a generates pulse wave source signal information W(t) that indicates the extracted pulse wave source signal wi(t). The luminance signal extraction unit 15a outputs the generated pulse wave source signal information W(t) to the luminance signal selection unit 16.

[0125] The luminance signal selection unit 16b selects an estimated pulse wave source signal wi(t) based on the pulse wave source signal information W(t) output from the luminance signal extraction unit 15a in step ST5, taking into account the subject's face orientation estimated by the face orientation estimation unit 13 in step ST4 (step ST6b). The luminance signal selection unit 16b outputs the selected pulse wave source signal information W(t) to the pulse wave estimation unit 17.

[0126] Note that in the flowchart shown in Figure 19, steps ST4 to ST41 are assumed to be performed after step ST3, but this is only one example. For example, the order of steps ST4 and ST3 may be reversed, or steps ST2 to ST3 and step ST4 may be performed in parallel. Step ST4 only needs to be performed after step ST1 and before step ST41 is performed.

[0127] Figure 20 is a flowchart illustrating the details of step ST5a in Figure 19. The luminance signal extraction unit 15a uses the measurement area ri(k) set by the measurement area setting unit 14 in step ST3 of Figure 19 as the measurement area ri(k) to be used (step ST521). Then, the luminance signal extraction unit 15a extracts the time-series pulse wave source signal wi(t) during the pulse wave estimation period from each of the measurement areas ri(k) used on the frame Im(k) based on the frame Im(k) acquired by the image acquisition unit 11 in step ST1 of Figure 9 and the measurement area information R(k) output from the measurement area setting unit 14 in step ST3 of Figure 9 (step ST522). The luminance signal extraction unit 15a extracts the pulse wave source signal wi(t) and generates pulse wave source signal information W(t) that shows the extracted time-series pulse wave source signal wi(t). The luminance signal extraction unit 15a outputs the generated pulse wave source signal information W(t) to the luminance signal selection unit 16b.

[0128] Figure 21 is a flowchart illustrating the details of step ST6b in Figure 19. The luminance signal selection unit 16b performs optimization on the time-series pulse wave source signal wi(t) extracted by the luminance signal extraction unit 15a during the pulse wave estimation period, based on the weight coefficients set by the weight setting unit 19 for the measurement area ri(k) from which the time-series pulse wave source signal wi(t) was extracted (step ST621). The luminance signal selection unit 16b selects the time-series pulse wave source signal wi(t) after optimization as the time-series pulse wave source signal wi(t) for estimation (step ST622). The luminance signal selection unit 16b outputs the selected pulse wave source signal information W(t) to the pulse wave estimation unit 17.

[0129] Thus, the pulse wave estimation device 1b sets a measurement region ri(k) in the region corresponding to the skin region on the captured image, which can be used to extract a pulse wave source signal wi(t) that shows a change in brightness and contains the pulse wave component of the subject. Specifically, the pulse wave estimation device 1a determines a measurement region ri(k) to be set in the region corresponding to the skin region on the captured image based on information indicating the target face orientation range, layout information, and reference measurement region information, and sets the determined measurement region ri(k). The pulse wave estimation device 1b sets a measurement area ri(k) to be used from among the measurement areas ri(k), extracts a time-series pulse wave source signal wi(t) based on the brightness change in the measurement area ri(k), and, taking into account the subject's face orientation estimated frame by frame based on the captured image, selects a time-series pulse wave source signal wi(t) to be used for estimating the subject's pulse wave from among the time-series pulse wave source signals wi(t) extracted during the pulse wave estimation period. The pulse wave estimation device 1b estimates the subject's pulse wave based on the selected time-series pulse wave source signal wi(t). In detail, the pulse wave estimation device 1b includes a weight setting unit 19 that sets weight coefficients for each measurement area ri(k) set by the measurement area setting unit 14 in the captured image in which the face orientation of the person (subject) has been estimated, based on the face orientation of the person (subject) estimated on a frame-by-frame basis by the face orientation estimation unit 13. The luminance signal extraction unit 15a extracts the pulse wave source signal wi(t) using the measurement area ri(k) set by the measurement area setting unit 14 as the measurement area ri(k) to be used as the measurement area ri(k). The luminance signal selection unit 16b selects the time-series pulse wave source signal wi(t) after performing optimization on the pulse wave source signal wi(t) extracted by the luminance signal extraction unit 15a based on the weight coefficients set by the weight setting unit 19 for the measurement area ri(k) from which the pulse wave source signal wi(t) was extracted, as the time-series pulse wave source signal wi(t) for estimation. As a result, the pulse wave estimation device 1b can extract a luminance signal containing sufficient pulse wave components to estimate a person's pulse wave, in other words, a pulse wave source signal wi(t), from the skin region of the captured image. Consequently, the pulse wave estimation device 1b can prevent a decrease in the accuracy of estimating the pulse wave of a subject due to the skin region of the person whose pulse wave is to be estimated, i.e., the subject, not being captured in the captured image, or due to so-called shadowing occurring in that skin region.

[0130] The hardware configuration of the pulse wave estimation device 1b according to Embodiment 3 is the same as the hardware configuration of the pulse wave estimation device 1 described using Figures 12A and 12B in Embodiment 1, so it is omitted from the illustration. In Embodiment 3, the functions of the image acquisition unit 11, skin region detection unit 12, face orientation estimation unit 13, measurement region setting unit 14, brightness signal extraction unit 15a, brightness signal selection unit 16b, pulse wave estimation unit 17, output unit 18, and weight setting unit 19 are realized by the processing circuit 101. That is, the pulse wave estimation device 1b includes a processing circuit 101 for extracting a brightness signal containing sufficient pulse wave components to estimate a person's pulse wave from the skin region on the captured image, and for controlling the estimation of the subject's pulse wave from the extracted brightness signal.

[0131] The processing circuit 101 reads and executes a program stored in the memory 105, thereby executing the functions of the image acquisition unit 11, the skin region detection unit 12, the face orientation estimation unit 13, the measurement region setting unit 14, the brightness signal extraction unit 15a, the brightness signal selection unit 16b, the pulse wave estimation unit 17, the output unit 18, and the weight setting unit 19. In other words, the pulse wave estimation device 1b includes a memory 105 for storing a program that, when executed by the processing circuit 101, will result in the execution of steps ST1 to ST7 in Figure 19 described above. It can also be said that the program stored in the memory 105 causes the computer to execute the processing procedures or methods of the image acquisition unit 11, the skin region detection unit 12, the face orientation estimation unit 13, the measurement region setting unit 14, the brightness signal extraction unit 15a, the brightness signal selection unit 16b, the pulse wave estimation unit 17, the output unit 18, and the weight setting unit 19. The storage unit, which is not shown in the diagram, is composed of, for example, memory 105. Furthermore, the pulse wave estimation device 1b includes devices such as an imaging device 2, and an input interface device 102 and an output interface device 103 for wired or wireless communication.

[0132] In the above embodiment 3, the subject was the vehicle driver, but this is merely one example. The subject may be a passenger other than the vehicle driver.

[0133] Furthermore, in the above embodiment 3, the pulse wave estimation device 1b was an in-vehicle device, and the image acquisition unit 11, skin area detection unit 12, face orientation estimation unit 13, measurement area setting unit 14, brightness signal extraction unit 15a, brightness signal selection unit 16b, pulse wave estimation unit 17, output unit 18, and weight setting unit 19 were provided in the in-vehicle device. The system is not limited to this, however, some of the following components may be mounted on the vehicle's in-vehicle device: the image acquisition unit 11, the skin region detection unit 12, the face orientation estimation unit 13, the measurement region setting unit 14, the brightness signal extraction unit 15a, the brightness signal selection unit 16b, the pulse wave estimation unit 17, the output unit 18, and the weight setting unit 19, while the others are provided on a server connected to the in-vehicle device via a network, with the in-vehicle device and the server forming the system. Alternatively, the image acquisition unit 11, skin area detection unit 12, face orientation estimation unit 13, measurement area setting unit 14, brightness signal extraction unit 15a, brightness signal selection unit 16b, pulse wave estimation unit 17, output unit 18, and weight setting unit 19 may all be provided on the server.

[0134] Furthermore, the pulse wave estimation device 1b according to the above embodiment 3 is not limited to in-vehicle devices mounted on vehicles, but can also be applied to other mobile devices or home appliances, for example. Also, the subjects are not limited to vehicle occupants, but can be various people.

[0135] Furthermore, in the above embodiment 3, the pulse wave estimation device 1b determined the measurement area setting unit 14 to set the measurement area ri(k) to be set in the area corresponding to the skin area on the captured image based on information indicating the target face orientation range, layout information, and reference measurement area information, but this is only one example. For example, if the target face orientation range is defined as the range of all face orientations that the subject can take within the range that the imaging device 2 can capture, and it is assumed that the imaging device 2 always captures the subject at a reference position and reference orientation, then the measurement area setting unit 14 does not need to consider the target face orientation range and layout information. In this case, for example, an administrator or the like may determine in advance, based on the image captured by the imaging device 2 of the assumed subject, a measurement region ri(k) that can be used to extract the pulse wave source signal wi(t) containing the subject's pulse wave component, generate information indicating the determined measurement region ri(k) as reference measurement region information, and store it in a memory unit or the like. The measurement area setting unit 14 does not take into account the information indicating the target face orientation range or the layout information, and simply sets the measurement area ri(k) defined in the reference measurement area information to the area corresponding to the skin area on the captured image.

[0136] As described above, according to Embodiment 3, the pulse wave estimation device 1b includes an image acquisition unit 11 that acquires captured images of a person (subject) on a frame-by-frame basis, a skin region detection unit 12 that detects the skin region of a person from the captured image, a measurement region setting unit 14 that sets a measurement region ri(k) in the region corresponding to the skin region on the captured image, which can be used to extract a pulse wave source signal wi(t) that shows a change in brightness and contains the pulse wave component of a person, a face orientation estimation unit 13 that estimates the face orientation of a person on a frame-by-frame basis based on the captured image, and a pulse wave source signal w from the measurement region ri(k) set by the measurement region setting unit 14. The pulse wave estimation device 1b is configured to include: a luminance signal extraction unit 15a that sets a measurement area ri(k) to be used to extract i(t) and extracts a time-series pulse wave source signal wi(t) based on the luminance change in the set measurement area ri(k); a luminance signal selection unit 16b that selects a time-series pulse wave source signal wi(t) to be used for estimating a person's pulse wave from the time-series pulse wave source signals wi(t) extracted by the luminance signal extraction unit 15a during the pulse wave estimation period, taking into account the person's face orientation estimated by the face orientation estimation unit 13; and a pulse wave estimation unit 17 that estimates a person's pulse wave based on the time-series pulse wave source signal wi(t) selected by the luminance signal selection unit 16b. Therefore, the pulse wave estimation device 1b can prevent a decrease in the accuracy of estimating the pulse wave of a subject due to the skin area of ​​the person whose pulse wave is to be estimated, i.e., the subject, not being captured in the captured image, or so-called shadows occurring in the skin area.

[0137] In detail, the pulse wave estimation device 1b includes a weight setting unit 19 that sets weight coefficients for each measurement area ri(k) set by the measurement area setting unit 14 in the captured image in which the face orientation of the person (subject) is estimated based on the face orientation of the person (subject) estimated on a frame-by-frame basis by the face orientation estimation unit 13; the luminance signal extraction unit 15a extracts the pulse wave source signal wi(t) using the measurement area set by the measurement area setting unit 14 as the measurement area ri(k) to be used; and the luminance signal selection unit 16b selects the time-series pulse wave source signal wi(t) after performing optimization on the pulse wave source signal wi(t) extracted by the luminance signal extraction unit 15a based on the weight coefficients set by the weight setting unit 19 for the measurement area ri(k) from which the pulse wave source signal wi(t) was extracted, as the time-series pulse wave source signal wi(t) for estimation. Therefore, the pulse wave estimation device 1b can prevent a decrease in the accuracy of estimating the pulse wave of a subject due to the skin area of ​​the person whose pulse wave is to be estimated, i.e., the subject, not being captured in the captured image, or due to so-called shadowing occurring in that skin area.

[0138] In the embodiments 1 to 3 described above, the pulse wave estimation unit 17 analyzed multiple principal components using a general signal separation technique such as PCA or ICA, generated a separated signal representing the analyzed principal components, and estimated the subject's pulse wave based on the separated signal. This is merely one example, and the pulse wave estimation unit 17 may estimate the subject's pulse wave by other methods. For example, the pulse wave estimation unit 17 may perform a Fourier transform on the selected pulse wave source signal information W(t) and calculate the peak frequency in the frequency power spectrum as the pulse rate.

[0139] Furthermore, in embodiments 1 to 3 described above, the measurement area setting unit 14 was assumed to set multiple measurement areas ri(k), and the brightness signal extraction units 15 and 15a were assumed to extract time-series pulse wave source signals wi(t) from the multiple measurement areas ri(k) used. However, this is just one example. In embodiments 1 to 3 described above, the time-series pulse wave source signal wi(t) may be extracted from a single measurement region ri(k). However, if the time-series pulse wave source signal wi(t) is extracted from a single measurement area ri(k), the pulse wave estimation unit 17 estimates the subject's pulse wave using a method that does not employ general signal separation techniques such as PCA or ICA (for example, by performing a Fourier transform on the selected pulse wave source signal information W(t) described above). [Industrial applicability]

[0140] The pulse wave estimation device according to this disclosure can prevent a situation in which the luminance signal of the subject's skin region, which should be used to estimate the subject's pulse wave, cannot be extracted because, during noise reduction, the pulse wave signal included in the luminance signal of the subject's skin region is also considered a noise component and removed. [Explanation of symbols]

[0141] 1,1a,1b Pulse wave estimation device, 11 Image acquisition unit, 12 Skin area detection unit, 13 Face orientation estimation unit, 14 Measurement area setting unit, 15,15a Brightness signal extraction unit, 16,16a,16b Brightness signal selection unit, 17 Pulse wave estimation unit, 18 Output unit, 19 Weight setting unit, 2 Imaging device, 101 Processing circuit, 102 Input interface device, 103 Output interface device, 104 Processor, 105 Memory.

Claims

1. An image acquisition unit that acquires captured images of people on a frame-by-frame basis, A skin region detection unit for detecting the human skin region from the captured image, A measurement area setting unit sets a measurement area in the region of the captured image corresponding to the skin area, which can be used to extract a pulse wave source signal that shows a change in brightness and contains the pulse wave component of the person; A face orientation estimation unit that estimates the orientation of the person's face on a frame-by-frame basis based on the captured image, A luminance signal extraction unit sets a measurement area to be used for extracting the pulse wave source signal from among the measurement areas set by the measurement area setting unit, and extracts the pulse wave source signal in a time series based on the luminance change in the set measurement area. A luminance signal selection unit selects a time-series pulse wave source signal to be used for estimating the person's pulse wave from among the time-series pulse wave source signals extracted by the luminance signal extraction unit during the pulse wave estimation period, taking into consideration the person's face orientation estimated by the face orientation estimation unit. A pulse wave estimation unit estimates the pulse wave of the person based on the time-series pulse wave source signal selected by the luminance signal selection unit. A pulse wave estimation device equipped with the following.

2. The measurement area setting unit is, Based on information indicating the target face orientation range, which is the range of the face orientation of the person to be targeted for pulse wave estimation; layout information indicating the positional relationship between the imaging device that captures the captured image and the person assumed to be the target person; and reference measurement area information which associates the measurement area with the face orientation of the person, from which it is assumed that the pulse wave source signal containing the pulse wave component of the person can be extracted when the imaging device captures the person assumed to be at a reference position and reference orientation, the measurement area to be set in the area corresponding to the skin area on the captured image is determined. The pulse wave estimation device according to claim 1, characterized by the features described above.

3. The luminance signal extraction unit extracts the pulse wave source signal using the measurement area that corresponds to the face orientation of the person estimated by the face orientation estimation unit based on the captured image in which the measurement area is set, from among the measurement areas set by the measurement area setting unit, as the measurement area to be used. The luminance signal selection unit calculates the distribution ratio of the person's face orientation for each frame estimated by the face orientation estimation unit during the pulse wave estimation period, and selects the time-series pulse wave source signal extracted from the measurement area commonly set in the captured image from which the person's face orientation, whose frequency of appearance is equal to or greater than the threshold for face orientation determination, as the time-series pulse wave source signal for estimation. A pulse wave estimation device according to claim 1 or 2, characterized in that it is a pulse wave estimation device.

4. The luminance signal extraction unit extracts the pulse wave source signal using the measurement area that corresponds to the face orientation of the person estimated by the face orientation estimation unit based on the captured image in which the measurement area is set, from among the measurement areas set by the measurement area setting unit, as the measurement area to be used. The luminance signal selection unit calculates the distribution ratio of the measurement areas used from which the luminance signal extraction unit extracted the pulse wave source signals during the pulse wave estimation period, and selects the time-series pulse wave source signals extracted from the measurement areas used whose occurrence frequency is equal to or greater than the threshold for area determination as the time-series pulse wave source signals for estimation. A pulse wave estimation device according to claim 1 or 2, characterized in that it is a pulse wave estimation device.

5. The weight setting unit sets weight coefficients for each measurement region set by the measurement region setting unit in the captured image in which the face orientation of the person has been estimated based on the face orientation of the person estimated on a frame-by-frame basis. The luminance signal extraction unit extracts the pulse wave source signal using the measurement area set by the measurement area setting unit as the measurement area to be used. The luminance signal selection unit selects the time-series pulse wave source signal, which has been optimized based on the weight coefficients set by the weight setting unit for the measurement area used from which the pulse wave source signal was extracted, as the time-series estimated pulse wave source signal. A pulse wave estimation device according to claim 1 or 2, characterized in that it is a pulse wave estimation device.

6. The aforementioned person is the driver of the vehicle. A pulse wave estimation device according to claim 1 or 2, characterized in that it is a pulse wave estimation device.

7. The image acquisition unit acquires captured images of a person frame by frame. The skin area detection unit performs the step of detecting the human skin area from the captured image, The measurement area setting unit sets a measurement area in the region of the captured image corresponding to the skin region, which can be used to extract a pulse wave source signal that shows a change in brightness and contains the pulse wave component of the person. The face orientation estimation unit estimates the face orientation of the person on a frame-by-frame basis based on the captured image, The luminance signal extraction unit sets a measurement area to be used for extracting the pulse wave source signal from among the measurement areas set by the measurement area setting unit, and extracts the pulse wave source signal in a time series based on the luminance change in the set measurement area. The luminance signal selection unit selects a time-series pulse wave source signal to be used for estimating the person's pulse wave from among the time-series pulse wave source signals extracted by the luminance signal extraction unit during the pulse wave estimation period, taking into consideration the person's face orientation estimated by the face orientation estimation unit. The pulse wave estimation unit estimates the person's pulse wave based on the time-series pulse wave source signal selected by the brightness signal selection unit. A pulse wave estimation method equipped with the following.

Images