Pulse wave assessment device, condition assessment device and pulse wave assessment method

Abstract

The system comprises an image acquisition unit (11) that acquires an image depicting a person, a skin area detection unit (12) that detects a skin area of ​​the person based on the image, a measurement area setting unit (13) that sets a measurement area in the image corresponding to the skin area for extracting a pulse wave signal that shows a luminance change, a pulse wave origin signal extraction unit (14) that extracts a pulse wave origin signal based on the luminance change in the measurement area of ​​the image, and a simulation signal generation unit (15) that generates a simulation signal based on position changes of the measurement area and a model of the distribution of ambient light in the image area, simulating the luminance changes of the measurement area caused by the position changes of the measurement area under the ambient light.and a pulse wave estimation unit (16) that estimates the person's pulse wave based on the pulse wave origin signal and the simulation signal.

Description

Technical field

[0001] The present disclosure relates to a pulse wave assessment device, a condition assessment device, and a pulse wave assessment method. Background technology

[0002] A technology is known in which a person's pulse wave is estimated without contact by detecting subtle luminance changes in the skin surface of a person (hereinafter referred to as the "skin area") based on a luminance signal of an area encompassing the person's skin in an image projected by an imaging device. If the person being assessed (hereinafter referred to as the "test subject") moves, the position of the test subject's skin area in the image changes, and this change in position reduces the precision of the pulse wave assessment. A conventional technology is known in which a signal based on a change in the position of the skin area is suppressed as noise using a luminance signal of the test subject's skin area.For example, patent document 1 discloses a pulse detection device by which specific frequency components obtained by a frequency analysis of position information of a face area in an imaging area are suppressed as noise in image signals from the face area. State-of-the-art documents, patent documents

[0003] Patent document 1: Patent publication no. JP 2018-68720 A Overview of the invention Problem to be solved by the invention

[0004] In a scene where the ambient light distribution in the imaging area is uneven, the pulse wave, which is estimated based on subtle luminance changes of a test subject's skin surface in the image, is influenced not only by changes in the position of the skin area but also by the ambient light. However, the conventional technology disclosed in patent document 1 does not take this into account, resulting in a decrease in the precision of the test subject's pulse wave estimation.

[0005] The present disclosure was made to solve this problem and aims to provide a pulse wave assessment device by which the pulse wave of the test subject can be assessed even in a scene with an uneven distribution of ambient light in the imaging light. Means of solving the task

[0006] The pulse wave assessment device according to the present disclosure comprises an image acquisition unit for acquiring an image depicting a person; a skin area detection unit for detecting a skin area of ​​the person from the image; a measurement range setting unit for setting a measurement range for extracting a pulse wave origin signal exhibiting a luminance change in an area corresponding to the skin area in the image; and a pulse wave origin signal extraction unit for extracting the pulse wave origin signal based on the luminance change in the measurement range of the image.a simulation signal generation unit for generating a simulation signal based on a change in the position of the measuring range and a distribution model of the ambient light in an imaging area, wherein the simulation signal simulates the luminance change of the measuring range caused by the changes in the position of the measuring range under the ambient light; and a pulse wave estimation unit for estimating a pulse wave of the person based on the pulse wave origin signal and the simulation signal. Effects of the invention

[0007] According to the present disclosure, the pulse wave of the test subject can be assessed even if the scene has an uneven distribution of ambient light in the imaging area. Brief explanation of the drawings [ Fig. 1] Fig. Figure 1 is a view showing a structural example of a state assessment device comprising a pulse wave assessment device according to a first embodiment. [ Fig. 2] Fig. Figure 2 is a view showing a structural example of the pulse wave assessment device according to the first embodiment. [ Fig. 3] Fig. 3A, Fig. 3B and Fig. Figure 3C are views illustrating examples of a method for setting a measurement range by the measurement range setting unit in the pulse wave estimation device according to the first embodiment. [ Fig. 4] Fig. Figure 4 is a view which, in the first embodiment, schematically shows an example of an ambient light model in an illustration when the two-dimensional Gaussian distribution is used in the ambient light model. [ Fig. 5] Fig. Figure 5 is a flowchart to explain the operation of the pulse wave assessment device according to the first embodiment. [ Fig. 6] Fig. Figure 6 is a flowchart to explain the operation of the condition assessment device according to the first embodiment. [ Fig. 7] Fig. Figure 7 shows views illustrating the influence of ambient light on a signal used for pulse wave assessment of a test subject based on an image when the ambient light is unevenly distributed, where 7A is a view showing an example of an image taken in a scene with a uniform distribution of ambient light, and Fig. 7B is a view showing an example of an illustration image taken in a scene with an uneven distribution of ambient light. [ Fig. 8] Fig. Figure 8 is a view illustrating an example of a method for generating a simulation signal by a simulation signal generation unit based on a plurality of ambient light models in the first embodiment. [ Fig. 9] Fig. Figure 9 is a view which, in the first embodiment, shows an example of a bounding rectangle of the measuring area and its central coordinates, and a representation of the distance between the vertex coordinates of a corner of the measuring area and the corresponding central coordinates. [ Fig. 10] Fig. Figure 10 is a view showing an example of a first simulation signal and a second simulation signal in the first embodiment, where the simulation signal generation unit multiplies the signal by a normalization coefficient and thereby adjusts the difference to the order of magnitude. [ Fig. 11] Fig. 11A and Fig. Figure 11B shows examples of the hardware setup of the pulse wave assessment device and the condition assessment device of the first embodiment. Forms for carrying out the invention

[0008] The following section explains in detail the methods for implementing the present disclosure with reference to the drawings. First embodiment

[0009] Fig. Figure 1 is a view showing a structural example of a state assessment device 2 comprising a pulse wave assessment device 1 according to the first embodiment.

[0010] The pulse wave assessment device 1 estimates the pulse wave of a person based on an image depicting that person. The person whose pulse wave is being assessed by the pulse wave assessment device 1 is referred to as the "test subject".

[0011] The pulse wave assessment device 1 acquires an image consisting of a series of frames Im(k) depicting an area in which at least one skin area should be located, encompassing the skin of the test subject (hereinafter referred to as the "skin presence area"), at a predetermined frame rate Fr. Fr does not represent a frame number assigned to the respective frames. For example, the frame assigned at the time following frame Im(k) is frame Im(k + 1). In the first embodiment, the skin area corresponds to the face of the test subject. This is merely an example, and the skin area may also be a different area than the face of the test subject.For example, the skin area could also be an area corresponding to a part of the face, such as the eyes, eyebrows, nose, mouth, forehead, cheeks, or chin of the test subject. Furthermore, the skin area could also be an area corresponding to a body part other than the face, such as the head, shoulders, hands, neck, or legs of the test subject. The skin area could also be multiple areas.

[0012] The pulse wave assessment device 1 then estimates the test subject's pulse wave based on a series of frames Im(k - Tp + 1) to Im(k) per a specific frame rate Tp, and outputs a pulse wave assessment result P(t) representing the estimated pulse wave (hereinafter referred to as "pulse wave information"). Specifically, the pulse wave assessment device 1 estimates the test subject's pulse wave by suppressing a signal that would become noise and is based on positional changes of the test subject's skin area from a signal based on luminance changes of the test subject's skin area in the series of frames Im(k - Tp + 1) to Im(k). The signal to be suppressed is generated by the pulse wave assessment device 1 taking into account the ambient light distribution in the imaging area.

[0013] Here, t represents an output number assigned for each specific frame rate Tp. For example, the pulse wave assessment result assigned at the time following the pulse wave assessment result P(t) is the pulse wave assessment result P(t + 1). The frame number k and the output number t are integers greater than or equal to 1. The frame rate Tp is an integer greater than or equal to 2.

[0014] A signal used in the assessment of the test subject's pulse wave, representing luminance changes in the test subject's skin area, includes, in addition to a pulse wave-based signal, for example, a signal based on the test subject's movement. A movement-based signal becomes noise, which reduces the accuracy of the pulse wave assessment. Furthermore, a signal representing a luminance change in the test subject's skin area is also influenced by the distribution of ambient light. In this case, the pulse wave assessment device 1 assesses the test subject's pulse wave by suppressing a signal that, taking ambient light into account, is based on the test subject's movement—in other words, on a change in the position of the test subject's skin area within the frame—as a noise signal.In the first embodiment, a “movement of the test subject” that causes a luminance change in the image, which is considered noise in relation to the assessment of the test subject’s pulse wave, includes movement resulting from a shift in the test subject’s head position, movement resulting from a change in facial direction, expressive movement (e.g., a laughing face or a mouth moving while speaking), and movement due to a detection deviation of the range for extracting a signal showing a luminance change (measurement range, which is explained in detail below).

[0015] The number of test subjects depicted in the illustration can be one person or several. For the sake of simplicity, the following explanation of the first embodiment will assume a single test subject depicted in the illustration.

[0016] The state assessment device 2 acquires the pulse wave assessment result P(t) from the pulse wave assessment device 1 and assesses the state of the test subject. In the first embodiment, the state of the test subject is assumed to be, as an example, a state in which the test subject is alert or not. Based on the pulse wave assessment result P(t), the state assessment device 2 assesses the level of alertness, which represents the degree of alertness of the test subject.

[0017] The state assessment device 2 outputs state information Z(t) regarding the assessed state of the test subject to an output device 3. For example, if the test subject's level of alertness has decreased, the state assessment device 2 outputs state information Z(t) to the output device 3, expressing that the test subject's level of alertness has decreased.

[0018] Output device 3 outputs information based on the state information Z(t) output by state assessment device 2. Output device 3 is, for example, an audio output device. If state assessment device 2 outputs, for example, state information Z(t) indicating that the test subject's alertness level has decreased, output device 3 emits a warning tone to alert the test subject to a decrease in their level of attention.

[0019] In the first embodiment, it is assumed, by way of example, that the pulse wave assessment device 1, the condition assessment device 2, and the output device 3 are installed in a vehicle (not shown), and the test subject is the driver of the vehicle. That is, the pulse wave assessment device 1 assesses the pulse wave of the vehicle driver. Furthermore, the condition assessment device 2 assesses the driver's condition. Then, based on the driver's condition as assessed by the condition assessment device 2, the output device 3 emits a warning tone to the driver.

[0020] Details of a structural example of the in Fig. The condition assessment device 2 shown in Figure 1 is described below, and first a structural example of the pulse wave assessment device 1 is explained in detail.

[0021] Fig. Figure 2 is a view showing a structural example of the pulse wave assessment device 1 according to the first embodiment.

[0022] As in Fig. As shown in Figure 2, the pulse wave assessment device 1 comprises an imaging unit 11, a skin area detection unit 12, a measurement range setting unit 13, a pulse wave origin signal extraction unit 14, a simulation signal generation unit 15, a pulse wave assessment unit 16 and an output unit 17.

[0023] Image acquisition unit 11 acquires an image depicting the test subject. More precisely, image acquisition unit 11 acquires an image depicting the driver of the vehicle, captured by an imaging device (not shown) installed in the vehicle. The imaging device is mounted in such a way that it can capture an area of ​​the driver's skin.

[0024] The image acquisition unit 11 outputs the acquired image to the skin area detection unit 12.

[0025] The imaging device of the first embodiment will now be explained. The imaging device comprises an imaging unit (not shown) and an illumination unit (not shown). The illumination unit comprises, for example, an LED (Light Emitting Diode). The illumination unit emits light into an imaging area of ​​the imaging unit. The imaging unit images the imaging area into which light has been emitted by the illumination unit. The imaging unit can comprise one illumination unit or a plurality of illumination units. In the first embodiment, as an example, two illumination units are included, which are positioned between each other at locations adjacent to the imaging unit. The illumination light from the illumination units is emitted most intensely near the center of the imaging area of ​​the imaging unit.As a result, the luminance in the image projected by the imaging unit is strongest in the center, and the luminance decreases from the center towards the outer section of the image. In the first embodiment, the "center" is not strictly limited to the "center" but essentially encompasses the center as well.

[0026] The skin area detection unit 12 detects a skin area of ​​the test subject based on the frame Im(k) included in the image acquired by the image acquisition unit 11. The skin area detection unit 12 can detect the skin area using a known method. For example, the skin area detection unit 12 can detect a skin area using a cascade-type facial sensor that uses hair-like feature sizes.

[0027] The skin area detection unit 12 generates skin area information S(k) that represents the detected skin area.

[0028] The skin area information S(k) can include information indicating the presence or absence of a skin area and information representing the position and size of a detected skin area in the image. In the first embodiment, a skin area is represented by a rectangular area in the image, and the skin area information S(k) includes information representing the position and size of the rectangular area in the image.

[0029] Specifically, the skin area information S(k) represents, for example, the presence / absence of evidence of the test subject's face, the central coordinates Fc (Fcx, Fcy) of a rectangle enclosing the test subject's face in the image, and the width Fcw and height Fch of the rectangle, if the skin area corresponds to the test subject's face. The presence / absence of evidence of the test subject's face is expressed, for example, by "1" if evidence was possible and by "0" if it was not. Furthermore, the central coordinates of the rectangle surrounding the face are represented by the coordinate system of frame Im(k). In frame Im(k), the origin is in the upper left, the direction to the right in frame Im(k) is the positive direction of the x-axis, and the direction downwards in frame Im(k) is the positive direction of the y-axis.

[0030] The skin area detection unit 12 outputs the generated skin area information S(k) to the measuring range setting unit 13.

[0031] Based on the frame Im(k) of the image acquired by the image acquisition unit 11 and the skin area information S(k) output by the skin area detection unit 12, the measurement range setting unit 13 sets a plurality of measurement ranges within the image area of ​​frame Im(k) corresponding to the skin area represented by the skin area information S(k). This allows the unit to extract a pulse wave origin signal indicating luminance changes. The measurement range setting unit 13 can acquire the image acquired by the image acquisition unit 11 via the skin area detection unit 12. Once the plurality of measurement ranges has been set, the measurement range setting unit 13 generates measurement range information R(k), which represents the plurality of set measurement ranges.The measurement range information R(k) comprises information representing the position and size of Rn-piece (positive integer) measurement ranges in the diagram. The respective measurement ranges are defined as measurement ranges ri(k) (i = 1, 2, ..., Rn). In the first embodiment, the measurement range ri(k) is a quadrilateral, and the position and size of the measurement range ri(k) are represented in the diagram by the coordinate values ​​of the four vertices of the quadrilateral.

[0032] This includes Fig. 3A, Fig. 3B and Fig. 3C Views to illustrate the example of a method for setting a measuring range by the measuring range setting unit 13 in the pulse wave estimation device 1 according to the first embodiment.

[0033] By means of Fig. Section 3 provides an example of the procedure for setting a plurality of measuring ranges using the measuring range setting unit 13.

[0034] First, as in Fig. 3A and Fig. Figure 3B shows that, through the measuring range setting unit 13 in a skin area sr, represented by the skin area information S(k), Ln-piece (positive integer) orientation points of facial organs such as the outer corner of the eye, the inner corner of the eye, the nose and the mouth are detected. Fig. 3A and Fig. In 3B, the orientation points are represented by circles. A vector in which the measuring range setting unit 13 has stored the coordinate values ​​of the detected orientation points is specified as L(k).

[0035] The measuring range setting unit 13 can detect the facial organs using a known method such as a model called a Constrained Local Model (CLM).

[0036] Next, the measuring range setting unit 13 sets the vertex coordinates of the quadrilaterals of the measuring ranges ri(k) based on the proven orientation points. For example, the measuring range setting unit 13 sets the vertex coordinates of quadrilaterals as shown in Fig. 3C is shown, where it sets Rn-piece measuring ranges ri(k).

[0037] To illustrate the setting of the measurement ranges ri(k) by the measurement range setting unit 13 in a section of the skin area sr corresponding to a cheek, the measurement range setting unit 13 selects an orientation point LA1 of the facial contour and an orientation point LA2 of the nose. The measurement range setting unit 13 can first select the orientation point LA2 of the nose, and then select the orientation point LA1 of the facial contour that is closest to the orientation point LA2 of the nose.

[0038] The measuring range setting unit then sets 13 auxiliary orientation points a1, a2 and a3, so that a line segment between orientation point LA1 and orientation point LA2 is divided into four parts.

[0039] Similarly, the measuring range setting unit 13 selects an orientation point LB1 of the facial contour and an orientation point LB2 of the nose. Furthermore, the measuring range setting unit 13 sets auxiliary orientation points b1, b2, and b3, so that a line segment between orientation point LB1 and orientation point LB2 is divided into four parts. Orientation points LB1 and LB2 can, for example, be selected as orientation points of the facial contour and the nose, respectively, that are adjacent to orientation points LA1 and LA2. The measuring range setting unit 13 sets a quadrilateral area encompassing the auxiliary orientation points a1, b1, b2, and a2 as the measuring range R1. The auxiliary orientation points a1, b1, b2, and a2 each become the corner point coordinates corresponding to the measuring range R1.

[0040] Likewise, the measuring range setting unit 13 sets a measuring range R2 encompassing the auxiliary orientation points a2, b2, b3 and a3, and the corner point coordinates of the measuring range R2.

[0041] An example of setting the measuring range ri(k) in the section corresponding to the cheek has been explained here, whereby the measuring range setting unit 13, for example, also sets a measuring range ri(k) and corner point coordinates of the measuring range ri(k) with respect to a skin area sr of another section of the cheek and a section corresponding to the chin. Fig. Although this is not shown in Figure 3C, the measuring range setting unit 13 can also set the measuring range ri(k) in a section corresponding to the forehead of the skin area sr of the test subject or in a section corresponding to the tip of the nose.

[0042] The measuring range setting unit 13 outputs the generated measuring range information R(k) to the pulse wave origin signal extraction unit 14 and the simulation signal generation unit 15.

[0043] The pulse wave origin signal extraction unit 14 extracts a pulse wave origin signal, based on frame Im(k) of the image acquired by the image acquisition unit 11 and the measurement range information R(k) output by the measurement range setting unit 13. This pulse wave origin signal shows luminance changes within a specified period, in other words, within a period corresponding to the frame rate Tp, from the individual or multiple measurement ranges ri(k) in frame Im(k), which are represented by the measurement range information R(k). The pulse wave origin signal is the source signal of a pulse wave. The pulse wave estimation device 1 uses the pulse wave origin signal to estimate the pulse wave of the test subject. The pulse wave estimation unit 16 performs this estimation. Details of the pulse wave estimation unit 16 are described below.

[0044] The pulse wave origin signal extraction unit 14 can acquire the image acquired by the image acquisition unit 11 via the skin area detection unit 12 and the measurement range setting unit 13.

[0045] Once the pulse wave origin signal has been extracted, the pulse wave origin signal extraction unit generates 14 pulse wave origin signal information W(t), which represents the extracted origin signal.

[0046] The pulse wave origin signal information W(t) comprises information representing a pulse wave origin signal wi(t) extracted from the measurement range ri(k). The pulse wave origin signal wi(t), which consists of time series data for the Tp component, is extracted, for example, based on frames Im(k - Tp + 1), Im(k - Tp + 2), ..., Im(k) of a past Tp component and measurement range information R(k - Tp + 1), R(k - Tp + 2), ..., R(k).

[0047] When extracting the pulse wave origin signal wi(t), the pulse wave origin signal extraction unit 14 calculates a luminance feature Gi(j) (j = k - Tp + 1, k - Tp + 2, ..., k) of the respective measurement range ri(k) for the respective frame Im(k) of the image. The luminance feature Gi(j) is a value calculated based on the luminance value in a frame Im(j) of the image for the respective measurement range ri(j). The luminance feature Gi(j) is an average or a standard deviation of the luminance value of the image elements included in the measurement range ri(j). In the first embodiment, the luminance feature Gi(j) is, by way of example, an average of the luminance value of the image elements included in the measurement range ri(j). The pulse wave origin signal extraction unit 14 sequences the Gi(j) calculated for the respective frame Im(k) of the image into time series and determines them as pulse wave origin signal wi(t).That is, the pulse wave origin signal extraction unit 14 determines the pulse wave origin signal wi(t) = [Gi(k - Tp + 1), Gi(k - Tp + 2), ..., Gi(k)].

[0048] The pulse wave origin signal extraction unit 14 generates the summarized pulse wave origin signals wi(t) in the respective measurement range ri(k) as pulse wave origin signal information W(t).

[0049] The pulse wave origin signal extraction unit 14 outputs the generated pulse wave origin signal information W(t) to the pulse wave assessment unit 16.

[0050] The pulse wave origin signal wi(t) includes, in addition to the pulse wave component and movement component (here, of the face) of the test subject described above, various noise components. For example, there are noise components due to a component defect in the imaging device. To suppress these noise components, filter processing as a pretreatment of the pulse wave origin signal wi(t) is desirable. For example, the pulse wave origin signal extraction unit 14 performs this filter processing.

[0051] In filter processing, the pulse wave origin signal wi(t) is processed, for example, using a low-pass filter, high-pass filter, or band-pass filter. The following explanation describes the application of a band-pass filter.

[0052] A Butterworth filter, for example, can be used as a bandpass filter. Desirable cutoff frequencies for the bandpass filter include, for example, a low cutoff frequency of 0.5 Hz and a high cutoff frequency of 5.0 Hz.

[0053] The simulation signal generation unit 15 generates a signal (hereinafter referred to as the "simulation signal") based on the frame Im(k) of the image acquired by the image acquisition unit 11, the measurement range information R(k) output by the measurement range setting unit 13, and an ambient light model. This signal simulates luminance changes of the measurement range that occur under ambient light within a specified period, in other words, within a period corresponding to the frame rate Tp, due to a change in the position of the measurement range. The simulation signal is a signal that becomes noise and is suppressed in the pulse wave origin signal during pulse wave estimation.

[0054] The simulation signal generation unit 15 generates the simulation signal for the respective plurality of measurement ranges ri(k), which are represented by the measurement range information R(k).

[0055] The simulation signal generation unit 15 can acquire the image acquired by the image acquisition unit 11 via the skin area detection unit 12 and the measurement range setting unit 13.

[0056] When the simulation signal generation unit 15 generates the simulation signal for the respective measurement ranges ri(k), simulation signal information M(t) adapted to the generated simulation signal is produced.

[0057] The simulation signal information M(t) comprises information representing the simulation signal mi(t) generated for the measurement range ri(k). The simulation signal mi(t), which consists of time series data for the Tp component, is generated, for example, based on frames Im(k - Tp + 1), Im(k - Tp + 2), ..., Im(k) of a past Tp component, measurement range information R(k - Tp + 1), R(k - Tp + 2), ..., R(k), and the ambient light model.

[0058] The simulation signal generation unit 15 is pre-equipped with the ambient light model.

[0059] The ambient light model is a model of the distribution of ambient light that, based on the spatial luminance distribution in an area into which the ambient light is emitted, outputs values ​​that simulate luminance values ​​of coordinates in the image.

[0060] For example, if the ambient light in the imaging process is the illumination light of the illumination unit of the imaging device, or in other words, in the case of ambient light being emitted into an imaging area to image an image such that the center of the image is bright and the surroundings are dark, the ambient light model is the two-dimensional Gaussian distribution expressed in the following formula (1). qx,y=ae−(x−xc)2+(y−yc)2σ2

[0061] q_(x, y) is a value simulating the luminance value at coordinates (x, y) in the image, where (x_c, y_c) are the central coordinates of the two-dimensional Gaussian distribution. a is a parameter that defines the height of the two-dimensional Gaussian distribution. σ is a parameter that defines the spread of the two-dimensional Gaussian distribution. The central coordinates of the two-dimensional Gaussian distribution are determined based on the spatial luminance distribution as the light emitted by the lighting unit is acquired by the acquiring unit.

[0062] Fig. Figure 4 is a view which, in the first embodiment, shows an example of the distribution of ambient light expressed by an ambient light model in an illustration image when the two-dimensional Gaussian distribution is used in the ambient light model.

[0063] The in Fig. The ambient light model shown in Figure 4 is an ambient light model for the case where the ambient light is illumination light from the illumination unit of the imaging device.

[0064] The simulation signal generation unit 15 converts the coordinate values ​​(x, y) in the image, which includes the measurement ranges ri(k), into the simulation signal mi(t) using the ambient light model.

[0065] When generating the simulation signal mi(t), the simulation signal generation unit 15 calculates the luminance value at the coordinates in the respective measurement ranges ri(k) for the respective frame Im(k) of the image using the ambient light model as a simulated value Vi(j) (j = k - Tp + 1, k - Tp + 2, ..., k). The simulation signal generation unit 15 arranges the Vi(j) calculated for the respective frame Im(k) of the image into time series and determines them as the simulation signal mi(t). That is, the simulation signal generation unit 15 determines the simulation signal mi(t) = [Vi(k - Tp + 1), Vi(k - Tp + 2), ..., Vi(k)].

[0066] The simulation signal generation unit 15 generates the combined simulation signals mi(t) in the respective measurement range ri(k) as simulation signal information M(t).

[0067] In the preceding explanation, the ambient light model is the two-dimensional Gaussian distribution, but this is merely an example. Any model other than the two-dimensional Gaussian distribution can be used as the ambient light model. For example, the actually measured distribution of the illumination light can also be used as the ambient light model if the ambient light is the illumination light from the illumination unit of the imaging device. That is, there is no restriction to a model based on a formula; the ambient light model can also be generated based on values ​​obtained by actually measuring the brightness at each position in the image.

[0068] Furthermore, the simulation signal generation unit 15 can calculate a simulation signal mi(t) for the measuring ranges ri(k), or for each of the coordinate points that possess the respective measuring ranges ri(k). In the first embodiment, it is specified that the simulation signal generation unit 15 calculates, as an example, a simulation signal mi(t) for the measuring ranges ri(k). The simulation signal generation unit 15 calculates, for example, a simulation signal mi(t) for the measuring range ri(k) using the centroid coordinate of a quadrilateral of the measuring range ri(k).

[0069] The simulation signal generation unit 15 outputs the generated simulation signal information M(t) to the pulse wave assessment unit 16.

[0070] Based on the pulse wave origin signal information W(t) output by the pulse wave origin signal extraction unit 14 and the simulation signal information M(t) output by the simulation signal generation unit 15, the pulse wave assessment unit 16 estimates the test subject's pulse wave. That is, the pulse wave assessment unit 16 estimates the test subject's pulse wave based on the pulse wave origin signal wi(t) extracted by the pulse wave origin signal extraction unit 14 and the simulation signal mi(t) generated by the simulation signal generation unit 15.

[0071] The pulse wave assessment unit 16 outputs the pulse wave assessment result P(t), as pulse wave information representing the assessed pulse wave, to the output unit 17.

[0072] The pulse wave information could, for example, also consist of time-series data of the test subject's pulse wave, as assessed by pulse wave assessment unit 16, or of the test subject's pulse rate. For the sake of simplicity, it is assumed here that the pulse wave information refers to the test subject's pulse rate (number of beats per minute).

[0073] The procedure for assessing the pulse wave of the test subject using the pulse wave assessment unit 16 is explained in detail.

[0074] For example, by subtracting the simulation signal mi(t) from the pulse wave origin signal wi(t) using the pulse wave estimation unit 16, a pulse wave signal di(t) is calculated in which luminance changes due to movement of the test subject's skin, more precisely, the test subject's face, are suppressed. That is, the pulse wave estimation unit 16 calculates the pulse wave signal di(t) for the respective measurement range ri(k).

[0075] Next, the pulse wave assessment unit 16 calculates pulse wave signal information D(t) as the sum of the pulse wave signals di(t) corresponding to the respective measurement ranges ri(k). That is, the pulse wave assessment unit 16 calculates the pulse wave signal information D(t) once for all measurement ranges ri(k).

[0076] Then, in the pulse wave assessment unit 16, a Fourier transformation of the pulse wave signal information D(t) is performed and the peak frequencies in the frequency-power spectrum are calculated as the pulse rate.

[0077] For example, if the simulation signal generation unit 15 calculates simulation signals mi(t) for each of the coordinate points in the respective measurement ranges ri(k), the pulse wave estimation unit 16 multiplies a coefficient corresponding to the number of simulation signals mi(t) calculated for a measurement range ri(k) (hereinafter referred to as the "simulation signal coefficient") by the respective simulation signals mi(t). The pulse wave estimation unit 16 then subtracts the simulation signal mi(t) after multiplication by the simulation signal coefficient from the pulse wave origin signal wi(t).

[0078] To give a concrete example, the simulation signal generation unit 15 sets, for example, the maximum value of the amplitude of the simulation signal mi(t) for each of the coordinate points in the respective measurement ranges ri(k) to mi_amp(t), and the maximum value of the amplitude of the pulse wave origin signal wi(t) to wi_amp(t). Here, the pulse wave estimation unit 16 multiplies the simulation signal coefficient “wi_amp / mi_amp” by the respective simulation signals mi(t). Then, the pulse wave estimation unit 16 subtracts the respective simulation signals mi(t) from the pulse wave origin signal wi(t) after multiplication by “wi_amp / mi_amp”.

[0079] The pulse wave assessment unit 16 outputs the pulse wave assessment result P(t) to the output unit 17.

[0080] The output unit 17 outputs the pulse wave assessment result P(t) issued by the pulse wave assessment unit 16 to the pulse wave information acquisition unit 21 of the state assessment device 2.

[0081] The functions of the output unit 17 can also be included by the pulse wave assessment unit 16.

[0082] Next, a structural example of the condition assessment device 2 according to the first embodiment will be explained.

[0083] As in Fig. As shown in Figure 1, the state assessment device 2 comprises the pulse wave assessment device 1, the pulse wave information acquisition unit 21, a state assessment unit 22 and an output unit 23.

[0084] The pulse wave information acquisition unit 21 acquires the pulse wave assessment result P(t) output by the pulse wave assessment device 1.

[0085] The pulse wave information acquisition unit 21 outputs the acquired pulse wave assessment result P(t) to the state assessment unit 22.

[0086] The state assessment unit 22 assesses the test subject's state based on the pulse wave assessment result P(t) output by the pulse wave information acquisition unit 21, in other words, the test subject's pulse wave assessed by the pulse wave assessment device 1. In the first embodiment, the state assessment unit 22 assesses the test subject's state based on the test subject's pulse rate assessed by the pulse wave assessment device 1. Specifically, the state assessment unit 22 assesses the driver's alertness level as the driver's state based on the test subject's pulse rate, in other words, the driver's state. For example, the alertness level is expressed by alertness levels of two stages (1: tired, 2: awake).

[0087] For example, if the pulse wave information acquisition unit 21 outputs the pulse wave assessment result P(t), this is stored by the state assessment unit 22, which calculates a pulse rate of the test subject for 10 minutes after the start of driving as a reference pulse rate for that test subject based on the stored pulse wave assessment result P(t). The state assessment unit 22 can detect the start of driving, for example, by switching on the vehicle's power supply. Furthermore, the state assessment unit 22 can also detect the start of driving based on a signal from various sensors installed in the vehicle, such as a shift position sensor, etc.

[0088] Generally, it is assumed that a person is in a waking state immediately after starting to drive. Therefore, the state assessment unit 22 calculates the test subject's pulse rate for 10 minutes after starting to drive as a reference pulse rate, which serves as the criterion for determining whether the person is awake. If the test subject's pulse rate has fallen below the reference pulse rate, it can be assessed that the test subject's level of alertness has decreased.

[0089] The state assessment unit 22 assesses the test subject's level of alertness by comparing the pulse wave assessment result P(t) output by the pulse wave information acquisition unit 21—in other words, the test subject's pulse rate assessed by the pulse wave assessment device 1—with the reference pulse rate. If the test subject's pulse rate has decreased from the reference pulse rate by at least one predefined threshold (hereinafter referred to as the "decrease detection threshold"), the state assessment unit 22 detects a decrease in the alertness level, i.e., alertness level "1".

[0090] In the first embodiment, the state assessment unit 22 calculated the reference pulse rate based on the pulse wave assessment result P(t) output by the pulse wave information acquisition unit 21; in other words, the pulse wave of the test subject assessed by the pulse wave assessment device 1. This is merely an example. For instance, a general pulse rate for a person's waking state can also be preset as the reference pulse rate and stored in the state assessment unit 22. The state assessment unit 22 then assesses the test subject's alertness based on the stored reference pulse rate.However, because the state assessment unit 22, as explained above, calculates the reference pulse rate of the test subject based on the pulse wave assessment result P(t) output by the pulse wave assessment device 1, an assessment of the test subject's level of alertness can be made taking into account individual differences.

[0091] The state assessment unit 22 can also assess the test subject's state of alertness by combining a procedure for assessing the test subject's level of alertness based on the pulse wave assessment result P(t) with a known alertness assessment tool. A conceivable known alertness assessment tool is a procedure in which the level of alertness is assessed by detecting an increasing eye-closed time fraction.

[0092] The state assessment unit 22 outputs state information Z(t) regarding the assessed state of the test subject to the output unit 23.

[0093] The output unit 23 outputs the state information Z(t) issued by the state assessment unit 22 to the output device 3.

[0094] The functions of the output unit 23 can also be encompassed by the state assessment unit 22.

[0095] As described above, in the first embodiment, the state information Z(t) regarding the state of the test subject, as assessed by the state assessment unit 22, is output to the output device 3; this is merely an example. The state assessment unit 22 can, for example, also store the state information Z(t). In this case, it is not absolutely necessary for the state assessment device 2 to be connected to the output device 3. Furthermore, a setup is also possible in which the state assessment device 2 does not include the output unit 23.

[0096] The operation of the pulse wave assessment device 1 according to the first embodiment is explained.

[0097] Fig. Figure 5 is a flowchart to explain the operation of the pulse wave assessment device 1 according to the first embodiment.

[0098] The image acquisition unit 11 acquires an image on which the test subject is depicted (step ST1).

[0099] The image acquisition unit 11 outputs the acquired image to the skin area detection unit 12.

[0100] The skin area detection unit 12 detects a skin area of ​​the test subject (step ST2) based on the frame Im(k) included in the image acquired in step ST1 by the image acquisition unit 11.

[0101] The skin area detection unit 12 generates skin area information S(k) that represents the detected skin area.

[0102] The skin area detection unit 12 outputs the generated skin area information S(k) to the measuring range setting unit 13.

[0103] Based on the frame Im(k) of the image acquired in step ST1 by the image acquisition unit 11 and the skin area information S(k) output in step ST2 by the skin area detection unit 12, the measurement range setting unit 13 sets a plurality of measurement ranges for extracting a pulse wave origin signal indicating luminance changes in the image area of ​​frame Im(k) corresponding to the skin area represented by the skin area information S(k) (step ST3).

[0104] If the majority of measuring ranges have been set, the measuring range setting unit generates 13 measuring range information R(k), which represents the majority of set measuring ranges.

[0105] The measuring range setting unit 13 outputs the generated measuring range information R(k) to the pulse wave origin signal extraction unit 14 and the simulation signal generation unit 15.

[0106] The pulse wave origin signal extraction unit 14 extracts, based on the frame Im(k) of the image acquired in step ST1 by the image acquisition unit 11 and the measurement range information R(k) output in step ST3 by the measurement range setting unit 13, a pulse wave origin signal (step ST4) that shows luminance changes in a specified period, in other words, in a period corresponding to the frame rate Tp, of the individual or the plurality of measurement ranges ri(k) in the frame Im(k) represented by the measurement range information R(k).

[0107] Once the pulse wave origin signal has been extracted, the pulse wave origin signal extraction unit generates 14 pulse wave origin signal information W(t), which represents the extracted origin signal.

[0108] The pulse wave origin signal extraction unit 14 outputs the generated pulse wave origin signal information W(t) to the pulse wave assessment unit 16.

[0109] The simulation signal generation unit 15 generates a simulation signal based on the frame Im(k) of the image acquired in step ST1 by the image acquisition unit 11, the measurement range information R(k) output in step ST4 by the measurement range setting unit 13 and an ambient light model. This simulation signal simulates luminance changes of the measurement range that occurred under the ambient light in a specified period of time, in other words, in a period corresponding to the frame rate Tp, due to a change in the position of the measurement range (step ST5).

[0110] The simulation signal generation unit then generates 15 simulation signal information M(t) that is adapted to the simulation signal ri(k) generated for the respective measurement ranges.

[0111] The simulation signal generation unit 15 outputs the generated simulation signal information M(t) to the pulse wave assessment unit 16.

[0112] Based on the pulse wave origin signal information W(t) output by the pulse wave origin signal extraction unit 14 in step ST4, and the simulation signal information M(t) output by the simulation signal generation unit 15 in step ST5, the pulse wave estimation unit 16 estimates the pulse wave of the test subject (step ST6).

[0113] The pulse wave assessment unit 16 outputs the pulse wave assessment result P(t), as pulse wave information representing the assessed pulse wave, to the output unit 17.

[0114] The output unit 17 outputs the pulse wave assessment result P(t) issued by the pulse wave assessment unit 16 to the pulse wave information acquisition unit 21 of the state assessment device 2.

[0115] In the flowchart in Fig. Figure 5 shows the processing sequence of steps ST4 and ST5, but this is merely an example. The processing sequence of steps ST4 and ST5 can also be reversed, and steps ST4 and ST5 can also be processed in parallel.

[0116] The operation of the condition assessment device 2 according to the first embodiment is explained.

[0117] Fig. Figure 6 is a flowchart to explain the operation of the condition assessment device 2 according to the first embodiment.

[0118] The pulse wave information acquisition unit 21 acquires the pulse wave assessment result P(t) output by the pulse wave assessment device 1. (Step ST11).

[0119] The pulse wave information acquisition unit 21 outputs the acquired pulse wave assessment result P(t) to the state assessment unit 22.

[0120] The condition assessment unit 22 assesses the condition of the test subject based on the pulse wave assessment result P(t) output by the pulse wave information acquisition unit 21 in step ST11 (step ST12).

[0121] The state assessment unit 22 outputs state information Z(t) regarding the assessed state of the test subject to the output unit 23.

[0122] The output unit 23 outputs the state information Z(t) issued by the state assessment unit 22 to the output device 3.

[0123] As described above, a method is known in which the pulse wave of a person is estimated without contact based on subtle luminance changes in a skin area of ​​a person in an image acquired by an imaging device. For example, a method is known in which, as in Reference Document 1 below, a plurality of measurement areas are set in a portrait of a test subject, a frequency-power spectrum of luminance signals acquired in the respective measurement areas is calculated, pulse waves corresponding to the peak frequencies of the frequency-power spectrum are synthesized, and the pulse rate is estimated based on the peak of the frequency-power spectrum of the synthesized pulse waves. [Reference document 1]

[0124] Mayank Kumar, et al., “Distance PPG: Robust non-contact vital signs monitoring using a camera,” Biomedical optics express, 6 (5), 1565-1588, 2015

[0125] However, the method disclosed in Reference Document 1 has the problem that the precision in assessing the pulse waves decreases when the test subject's face moves. This is because, when the test subject's face moves, a component corresponding to the movement appears as a peak in the frequency-power spectrum, so that instead of a frequency component corresponding to the pulse wave, the component corresponding to the movement of the face is falsely detected as a pulse wave.

[0126] As a measure against this problem, a technology is known, as described above, by which specific frequency components obtained by a frequency analysis of position information of a face area in an imaging area are suppressed as noise from image signals of the face area.

[0127] However, conventional technology fails to account for the fact that in a scene where the ambient light distribution in the imaging area is uneven, the pulse wave, which is estimated based on subtle luminance changes of a test subject's skin surface in the image, is influenced not only by changes in the position of the skin area but also by ambient light. As a result, conventional technology does not achieve sufficient noise reduction, leading to low precision in estimating the test subject's pulse wave.

[0128] Fig. Figure 7 illustrates the effects of ambient light on a signal used to assess a test subject's pulse wave activity based on an image when the ambient light is unevenly distributed. Figure 7A shows an example of an image created in a scene with uniform ambient light distribution. Fig. Figure 7B is a view showing an example of an image taken in a scene with an uneven distribution of ambient light.

[0129] In Fig. 7A and Fig. 7B shows the illustration images as ImA and ImB respectively. The in Fig. 7A and Fig. The color depth shown in Figure 7B represents a luminance distribution corresponding to the ambient light. This means that the luminance is higher the brighter (whiter) the color, and lower the luminance is the darker (blacker) the color.

[0130] The image shows a test subject (SA in Fig. 7A and SB in Fig. 7B) shown.

[0131] For example, this involves the test subject moving their face so that the test subject's face appears in a temporal sequence within the frame (frame of the Tp component) of a plurality of images. Fig. 7 moved from top left to right. For simplicity, in Fig. 7 only one image of a frame is shown, and the movement of the position of the test subject's face in the frame of a plurality of images in temporal sequence is expressed by an arrow.

[0132] As in Fig. As shown in Figure 7A, in a scene with uniform ambient light, neither a signal representing the position of the face nor a signal representing the luminance of the face's surface is affected by the ambient light, even if the test subject moves their face; in other words, regardless of the test subject's face position in a multiple of frames of the images in temporal sequence. On the other hand, as shown in Fig. Figure 7B shows that, in a scenario with uneven ambient light, a signal representing the position of the face is not affected by the ambient light, even when the test subject moves their face. However, a signal representing the luminance of the face's surface is affected by the ambient light as the test subject moves their face—in other words, as the position of the test subject's face changes in multiple frames of the image in successive time. The effect on the luminance of the test subject's face surface increases the closer the test subject's face is to the center of the image. The conventional technology described above does not provide an equivalent result in a scenario using... Fig. The scene described in 7A is possible, while a corresponding example is possible using a device. Fig. The scene described in 7B is not possible.

[0133] In contrast, the pulse wave assessment unit 1 according to the first embodiment generates a simulation signal mi(t) based on the frame Im(k) of the image, the measurement range information R(k), and the ambient light model. This simulation signal mi(t) simulates luminance changes of the measurement range that occur under ambient light within a predetermined period—in other words, within a period corresponding to the frame rate Tp—due to a change in the position of the measurement range. The pulse wave assessment unit 1 then uses the generated simulation signal mi(t) for noise suppression.

[0134] In pulse wave assessment device 1, the simulation signal mi(t) is generated taking into account that, in a scene with an uneven distribution of ambient light in the image area, the pulse wave assessed based on subtle luminance changes of the test subject's skin in the image is influenced by the ambient light. This ensures that even in a scene with an uneven distribution of ambient light, noise caused by changes in the position of the test subject's skin is suppressed, and the test subject's pulse wave can be assessed. Then, according to the first embodiment, the state assessment device 2 can assess the test subject's state, even in a scene with an uneven distribution of ambient light in the image area.

[0135] In the preceding first embodiment, a single ambient light model was used, but multiple ambient light models are also possible. The simulation signal generation unit 15 can generate a simulation signal mi(t) and simulation signal information M(t) adapted to the respective simulation signal mi(t) using a plurality of ambient light models.

[0136] For example, the simulation signal generation unit 15 can, based on information obtained from a sensor etc. with which the vehicle is equipped, determine the imaging state and switch from the majority of ambient light models to the ambient light model used, generate a simulation signal mi(t) and simulation signal information M(t) adapted to the simulation signal mi(t).

[0137] To illustrate this with a concrete example, the simulation signal generation unit 15 generates, for example, the simulation signal generation unit 15 using three environmental light models: an environmental light model for the case of environmental light due to illumination light emitted by the illumination unit encompassed by the imaging device (first environmental light model), an environmental light model for the case of environmental light due to light penetrating the imaging area in such a way that the luminance on the left side of the image increases (second environmental light model), and an environmental light model for the case of environmental light due to light penetrating the imaging area in such a way that the luminance on the right side of the image increases (third environmental light model). The simulation signal mi(t) and simulation signal information M(t) adapted to the respective simulation signal mi(t) are used.Light penetrating the imaging area in such a way that the luminance increases on the left or right side of the image is assumed to be light penetrating from outside the vehicle, e.g., sunlight. The simulation signal generation unit 15 is equipped with the first, second, and third environmental light models, which were generated in advance.

[0138] Fig. Figure 8 is a view illustrating an example of a method for generating a simulation signal mi(t) by the simulation signal generation unit 15 based on a plurality of ambient light models in the first embodiment.

[0139] In Fig. 8 ImC represents an image produced by an imaging device in a vehicle during a nighttime journey; ImD represents an image produced by the imaging device in the vehicle during a daytime journey in an environment where light enters the imaging area in such a way that the luminance on the left side of the image is increased; and ImE represents an image produced by the imaging device in the vehicle during a daytime journey in an environment where light enters the imaging area in such a way that the luminance on the right side of the image is increased. Furthermore, in Fig. Figure 81a represents a diagram illustrating the intensity distribution of light, as expressed by the ambient light model, based on the ambient light in the state depicted by the image shown in ImC; Figure 81b represents a diagram illustrating the intensity distribution of light, as expressed by the ambient light model, based on the state depicted by the image shown in ImD; and Figure 81c represents a diagram illustrating the intensity distribution of light, as expressed by the ambient light model, based on the state depicted by the image shown in ImE. In Figures 81a, 81b, and 81c, the intensity of the ambient light is represented by the depth of the color, with a higher intensity being expressed as the lighter the color. For easier understanding, in Fig. 8 the imaging unit (401 in Fig. 8) and the lighting units (402 in Fig. 8), which are encompassed by the imaging device, is illustrated. Furthermore, in Fig. 8 Dr shows the face of the test subject, here the driver.

[0140] For example, in the driving scene at night, the ambient light is exclusively illumination light emitted by the lighting units comprising the imaging device. Thus, the simulation signal generation unit 15 generates a simulation signal mi(t) and simulation signal information M(t) using the first ambient light model, which is described in 81a of Fig. Figure 8 expresses the light intensity distribution shown. The simulation signal generation unit 15 can, for example, determine that it is night based on sensor information from a light sensor that detects the brightness of the vehicle's surroundings. For example, in the daytime driving scene, where light from outside the vehicle enters such that the luminance on the left side of the image increases, the ambient light is light entering from outside the vehicle. The simulation signal generation unit 15 then generates a simulation signal mi(t) and simulation signal information M(t) using the second ambient light model, which is described in Figure 81b. Fig. Figure 8 expresses the light intensity distribution shown. The simulation signal generation unit 15 can, for example, determine, based on sensor information from the lighting sensor, that this is a condition in which light from outside the vehicle penetrates in such a way that the luminance on the left side of the illustration becomes stronger.

[0141] For example, in the daytime driving scene, where light from outside the vehicle enters in such a way that the luminance on the right side of the image becomes stronger, the ambient light is light entering from outside the vehicle. The simulation signal generation unit 15 therefore generates a simulation signal mi(t) and simulation signal information M(t) using the third ambient light model, which is described in 81c of Fig. Figure 8 expresses the light intensity distribution shown. The simulation signal generation unit 15 can, for example, determine, based on sensor information from the lighting sensor, that this is a condition in which light from outside the vehicle penetrates in such a way that the luminance on the right side of the illustration becomes stronger.

[0142] In the example described above, the simulation signal generation unit 15 switches between a plurality of ambient light models in order to generate a simulation signal mi(t) and simulation signal information M(t) adapted to the simulation signal mi(t), although this is only an example.

[0143] For example, the simulation signal generation unit 15 can also use a plurality of environmental light models, generate a simulation signal mi(t) for each environmental light model used, and multiply the generated simulation signal mi(t) by a coefficient corresponding to the state (hereinafter referred to as the "model coefficient"). The simulation signal generation unit 15 then generates simulation signal information M(t) adapted to the simulation signal mi(t) after multiplication by the model coefficient.

[0144] As a concrete example, in the simulation signal generation unit 15, for instance, during a night drive, "1" is preset as the model coefficient corresponding to the first ambient light model, and "0" as the model coefficient corresponding to the second and third ambient light models. In this way, the model coefficient in the simulation signal generation unit 15 is set such that the model coefficient corresponding to an ambient light model appropriate to the current state becomes larger.

[0145] The simulation signal generation unit 15 generates a simulation signal mi(t) (hereinafter referred to as the "first simulation signal") using the first ambient light model, a simulation signal mi(t) (hereinafter referred to as the "second simulation signal") using the second ambient light model, and a simulation signal mi(t) (hereinafter referred to as the "third simulation signal") using the third ambient light model. The simulation signal generation unit 15 then multiplies the first simulation signal by the model coefficient "1" and the second and third simulation signals by the model coefficient "0". Finally, the simulation signal generation unit 15 generates simulation signal information M(t) adapted to the first, second, and third simulation signals after multiplication by the model coefficient.

[0146] By means of a setup in which the simulation signal generation unit 15 generates a simulation signal mi(t) and simulation signal information M(t) using a plurality of ambient light models, the pulse wave assessment device 1 can therefore assess the pulse wave of the test subject according to different scenes and with high precision.

[0147] Furthermore, in the aforementioned first embodiment, the pulse wave assessment device 1 calculates a pulse wave signal di(t) by unconditionally subtracting the simulation signal mi(t) from the pulse wave origin signal wi(t) using the pulse wave assessment unit 16. Luminance changes in this di(t) resulting from movement of the test subject's skin, specifically the test subject's face, are suppressed in this di(t), although this is merely an example. The pulse wave assessment unit 16 can also calculate the pulse wave signal di(t) by subtracting the simulation signal mi(t) from the pulse wave origin signal wi(t) when a certain condition is met, and determine the pulse wave origin signal wi(t) as the pulse wave signal di(t) when this condition is not met, without subtracting the simulation signal mi(t) from the pulse wave origin signal wi(t).Specifically, the pulse wave assessment unit 16 can, if a maximum value of the amplitude of the simulation signal mi(t) is less than a preset threshold (hereinafter referred to as the "amplitude detection threshold"), consider the noise due to movement of the test subject to be sufficiently small and, without performing the preceding subtraction, determine the pulse wave origin signal wi(t) as the pulse wave signal di(t), and if the maximum value of the amplitude of the simulation signal mi(t) is greater than or equal to the amplitude detection threshold, perform the subtraction and calculate the pulse wave signal di(t).

[0148] The movements of the test subject are not limited to left or right movements, but can also be forward or backward movements. In the first embodiment, the simulation signal generation unit 15 can therefore also generate the simulation signal mi(t) and simulation signal information M(t) adapted to the respective simulation signal mi(t), taking this into account. More precisely, the simulation signal generation unit 15 can also generate the simulation signal mi(t) based on a signal in which, for example, the magnitude (amplitude) of luminance changes in the measuring range ri(k) that occurred due to a left or right movement of the test subject and the magnitude of luminance changes in the measuring range ri(k) that occurred due to a forward or backward movement of the test subject have been normalized.

[0149] The simulation signal generation unit 15 determines the simulation signal mi(t) generated by the method described in the preceding first embodiment as the simulation signal mi(t) simulating luminance changes of the measuring range ri(k) that have occurred due to a movement of the test subject to the left or right (hereinafter referred to as the “first simulation signal mi1(t)”).

[0150] Apart from the first simulation signal mi1(t), the simulation signal generation unit 15 generates a simulation signal mi(t) that simulates luminance changes of the measuring range ri(k) caused by a movement of the test subject forwards or backwards (hereinafter referred to as the "second simulation signal mi2(t)").

[0151] The specific procedure by which the simulation signal generation unit 15 generates the second simulation signal mi2(t) is explained.

[0152] The simulation signal generation unit 15 first calculates a boundary rectangle Rec(k) for all measurement ranges ri(k) and determines their central coordinate O(k) (x, y) = (xc, yc). The view on the left side in Fig. Figure 9 is a view showing an example of the boundary rectangle Rec(k) of the measurement ranges ri(k) and their central coordinate O(k).

[0153] The simulation signal generation unit 15 then calculates a distance Dis(k) between the respective corner coordinates of the respective measurement ranges ri(k) and the central coordinate O(k). For example, if the corner coordinates of a corner of a measurement range ri(k) are given as (x, y) = (xi_rb, yi_rb) and the corresponding distance between these and the central coordinate O(k) as Dis_i_rb, the simulation signal generation unit 15 calculates Dis_i_rb using the following formula (2). The view on the right in Fig. Figure 9 is a pictorial representation of the distance between the corner coordinates of a corner of a measurement area ri(k) and the corresponding central coordinate O(k). Dis_i_rb=(xi_rb−xc)2+(yi_rb−yc)2

[0154] Then the simulation signal generation unit 15 determines the distance Dis(k) determined above between the corner coordinates of the respective measurement areas ri(k) and the central coordinate O(k) of the boundary rectangle as the second simulation signal mi2(t).

[0155] In the example described above, the simulation signal generation unit 15 calculated the distance between the corner coordinates of the respective measurement areas ri(k) and the central coordinate O(k) of the bounding rectangle, and there is no limitation to this. For example, the simulation signal generation unit 15 can also calculate the distance between the centroid coordinates of the respective measurement areas ri(k) and the central coordinate O(k) of the bounding rectangle Rec(k).

[0156] Furthermore, in the example described above, the simulation signal generation unit 15 can also use an average value of the corner coordinates of the respective measurement ranges ri(k), or in other words, a centroid, instead of the central coordinate O(k) of the bounding rectangle Rec(k), and calculate the distance Dis(k) between this centroid and the corner coordinates of the respective measurement ranges ri(k). It is sufficient if the simulation signal generation unit 15 defines reference coordinates for the respective measurement ranges ri(k).

[0157] Next, in the simulation signal generation unit 15, the first simulation signal mi1(t) and the second simulation signal mi2(t) are normalized by multiplying them by a coefficient (hereinafter referred to as the "normalization coefficient"). In other words, in the simulation signal generation unit 15, the difference in magnitude between the first simulation signal mi1(t) and the second simulation signal mi2(t) is adjusted by multiplying them by the normalization coefficient.

[0158] For example, in the simulation signal generation unit 15, either the first simulation signal mi1(t) or the second simulation signal mi2(t) is multiplied by the normalization coefficient. Alternatively, in the simulation signal generation unit 15, both the first simulation signal mi1(t) and the second simulation signal mi2(t) can be multiplied by the normalization coefficient. The normalization coefficient can be a preset value, or the simulation signal generation unit 15 can calculate a normalization coefficient that adjusts to an average value of the order of magnitude of the first simulation signal mi1(t) and the second simulation signal mi2(t).

[0159] Fig. Figure 10 is a view which, in the first embodiment, shows an example of the first simulation signal mi1(t) and the second simulation signal mi2(t), where the simulation signal generation unit 15 multiplies by the normalization coefficient and thereby adjusts the difference to the order of magnitude.

[0160] Then, the simulation signal generation unit 15 determines the first simulation signal mi1(t) and the second simulation signal mi2(t) together as the simulation signal mi(t) after multiplication with the normalization coefficient.

[0161] In this way, because the simulation signal generation unit 15 generates the simulation signal mi(t) based on a signal in which the magnitude of a signal, the luminance changes of the measuring range ri(k) that occurred due to a movement of the test subject to the left or right (first simulation signal mi1 (t)), and the magnitude of a signal, the luminance changes of the measuring range ri(k) that occurred due to a movement of the test subject forward or backward (second simulation signal mi2(t)), have been normalized, noise due to position changes of the skin area of ​​the test subject can be suppressed and the pulse wave of the test subject can be assessed by the pulse wave assessment device 1, even taking into account a movement of the test subject forward and backward.

[0162] Furthermore, in the aforementioned first embodiment of the pulse wave assessment device 1, a plurality of measurement ranges were set by the measurement range setting unit 13 in the image area of ​​frame Im(k), which corresponds to the skin area represented by the skin area information S(k), although this is merely an example. The measurement range setting unit 13 can also, for example, set only one measurement range.

[0163] Furthermore, when generating a simulation signal from a plurality of measurement ranges set by the measurement range setting unit 13 in the pulse wave estimation device 1, the simulation signal generation unit 15 can select a measurement range and generate a simulation signal only for the selected measurement range.

[0164] If only one measuring range is set by the measuring range setting unit 13, or if a measuring range is selected by the simulation signal generation unit 15, then the preferred measuring range is a skin area with few movements, such as mouth movements due to speaking or movements due to changes in expression. "Movements of a skin area" here do not refer to movements resulting from a change in the position of the head itself, but rather to "movements of a skin area" caused by factors other than a change in the position of the head itself.

[0165] If only one measuring range is set by the measuring range setting unit 13, or if a measuring range is set by the simulation signal generation unit 15, it is, for example, preferred if the set or selected measuring range is a skin area corresponding to a cheek or forehead. However, if the skin area corresponding to the forehead is covered by forehead hair, etc., a skin area corresponding to the cheek is preferred instead.

[0166] In the first embodiment described above, the pulse wave assessment device 1 is encompassed by the state assessment device 2; this is merely an example. The pulse wave assessment device 1 can also be provided outside the state assessment device 2 and connected to the state assessment device 2 outside the state assessment device 2.

[0167] In the first embodiment described above, the illumination unit is encompassed by the imaging device; this is merely an example. For instance, the illumination unit can also be located outside the imaging device and emit light into the imaging area from outside the imaging device.

[0168] Furthermore, in the aforementioned first embodiment, the test subject is the driver of a vehicle, although this is merely an example. The test subject could also be a passenger other than the driver of a vehicle.

[0169] In the foregoing first embodiment, the pulse wave assessment device 1 and the condition assessment device 2 are onboard devices, and the image acquisition unit 11, the skin area detection unit 12, the measurement range setting unit 13, the pulse wave origin signal extraction unit 14, the simulation signal generation unit 15, the pulse wave assessment unit 16, the output unit 17, the pulse wave information acquisition unit 21, the condition assessment unit 22 and the output unit 23 are included by the onboard devices.

[0170] Without limitation hereto, a system can also be formed by an onboard device and a server in which the onboard device of the vehicle is equipped with part of the image acquisition unit 11, the skin area detection unit 12, the measurement range setting unit 13, the pulse wave origin signal extraction unit 14, the simulation signal generation unit 15, the pulse wave assessment unit 16, the output unit 17, the pulse wave information acquisition unit 21, the condition assessment unit 22 and the output unit 23, and the remainder are comprised of the server, which is connected to the onboard device in question via a network.

[0171] Furthermore, the image acquisition unit 11, the skin area detection unit 12, the measurement range setting unit 13, the pulse wave origin signal extraction unit 14, the simulation signal generation unit 15, the pulse wave assessment unit 16, the output unit 17, the pulse wave information acquisition unit 21, the state assessment unit 22 and the output unit 23 can all be included by one server.

[0172] The pulse wave assessment device 1 and the condition assessment device 2 according to the preceding first embodiment are not limited to onboard devices with which a vehicle is equipped, but can also be applied, for example, to electrical household appliances. Furthermore, the test subject is not limited to a single occupant of a vehicle, but can be several different people.

[0173] As a concrete example, a television set up in the living room of an apartment could be equipped with pulse wave assessment device 1 and state assessment device 2. In this case, the test subject is a user, such as the resident of the apartment. Pulse wave assessment device 1 estimates the user's pulse wave based on an image projected by an imaging device built into the television. The imaging device should be able to image at least a portion of the test subject's skin presence. State assessment device 2, based on a pulse wave assessment result P(t) derived from the user's pulse wave as assessed by pulse wave assessment device 1, uses a rise or fall in the pulse rate to assess and record the user's physical condition.

[0174] Fig. 11A and Fig. Figure 11B shows examples of the hardware setup of the pulse wave assessment device 1 of the first embodiment.

[0175] In the first embodiment, the functions of the image acquisition unit 11, the skin area detection unit 12, the measurement range setting unit 13, the pulse wave origin signal extraction unit 14, the simulation signal generation unit 15, the pulse wave assessment unit 16, and the output unit 17 are implemented by a processing circuit 101. That is, the pulse wave assessment device 1 includes the processing circuit 101 to control the assessment of the test subject's pulse wave using an ambient light model.

[0176] The processing circuit 101 can use special hardware such as in Fig. 11A may be shown, but it can also be like in Fig. Figure 11B shows a processor 104 that executes programs stored in memory.

[0177] If the processing circuit 101 is a special piece of hardware, the processing circuit 101 corresponds, for example, to a simple circuit, a complex circuit, a programmed processor, parallel programmed processors, ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array) or a combination thereof.

[0178] If the processing circuit is the processor 104, the functions of the image acquisition unit 11, the skin area detection unit 12, the measurement range setting unit 13, the pulse wave origin signal extraction unit 14, the simulation signal generation unit 15, the pulse wave assessment unit 16, and the output unit 17 are implemented by software, firmware, or a combination of software and firmware. The software or firmware is described as a program and stored in memory 105. The processor 104 reads and executes programs stored in memory 105, thereby performing the functions of the imaging unit 11, the skin area detection unit 12, the measurement range setting unit 13, the pulse wave origin signal extraction unit 14, the simulation signal generation unit 15, the pulse wave assessment unit 16 and the output unit 17.That is, when executed by the processor 104, the pulse wave assessment device 1 includes the memory 105 for storing programs, which are ultimately executed in the steps ST1 to ST6 described above. Fig. 5. It can further be said that the programs stored in memory 105 serve to execute the sequence or procedure of the processing operations of the image acquisition unit 11, the skin area detection unit 12, the measurement range setting unit 13, the pulse wave origin signal extraction unit 14, the simulation signal generation unit 15, the pulse wave assessment unit 16, and the output unit 17 on a computer. Memory 105 can be a non-volatile or volatile semiconductor memory such as RAM, ROM (Read Only Memory), Flash memory, EPROM (Erasable Programmable Read Only Memory), or EEPROM (Electrically Erasable Programmable Read Only Memory), a magnetic disk, a floppy disk, an optical disk, a compact disc, a minidisc, or a DVD (Digital Versatile Disc).

[0179] The functions of the image acquisition unit 11, the skin area detection unit 12, the measurement range setting unit 13, the pulse wave origin signal extraction unit 14, the simulation signal generation unit 15, the pulse wave assessment unit 16, and the output unit 17 can also be implemented partly by special hardware and partly by software or firmware. For example, the processing circuit 101, as special hardware, can implement the functions of the image acquisition unit 11 and the output unit 17, and the functions of the skin area detection unit 12, the measurement range setting unit 13, the pulse wave origin signal extraction unit 14, the simulation signal generation unit 15, and the pulse wave assessment unit 16 can be implemented by the processor 104 reading and executing programs stored in memory 105.

[0180] Furthermore, the pulse wave assessment device 1 comprises an input interface device 102 and an output interface device 103 for carrying out wired or wireless communication with a device such as an imaging device, etc.

[0181] The Fig. 11A and Fig. The structures shown in Figure 11B are examples of the hardware setup of the condition assessment device 2 according to the first embodiment.

[0182] In the first embodiment, the functions of the pulse wave information acquisition unit 21, the state assessment unit 22, and the output unit 23 are implemented by means of the processing circuit 101. That is, the state assessment device 2 includes the processing circuit 101 in order to control the assessment of the test subject's state based on the pulse wave information regarding the test subject's pulse wave, which was assessed by the pulse wave assessment device 1.

[0183] If the processing circuit is processor 104, the functions of the pulse wave information acquisition unit 21, the state assessment unit 22, and the output unit 23 are implemented by software, firmware, or a combination of software and firmware. The software or firmware is described as a program and stored in memory 105. Processor 104 reads and executes programs stored in memory 105, thereby performing the functions of pulse wave information acquisition unit 21, state assessment unit 22, and output unit 23. That is, when executed by processor 104, the state assessment device 2 includes memory 105 for storing programs, which are ultimately executed in steps ST11 to ST12 described above. Fig.6. It can further be said that the programs stored in memory 105 are the sequence or a procedure for executing the pulse wave information acquisition unit 21, the state assessment unit 22 and the output unit 23 on a computer.

[0184] The functions of the pulse wave information acquisition unit 21, the state assessment unit 22, and the output unit 23 can also be implemented partly by special hardware and partly by software or firmware. For example, the processing circuit 101, as special hardware, can implement the functions of the pulse wave acquisition unit 21 and the output unit 23, and the functions of the state assessment unit 22 can be implemented by the processor 104 reading and executing a program stored in memory 105.

[0185] Furthermore, the condition assessment device 2 comprises the input interface device 102 and the output interface device 103 for carrying out wired or wireless communication with a device such as the pulse wave assessment device 1 or the output device 3.

[0186] As stated above, the assembly of the pulse wave assessment device 1 according to the first embodiment comprises the image acquisition unit 11, which acquires an image depicting a person (test subject); the skin area detection unit 12, which detects a skin area of ​​the person based on the image; the measurement area setting unit 13, which sets a measurement area ri(k) in a region corresponding to the skin area in the image for extracting a pulse wave signal that shows a luminance change; the pulse wave origin signal extraction unit 14, which extracts a pulse wave origin signal wi(t) based on a luminance change in the measurement area ri(k) of the image; and the simulation signal generation unit 15, which generates a simulation signal mi(t) based on position changes of the measurement area ri(k) and a model of the distribution of ambient light in the image area.The device simulates the luminance changes of the measuring range ri(k) caused by changes in the measuring range's position under the ambient light, and the pulse wave estimation unit 16, based on the pulse wave origin signal wi(t) and the simulation signal mi(t), estimates the person's pulse wave. Therefore, the pulse wave estimation device 1 can estimate the test subject's pulse wave even in scenes with uneven ambient light distribution in the imaging area.

[0187] Furthermore, the state assessment device 2 according to the preceding first embodiment is designed to assess the state of the test subject based on the pulse rate of the person (test subject) as assessed by the pulse wave assessment device 1 with the aforementioned design. Therefore, the state assessment device 2 can assess the state of the test subject based on a pulse wave assessed with high precision, even in scenes with uneven ambient light distribution in the imaging area. Therefore, the state assessment device 2 can assess the state of the test subject with high precision.

[0188] Any modification of the components of the embodiments or the omission of any components of the embodiments is possible.

[0189] The following section summarizes various aspects of the present revelation as additions. (Addendum 1)

[0190] Pulse wave assessment device, comprising a picture acquisition unit that acquires a picture depicting a person, a skin area detection unit that detects a skin area of ​​the person based on the image, a measurement range setting unit that sets a measurement range in an area corresponding to the skin area in the image for extracting a pulse wave signal that shows a change in luminance, a pulse wave origin signal extraction unit that extracts a pulse wave origin signal based on a luminance change in the measurement area of ​​the image, a simulation signal generation unit that, based on position changes of the measuring range and a model of the distribution of ambient light in the imaging area, generates a simulation signal that simulates the luminance changes of the measuring range caused by the position changes of the measuring range under the ambient light, and a pulse wave estimation unit that estimates the person's pulse wave based on the pulse wave origin signal and the simulation signal. (Addendum 2)

[0191] Pulse wave estimation device according to addition 1, characterized in that the pulse wave estimation unit estimates the pulse wave based on a signal in which the simulation signal in the pulse wave origin signal has been suppressed. (Addendum 3)

[0192] Pulse wave estimation device according to addition 1 or addition 2, characterized in that the model of the distribution of ambient light is a model which, based on the spatial luminance distribution in an area into which the ambient light is emitted, outputs values ​​that simulate luminance values ​​of coordinates in the image. [Addendum 4]

[0193] Pulse wave assessment device according to one of the additions 1 to 3, characterized in that the ambient light is an irradiation light emitted by a lighting unit encompassed by the imaging device through which the image was created. [Addendum 5]

[0194] Pulse wave assessment device according to one of the additions 1 to 4, characterized in that the simulation signal generation unit generates the simulation signal based on a plurality of different models of the distribution of ambient light, which correspond to situations in which the ambient light is emitted and position changes of the measuring range. [Addendum 6]

[0195] Pulse wave assessment device according to one of the additions 1 to 4, characterized in that the simulation signal generation unit generates the simulation signal based on a signal in which the magnitude of a signal representing luminance changes of the measuring range that have occurred due to a movement of the person to the left or right, and the magnitude of a signal representing luminance changes of the measuring range that have occurred due to a movement of the person forward or backward, have been normalized. [Addendum 7]

[0196] Condition assessment device comprising a condition assessment unit which assesses the condition of the person based on the pulse wave of the person assessed by a pulse wave assessment device of one of the additions 1 to 6. [Addendum 8]

[0197] Condition assessment device according to addition 7, characterized in that the condition assessment unit assesses the alertness of the person as the condition of the person. [Addendum 9]

[0198] State assessment device according to addition 8, characterized in that the state assessment unit calculates a reference pulse rate based on the pulse wave of the person, which was assessed by the pulse wave assessment device in a state in which the person is presumed to be alert, and assesses the level of alertness of the person by comparing the pulse wave assessed by the pulse wave assessment device and the reference pulse rate. [Addendum 10]

[0199] Pulse wave assessment method, comprehensive a step in which an image acquisition unit acquires an image depicting a person, a step in which a skin area detection unit uses the image to detect a skin area of ​​the person, a step in which a measurement range setting unit sets a measurement range in an area corresponding to the skin area in the image for extracting a pulse wave origin signal that shows a luminance change, a step in which a pulse wave origin signal extraction unit extracts a pulse wave origin signal based on a luminance change in the measurement range of the image, a step in which a simulation signal generation unit, based on position changes of the measuring range and a model of the distribution of ambient light in the imaging area, generates a simulation signal that simulates the luminance changes of the measuring range caused by the position changes of the measuring range under the ambient light, and a step in which a pulse wave estimation unit estimates the person's pulse wave based on the pulse wave origin signal and the simulation signal. Industrial application area

[0200] In the pulse wave estimation device according to the present disclosure, taking into account that in a scene with an uneven distribution of ambient light in the image area, the pulse wave, which is estimated on the basis of fine luminance changes of the skin area of ​​the test subject in the image, is influenced by the ambient light, the pulse wave of the test subject is estimated so that, compared to when this influence is not taken into account, the pulse wave of the test subject can be estimated with high precision. Explanation of reference symbols 1 Pulse wave assessment device 11 Image acquisition unit 12 skin area detection units 13 Measuring range setting unit 14 Pulse wave origin signal extraction unit 15 Simulation signal generation unit 16 Pulse wave assessment unit 17 output units 2 Condition assessment device 21 Pulse wave information acquisition unit 22 Condition Assessment Unit 23 output units 3 Output device 401 imaging unit 402 Lighting unit 101 Processing circuit 102 Input interface device 103 Output interface device 104 processor 105 storage QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] JP 2018-68720 A

[0003] Cited non-patent literature

[0000] Mayank Kumar, et al., “Distance PPG: Robust non-contact vital signs monitoring using a camera,” Biomedical optics express, 6 (5), 1565-1588, 2015

[0124]

Claims

[1] Pulse wave estimation device comprising an image acquisition unit for acquiring an image depicting a person; a skin area detection unit for detecting a skin area of ​​the person from the image; a measurement area setting unit for setting a measurement area for extracting a pulse wave origin signal showing a luminance change in an area corresponding to the skin area in the imaging image; a pulse wave origin signal extraction unit for extracting the pulse wave origin signal based on the luminance change in the measurement range of the imaging image; a simulation signal generation unit for generating a simulation signal based on a position change of the measuring area and a distribution model of the ambient light in an imaging area, wherein the simulation signal simulates the luminance change of the measuring area caused by the position changes of the measuring area under the ambient light; and a pulse wave estimation unit for estimating a pulse wave of the person based on the pulse wave original signal and the simulation signal. [2] Pulse wave estimation device according to claim 1, characterized by that the pulse wave estimation unit estimates the pulse wave based on a signal in which the simulation signal has been suppressed in the pulse wave original signal. [3] Pulse wave estimation device according to claim 1, characterized bythat the ambient light distribution model is a model that outputs a value that simulates luminance values ​​of coordinates in the image image based on spatial luminance distribution in an area into which the ambient light is emitted. [4] Pulse wave estimation device according to claim 1, characterized by that the ambient light is an irradiation light emitted by a lighting unit included in an imaging device by which the imaging image was formed. [5] Pulse wave estimation device according to claim 1, characterized by that the simulation signal generating unit generates the simulation signal based on a plurality of different distribution models of the ambient light corresponding to situations in which the ambient light is emitted and the position change of the measuring area. [6] Pulse wave estimation device according to claim 1, characterized bythat the simulation signal generating unit generates the simulation signal based on a signal in which a magnitude of a signal representing the luminance change of the measuring area that occurred due to a movement of the person to the left or right and the magnitude of a signal representing the luminance change of the measuring area that occurred due to a movement of the person to the front or back have been normalized. [7] A condition estimation device comprising a condition estimation unit for estimating a condition of the person based on the pulse wave of the person estimated by a pulse wave estimation device according to any one of claims 1 to 6. [8] Condition estimation device according to claim 7, characterized by that the state assessment unit assesses the person's state as a level of alertness of the person. [9] Condition estimation device according to claim 8, characterized bythat the state estimation unit calculates a reference pulse rate based on the pulse wave of the person estimated by the pulse wave estimation device in a state in which the person is presumed to be alert, and estimates the alertness level of the person by a comparison between the pulse wave estimated by the pulse wave estimation device and the reference pulse rate. [10] Pulse wave assessment method, including a step of acquiring, by an image acquisition unit, an image depicting a person; a step of detecting, by a skin area detecting unit, a skin area of ​​the person from the image image; a step of setting, by a measurement area setting unit, a measurement area for extracting a pulse wave origin signal showing a luminance change in an area corresponding to the skin area in the imaging image; a step of extracting, by a pulse wave original signal extraction unit, a pulse wave original signal based on the luminance change in the measurement range of the imaging image; a step of generating, by a simulation signal generating unit, a simulation signal based on a position change of the measurement area and a distribution model of the ambient light in an imaging area, wherein the simulation signal simulates the luminance change of the measurement area caused by the position changes of the measurement area under the ambient light; and a step of estimating, by a pulse wave estimation unit, a pulse wave of the person based on the pulse wave original signal and the simulation signal.

Images