Pulse wave estimation device and pulse wave estimation method
The pulse wave estimation device uses facial organ detection and tracking to set measurement areas in consecutive images, addressing accumulated deviations and improving accuracy in pulse wave estimation.
Patent Information
- Application Number
- JP2023500249
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-02-19
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2041-02-19
AI Technical Summary
Existing biological information measuring devices experience accumulated deviations in skin area positioning due to sequential tracking processing without regular reset processes, leading to inaccuracies in pulse wave estimation.
A pulse wave estimation device that includes facial organ detection and tracking units to set distinct measurement areas in consecutive images, measuring brightness differences to estimate pulse waves accurately, thereby avoiding continuous deviations.
The device achieves higher accuracy in pulse wave estimation by preventing the accumulation of positional deviations between image frames, enhancing precision in pulse wave measurement.
Smart Images

Figure 0007584617000001 
Figure 0007584617000002 
Figure 0007584617000003
Abstract
Description
[Technical field]
[0001] The present disclosure relates to a pulse wave estimation device and a pulse wave estimation method. [Background technology]
[0002] The bioinformation measuring device described in Patent Document 1, which measures the condition of a living body, for example, blood pressure, in a non-contact manner, needs to set a skin area for measuring the blood pressure in an image of a facial organ in each of a plurality of image frames in which the living body is imaged. In order to set the skin area, the bioinformation measuring device performs, for example, a facial organ detection process of detecting the position of the facial organ in one image frame. On the other hand, for a plurality of image frames subsequent to the one image frame, instead of performing the above-mentioned facial organ detection process, the bioinformation measuring device performs a tracking process of setting the position of the skin area in the current image frame by referring to the position of the skin area in the previous image frame.
[0003] When a movement of the facial organs of the living body occurs between image frames before and after the above-mentioned tracking process, for example, when a change in facial expression occurs or the position of the face moves, a deviation occurs between the position of the skin area in the image frame before the tracking process, which is before the movement of the facial organs occurs, and the position of the skin area in the image frame after the movement of the facial organs occurs. Therefore, when the tracking process is performed sequentially on multiple consecutive image frames, the deviation accumulates. In order to eliminate the accumulated deviation, i.e., to reset, the bioinformation measuring device performs a reset process, which is the above-mentioned facial organ detection process, for every predetermined number of image frames. [Prior art documents] [Patent documents]
[0004] [Patent Document 1] JP 2016-190022 A Summary of the Invention [Problem to be solved by the invention]
[0005] However, in the above-mentioned bio-information measuring device, there was a problem in that the above-mentioned accumulated discrepancy occurred between the position of the skin area in one image frame detected by the above-mentioned facial organ detection process and the position of the skin area detected by the above-mentioned tracking process in another image frame which follows the first image frame without being separated by more than the above-mentioned predetermined number of image frames and therefore has not yet been subjected to the reset process, due to the tracking process being performed sequentially between the first image frame and the other image frame.
[0006] The above-mentioned bioinformation measuring device also had the problem that, between image frames before and after the above-mentioned reset process, an accumulated shift occurred between the position of a skin area with the accumulated shift in the image frame before the reset process was performed and the position of a skin area with no shift as a result of the reset process in the image frame after the reset process was performed.
[0007] An object of the present disclosure is to provide a pulse wave estimation device that solves at least one of the two problems described above. [Means for solving the problem]
[0008] In order to solve the above-mentioned problems, a pulse wave estimation device according to the present disclosure includes an acquisition unit that acquires a plurality of consecutive image frames including a first image frame and a second image frame that are successive to each other, of a subject whose pulse wave is to be estimated; a detection unit that detects face organ points that indicate positions of face organs of the subject in the first image frame; a tracking unit that sets tracking points that indicate positions of the face organs in the second image frame after movement of the face organ has occurred, by referring to the first image frame; and a tracking unit that sets, in the first image frame, a tracking point at which a luminance value for estimating the pulse wave should be measured, based on the detected face organ points in the first image frame. 1.A first setting unit that sets a position of a measurement region, and based on the set tracking point in the second image frame, the second image frame is set to the position of the measurement region. A second measurement area different from the first measurement area A second setting unit that sets the position of the measurement area; By the first setting part In the first image frame set 1. A measurement area, and By the second setting part In the set second image frame Second A measurement unit that measures the luminance value of a pixel in a measurement region, and 1. The luminance value of the measurement area and the measured luminance value in the second image frame are Second and an estimation unit that estimates the pulse wave of the subject based on a luminance difference that is a difference from the luminance value of the measurement area. Effect of the Invention
[0009] The pulse wave estimation device according to the present disclosure can avoid the situation where an accumulated deviation occurs as tracking processes are performed sequentially. The pulse wave estimation device according to the present disclosure can also avoid the situation where an accumulated deviation occurs between image frames before and after a reset process. Therefore, the pulse wave estimation device according to the present disclosure can estimate the subject's pulse wave with higher accuracy than a pulse wave estimation device in which at least one of the above two accumulated deviations occurs. [Brief description of the drawings]
[0010] [Figure 1] 1 is a functional block diagram of a pulse wave estimation device 1 according to an embodiment. [Diagram 2] 2 shows the operation of the imaging unit 11 and the detection unit 12 of the embodiment. [Diagram 3] 2 shows a configuration of a setting unit 13 according to the embodiment. [Figure 4] 4 shows the operation of a face organ detection unit 13A according to the embodiment. [Diagram 5] 13 shows the operation of the tracking unit 13B of the embodiment. [Figure 6] 1 shows the configuration of a pulse wave estimation device 1 according to an embodiment. [Figure 7]4 is a flowchart showing the operation of the pulse wave estimation device 1 according to the embodiment. [Figure 8] 4 is a time chart showing the operation of the pulse wave estimation device 1 according to the embodiment. [Figure 9] 4 shows the operation of a pulse wave estimation device of a comparative example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0011] An embodiment of a pulse wave estimation device according to the present disclosure will be described.
[0012] Embodiment 1. <Embodiment> Configuration of the embodiment FIG. 1 is a functional block diagram of a pulse wave estimation device 1 according to an embodiment.
[0013] The functions of the pulse wave estimation device 1 of the embodiment will be described below with reference to FIG.
[0014] The pulse wave estimation device 1 of the embodiment includes, as shown in FIG. 1, an imaging unit 11, a detection unit 12, a setting unit 13, a measurement unit 14, and an estimation unit 15 to estimate a pulse wave based on the brightness of pixels in a captured image.
[0015] The imaging unit 11 corresponds to an “acquisition unit,” the measurement unit 14 corresponds to a “measurement unit,” and the estimation unit 15 corresponds to an “estimation unit.” The correspondence relationship regarding the setting unit 13 will be described later.
[0016] As described later, the detection unit 12 for detecting the skin region does not correspond to a "detection unit" for detecting face organ points, but corresponds to a setting unit for performing face organ detection processing and tracking processing. 13 The face organ detection unit 13A in the image forming apparatus 1 corresponds to the "detection section" and also corresponds to the "first setting section."
[0017] In the following, for ease of explanation and understanding, the reference numerals may be collectively referred to by the same reference numerals without the subscripts attached thereto. For example, the image frames F1 to Fm may be collectively referred to as the image frame F, and the skin regions S1 to Sm may be collectively referred to as the skin region S.
[0018] FIG. 2 shows the operation of the imaging unit 11 and the detection unit 12 of the embodiment.
[0019] The photographing unit 11 is, for example, a camera, and photographs an image G of a subject T whose pulse wave is to be estimated, for example, an image G centered on the upper body of the subject T, particularly the face of the subject T, as shown in Fig. 2. The image G includes a plurality of consecutive image frames F1, F2, F3, ..., Fm, as shown in Fig. 2, where m is an integer of 2 or more.
[0020] For example, image frame F1 and image frame F2 precede and follow each other, and similarly, image frame F2 and image frame F3 precede and follow each other.
[0021] The photographing section 11 outputs the photographed multiple image frames F1 to Fm to the detection section 12.
[0022] 2, the detection unit 12 detects a skin region S1 from, for example, an image frame F1. The skin region S1 is a rectangular region that represents the overall position, shape, size, etc. of the face of the subject T in the image frame F1.
[0023] The detection unit 12 detects skin regions S2, S3, . . . , Sm from the other image frames F2, F3, . . . , Fm in the same manner as detecting the skin region S1 from the image frame F1.
[0024] In the following, for ease of explanation and understanding, for example, both the skin region S1 itself and the information indicating the skin region S1 will be collectively referred to as the skin region S1. The same applies to the other skin regions S2 to Sm.
[0025] The skin regions S1 to Sm may represent other parts of the subject T, such as the neck, shoulders, arms, hands, etc., instead of the face of the subject T described above. The skin regions S1 to Sm may represent parts of the face of the subject T, such as the positions of a part of the face of the subject T including one or more of the forehead, eyebrows, eyes, nose, mouth, cheeks, chin, etc., instead of the entire face of the subject T described above. The number of skin regions S1 to Sm is not limited to the entire face of the subject T and the parts of the face of the subject T described above. More specifically, the skin regions S1 to Sm may represent only one, such as the entire position of the face of the subject T, or two, such as the position of the right cheek of the subject T and the position of the left cheek of the subject T, or three, such as the position of the nose of the subject T, the position of the mouth of the subject T, and the position of the chin of the subject T.
[0026] Returning to FIG. 1, the detection unit 12 outputs a plurality of image frames F1 to Fm and a plurality of skin regions S1 to Sm to the setting unit 13.
[0027] FIG. 3 shows a configuration of the setting unit 13 according to the embodiment.
[0028] 3, the setting section 13 has a face organ detection unit 13A and a tracking unit 13B. The face organ detection unit 13A corresponds to a "detection section" and a "first setting section", and the tracking unit 13B corresponds to a "tracking section" and a "second setting section".
[0029] face organ The detection unit 13A performs face organ detection processing on the preceding image frame F2 so as to be able to calculate the luminance difference between two adjacent frames, for example, the luminance difference between the image frame F2 and the immediately following image frame F3.
[0030] The tracking unit 13B is a face organLike detection unit 13A, it is possible to calculate the luminance difference between two adjacent frames, but unlike face organ detection unit 13A, it performs tracking processing on the subsequent image frame F2 so as to be able to calculate the luminance difference between, for example, image frame F2 and the immediately preceding image frame F1.
[0031] To summarize the face organ detection process by the face organ detection unit 13A and the tracking process by the tracking unit 13B, the face organ detection process and the tracking process are performed for each image frame F2, F3, . . .
[0032] For example, face organ detection unit 13A sets one or more rectangular measurement areas KR1(1) to KR1(n) (shown in FIG. 4) in skin area S1 received from detection section 12, where luminance is to be measured to estimate the pulse wave of subject T. Here, n is an integer of 1 or more.
[0033] FIG. 4 shows the operation of the face organ detection unit 13A of the embodiment.
[0034] The face organ detection unit 13A uses, for example, a Constrained Local Model (CLM), which is a model of a conventionally known face organ detection method. The face organ detection unit 13A performs face organ detection processing using the above-mentioned CLM, and specifically, as shown in Fig. 4 (within the left frame), for example, detects the coordinate values of a plurality of face organ points KK1(1) to KK1(p) in the skin region S1. Here, the face organ points KK1(1) to KK1(p) are feature points for identifying the face of the subject T. p is an integer of 2 or more.
[0035] Following the face organ detection process, face organ detection unit 13A sets the coordinate values of measurement areas KR1(1) to KR1(n) based on the face organ points KK1(1) to KK1(p) detected in skin area S1 (shown in the left frame in FIG. 4), as shown in FIG. 4 (right frame).
[0036] The face organ detection unit 13A performs the same processing as that performed on the skin region S1 for the other skin regions S2 to Sm other than the skin region S1, i.e., detects the coordinate values of the face organ points KK and sets the coordinate values of the measurement region KR.
[0037] The facial organ detection unit 13A detects, for example, the coordinate values of a plurality of facial organ points KK2(1), KK2(2), ... (not shown) in the skin region S2, and sets the coordinate values of a plurality of measurement regions KR2(1), KR2(2), ... (not shown).
[0038] FIG. 5 shows the operation of the tracking unit 13B of the embodiment.
[0039] The tracking unit 13B performs tracking processing between previous and next image frames using a Kanade-Lucas-Tomasi (KLT) tracker, which is a conventionally known tracking technique.
[0040] The tracking unit 13B performs tracking processing based on, for example, an image frame F1, a skin region S1 in the image frame F1, and an image frame F2 subsequent to the image frame F1, as shown in Fig. 5. As a result, the tracking unit 13B sets the coordinate values of tracking points TR2(1), TR2(2), TR2(3), TR2(4), ..., which are feature points for identifying the face of the subject T, in the skin region S2 of the image frame F2, as shown in Fig. 5.
[0041] Following the tracking process, tracking As shown in FIG. 5, the unit 13B sets the coordinate values of measurement regions KR2(1), KR2(2), KR2(3), . . . based on tracking points TR2(1), TR2(2), TR2(3), TR2(4), . . .
[0042] Comparing face organ detection unit 13A and tracking unit 13B, face organ detection unit 13A uses the above-mentioned face organ points KK1(1) to KK1(p) (shown in FIG. 4) as its reference, while tracking unit 13B uses the above-mentioned tracking points TR2(1), TR2(2), TR2(3), TR2(4), ... (shown in FIG. 5) as its reference.
[0043] The tracking unit 13B performs the same process as that performed on the skin region S2 for the other skin regions S3 to Sm other than the skin region S2, that is, sets the coordinate values of the tracking point TR and the coordinate values of the measurement region KR.
[0044] As shown in FIG. 1, the setting unit 13 outputs the measurement area KR, that is, the measurement areas KR1(1), KR1(2), ... (shown in FIG. 4), KR2(1), KR2(2), ... (shown in FIG. 5), KRm(1), KRm(2), ... (not shown) for the skin areas S1 to Sm to the measurement unit 14.
[0045] Returning to FIG. 1, the description of the configuration of pulse wave estimation device 1 will continue.
[0046] The measurement unit 14 measures the luminance values of the pixels contained in each measurement area KR between the measurement areas KR of the previous and next image frames F received from the setting unit 13, and generates a luminance signal that indicates the difference between the measured luminance values in time series.
[0047] For example, for image frames F1 and F2 shown in FIG. 5, the measurement unit 14 calculates the difference between the average value of the luminance values of multiple pixels included in a measurement area KR1(1) in the image frame F1 and the average value of the luminance values of multiple pixels included in a measurement area KR2(1) in the image frame F2 that corresponds to the measurement area KR1(1) of the image frame F1.
[0048] The measurement unit 14 then calculates the difference between the average brightness value of multiple pixels contained within a measurement area KR2(1) in the image frame F2 and the average brightness value of multiple pixels contained within a measurement area KR3(1) (not shown) in the image frame F3 (not shown), which corresponds to the measurement area KR2(1) in the image frame F2.
[0049] The measurement unit 14 then calculates the difference between the average brightness value of multiple pixels contained within a measurement area KR3(1) in the image frame F3 and the average brightness value of multiple pixels contained within a measurement area KR4(1) (not shown) in the image frame F4 (not shown), which corresponds to the measurement area KR3(1) in the image frame F3.
[0050] The measurement section 14 may use, for example, a variance value instead of the above-mentioned average value.
[0051] The measurement unit 14 performs the same process as described above for the measurement regions KR of the image frame F other than the measurement regions KR1(1), KR2(1), KR3(1), and KR4(1) of the image frames F1, F2, F3, and F4 described above. The measurement unit 14 outputs a luminance signal KS indicating the pixel difference between the measurement regions KR of the previous and next image frames F to the estimation unit 15, as shown in FIG.
[0052] The estimation unit 15 estimates a pulse wave MH based on the luminance signal KS received from the measurement unit 14. The estimation unit 15 further calculates a pulse rate MS based on the estimated pulse wave MH. The estimation unit 15 outputs at least one of the estimated pulse wave MH and the calculated pulse rate MS.
[0053] FIG. 6 shows the configuration of a pulse wave estimation device 1 according to an embodiment.
[0054] As shown in FIG. 6, the pulse wave estimation device 1 of the embodiment includes an input unit 1N, a processor 1P, an output unit 1S, a storage medium 1K, and a memory 1M in order to perform the above-mentioned functions.
[0055] The input unit 1N is composed of, for example, a camera, a microphone, a keyboard, a mouse, and a touch panel. The processor 1P is the well-known core of a computer that operates hardware according to software. The output unit 1S is composed of, for example, an LCD monitor, a printer, and a touch panel. The memory 1M is composed of, for example, a dynamic random access memory (DRAM) and a static random access memory (SRAM). The storage medium 1K is composed of, for example, a hard disk drive (HDD), a solid state drive (SSD), and a read only memory (ROM).
[0056] The storage medium 1K stores a program 1PR. The program 1PR is a group of instructions that defines the content of the processing to be executed by the processor 1P.
[0057] Regarding the relationship between the functions and configuration in the pulse wave estimation device 1, in hardware, a processor 1P executes a program 1PR stored in a storage medium 1K while using a memory 1M, and, as necessary, realizes the functions of each part from the imaging unit 11 to the estimation unit 15 by controlling the operation of the input unit 1N and the output unit 1S.
[0058] <Operation of the embodiment> Fig. 7 is a flowchart showing the operation of pulse wave estimation device 1 of the embodiment. Fig. 8 is a time chart showing the operation of pulse wave estimation device 1 of the embodiment. The operation of pulse wave estimation device 1 will be described below with reference to the flowchart of Fig. 7 and the time chart of Fig. 8.
[0059] In the following, for ease of explanation and understanding, the measurement region KR is given the suffixes "(KK)" and "(TR)". The suffix "(KK)" refers to face part detection processing, while the suffix "(TR)" refers to tracking processing.
[0060] 2, the photographing unit 11 photographs an image G of a subject T whose pulse wave is to be estimated, that is, a plurality of image frames F1 to Fm. The photographing unit 11 outputs the photographed plurality of image frames F1 to Fm to the detection unit 12.
[0061] Step ST12: The detection unit 12 extracts a plurality of skin regions S1 to Sm from the plurality of image frames F1 to Fm, as shown in Fig. 2. The detection unit 12 outputs the plurality of image frames F1 to Fm and the plurality of skin regions S1 to Sm to the setting unit 13 in sequence.
[0062] Step ST13: When the image frame F1 is output from the detection section 12, since the image frame F1 is the first image frame, in the setting section 13, the face organ detection unit 13A detects the coordinate values of a plurality of face organ points KK1 (not shown) in the skin region S1 of the image frame F1 as shown in Fig. 8. The face organ detection unit 13A also sets the coordinate values of a measurement region KR1 (KK) in the skin region S1, for example, as shown in Fig. 8, based on the coordinate values of the detected plurality of face organ points KK1.
[0063] Step ST14A: When the detection unit 12 outputs the image frame F2 following the image frame F1, the setting unit 13, since the image frame F2 is the second image frame, first, the tracking unit 13B performs tracking processing based on the image frame F1, the skin region S1, and the image frame F2 as shown in Fig. 8. As a result, the tracking unit 13B sets a plurality of tracking points TR2 (not shown) in the skin region S2 in the image frame F2. The tracking unit 13B sets the coordinate values of the measurement region KR2 (TR) that should correspond to the measurement region KR1 (KK) as shown in Fig. 8 based on the set plurality of tracking points TR2.
[0064] Step ST14B: In the setting section 13, next, the face organ detection unit 13A detects the coordinate values of a plurality of face organ points KK2 (not shown) in the skin region S2 of the image frame F2 as shown in Fig. 8. The face organ detection unit 13A also sets the coordinate values of the measurement region KR2 (KK) in the skin region S2 as shown in Fig. 8 based on the coordinate values of the detected plurality of face organ points KK2.
[0065] Step ST15A: When the image frame F3 is output from the detection unit 12, in the setting unit 13, since the image frame F3 is the third image frame, first, the tracking unit 13B performs tracking based on the image frame F2, the skin region S2, and the image frame F3 as shown in Fig. 8, similarly to step ST14A. As a result, the tracking unit 13B sets a plurality of tracking points TR3 (not shown) in the skin region S3 in the image frame F3. The tracking unit 13B sets the coordinate values of the measurement region KR3 (TR) that should correspond to the measurement region KR2 (KK) based on the set plurality of tracking points TR3.
[0066] Step ST15B: In the setting section 13, next, similarly to step ST14B, the face organ detection unit 13A detects the coordinate values of the face organ point KK3 (not shown) in the skin region S3 of the image frame F3 as shown in Fig. 8. The face organ detection unit 13A also sets the coordinate values of the measurement region KR3 (KK) in the skin region S3 based on the coordinate values of the detected face organ point.
[0067] For the subsequent image frame F4, image frame F6, . . . , image frame Fm, the same processes as those in steps ST15A and ST15B are performed.
[0068] Step ST16: The measurement unit 14 measures the luminance value of the measurement region KR, specifically, measures the measurement region KR1(KK), measurement region KR2(TR), measurement region KR2(KK), measurement region KR3(TR), . . . .
[0069] After the above-mentioned measurement, the measurement unit 14 calculates the brightness difference KD. For example, the measurement unit 14 calculates the average value of the brightness values of the pixels in the measurement area KR1 (KK) shown in FIG. 8 and the average value of the brightness values of the pixels in the measurement area KR2 (TR) shown in FIG. 8, and further calculates the difference between the two average values, that is, the brightness difference KD (1-2). Similarly, the measurement unit 14 calculates the average value of the brightness values of the pixels in the measurement area KR2 (KK) shown in FIG. 8 and the average value of the brightness values of the pixels in the measurement area KR3 (TR) shown in FIG. 8, and further calculates the difference between the two average values, that is, the brightness difference KD (2-3). Similarly, the measurement unit 14 calculates the brightness differences KD between the other image frames F3, F4, . . . The measurement unit 14 outputs the brightness signals KS indicating the brightness differences KD (1-2), KD (2-3), . . .
[0070] Step ST17: When the luminance signal KS is output from the measurement unit 14, the estimation unit 15 estimates a pulse wave MH based on the luminance signal KS. The estimation unit 15 also calculates a pulse rate MS based on the estimated pulse wave MH. The estimation unit 15 calculates a pulse rate MS based on the estimated pulse wave MH and the calculated pulse rate MS. beat Output at least one of several MS.
[0071] Effects of the embodiment As described above, according to the pulse wave estimation device 1 of the embodiment, the face organ detection unit 13A sets a measurement area KR1(KK) for the image frame F1 based on the face organ point KK1. The tracking unit 13B sets a measurement area KR2(TR) for the image frame F2 following the image frame F1 based on the tracking point TR2, and the face organ detection unit 13A sets a measurement area KR2(KK) for the image frame F2 based on the face organ point KK2. Furthermore, for the subsequent image frames F3 to Fm, the measurement areas KR(KK) and KR(TR) are similarly set. After the above-mentioned settings, the measurement unit 14 measures pixel values within the measurement areas KR(KK) and KR(TR) and calculates the luminance difference KD of the measured pixel values between the preceding and succeeding measurement areas KR, and the estimation unit 15 estimates the pulse wave MH based on the luminance differences KD(1-2), KD(2-3), ..., and calculates the pulse wave MH. beat Calculate the number MS.
[0072] This makes it possible to avoid a situation in which deviations continue to accumulate as tracking continues to be performed sequentially, and also to avoid a situation in which deviations accumulate between image frames before and after a conventional reset process.
[0073] As a result, the luminance difference KD between the measurement regions KR1 (KK) and KR2 (TR), the luminance difference KD between KR2 (KK) and KR3 (TR), etc. can be measured with higher accuracy than in the past. As a result, the estimation of the pulse wave MH of the subject T and the pulse beat Calculations of several MS can be performed with greater accuracy than before.
[0074] Comparative example. Comparative Example FIG. 9 shows the operation of a pulse wave estimation device of a comparative example.
[0075] In the pulse wave estimation device of the comparative example, as shown in Figure 9, a measurement area KR1 (KK) is set by performing facial organ detection processing only on the first image frame F1, while measurement areas KR2 (TR), KR3 (TR), ... are set on the other subsequent image frames F2, F3, ... by performing tracking processing without performing facial organ detection processing.
[0076] More specifically, for the first image frame, image frame F1, a facial organ detection process is performed on image frame F1 to detect multiple facial organ points KK1 (not shown), and a measurement area KR1(KK) is set within skin area S1 based on the detected multiple facial organ points KK1.
[0077] In the pulse wave estimation device of the comparative example, for image frame F2 following image frame F1, a facial organ detection process is not performed, but a tracking process is performed based on image frame F1, skin area S1, and image frame F2 to detect multiple tracking points TR2 (not shown), and a measurement area KR2(TR) is set within skin area S2 based on the detected tracking points TR2. In the pulse wave estimation device of the comparative example, for image frames F3, ..., F(k-1) following image frame F2, the tracking process is continued in the same manner as for image frame F2, to set measurement areas KR3(TR), ..., KR(k-1)(TR).
[0078] On the other hand, in the pulse wave estimation device of the comparative example, it is specified that the tracking process should be stopped, i.e., reset, for approximately k image frames (k is a predetermined positive integer). As a result, in the pulse wave estimation device of the comparative example, the tracking process is stopped for image frame F(k), i.e., the tracking process is reset, and then face organ detection processing is performed on image frame F(k) in the same manner as the face organ detection processing performed on image frame F1. As a result, a plurality of face organ points KK(k) (not shown) are detected, and a measurement region KR(k)(KK) is set within skin region S(k) based on the detected face organ points KK(k). In the pulse wave estimation device of the comparative example, for image frame F(k+1) and the like subsequent to image frame F(k), the tracking process is continued to set measurement regions KR(k+1)(TR) and the like in the same manner as for image frame F2 and the like subsequent to image frame F1.
[0079] In the pulse wave estimation device of the comparative example, as described above, face organ detection processing is performed on image frames F1, F(k), etc., while tracking processing is performed on other image frames F2, F3, etc. following image frame F1, and image frame F(k+1) following image frame F(k), etc., without face organ detection processing. As a result, in the pulse wave estimation device of the comparative example, tracking processing is performed sequentially on image frame F2, tracking processing on image frame F3, tracking processing on image frame F4 (not shown), etc. By performing the tracking processing sequentially, the deviation between measurement region KR2(TR) and measurement region KR3(TR), the deviation between measurement region KR3(TR) and measurement region KR4(TR), etc. are accumulated. As a result, for example, the position of the measurement area KR(k-1) (TR) of the image frame F(k-1) immediately preceding the image frame F(k) on which the facial organ detection process is performed will be shifted a distance equivalent to the accumulated deviation described above compared to the position of the measurement area KR1 (KK) of the first image frame F1.
[0080] In the pulse wave estimation device of the comparative example, the tracking process is also reset, i.e., the position of the measurement area KR(k)(KK) for the image frame F(k) where only the face part detection process is performed becomes substantially the same as the position of the measurement area KR1(KK) for the image frame F1 where only the face part detection process is performed. Therefore, the position of the measurement area KR(k)(KK) for the image frame F(k) and the position of the measurement area KR(k-1)(TR) for the image frame F(k-1) are separated by an amount approximately equal to the distance between the positions of the measurement area KR1(KK) and the measurement area KR(k-1)(TR) as described above.
[0081] In contrast to the pulse wave estimation device of the comparative example, the pulse wave estimation device 1 of the embodiment basically performs face part detection processing and tracking processing for each image frame F, such as image frames F1, F2, F3, as described above. Therefore, the pulse wave estimation device 1 of the embodiment can avoid the situation in which a distance due to the accumulation of deviation occurs between the position of the measurement region KR(k-1)(TR) of the image frame F(k-1) and the position of the measurement region KR1(KK) of the first image frame F1. Similarly, it can also avoid the situation in which a distance approximately equal to the above-mentioned distance occurs between the position of the measurement region KR(k-1)(TR) of the image frame F(k-1) and the position of the measurement region KR(k)(KK) of the image frame F(k).
[0082] Variations. <Modification> Instead of performing the above-mentioned face organ detection processing and tracking processing for each image frame F, i.e., for each of image frames F1, F2, F3, ..., the modified pulse wave estimation device 1 may perform the face organ detection processing and tracking processing only for a predetermined number of image frames F, and perform only the tracking processing for the other image frames F. More specifically, for example, after performing face organ detection processing on image frame F1, the modified pulse wave estimation device 1 may perform, for subsequent image frames F, only the tracking processing on image frames F2 and F3, both the face organ detection processing and the tracking processing on image frame F4, only the tracking processing on image frames F5 and F6, and both the face organ detection processing and the tracking processing on image frame F7.
[0083] Any of the components of the embodiments may be modified or omitted. [Industrial Applicability]
[0084] The pulse wave estimation device and pulse wave estimation method according to the present disclosure are, for example, beat It can be used to calculate numbers. [Explanation of symbols]
[0085] 1 Pulse wave estimation device, 11 imaging unit, 12 detection unit, 13 setting unit, 14 measurement unit, 15 estimation unit, 13A facial organ detection unit, 13B tracking unit, 1N input unit, 1P processor, 1S output unit, 1M memory, 1K storage medium, 1PR program, T subject, G image, F image frame, S skin area, KK facial organ point, TR tracking point, KR measurement area, KD brightness difference, KS brightness signal, MH pulse wave, MS pulse rate.
Claims
1. an acquisition unit that acquires a plurality of consecutive image frames including a first image frame and a second image frame that are successive to each other, of a subject whose pulse wave is to be estimated; a detector for detecting facial organ points indicative of positions of facial organs of the subject within the first image frame; a tracking unit that refers to the first image frame and sets a tracking point that indicates a position of the facial organ in the second image frame after the movement of the facial organ occurs; a first setting unit that sets a position of a first measurement area in the first image frame, where a luminance value for estimating the pulse wave is to be measured, based on the detected face organ points in the first image frame; a second setting unit that sets a position of a second measurement area, different from the first measurement area, within the second image frame based on a tracking point within the set second image frame; a measurement unit that measures luminance values of pixels in a first measurement area in a first image frame set by the first setting unit and in a second measurement area in a second image frame set by the second setting unit; an estimation unit that estimates a pulse wave of the subject based on a luminance difference that is a difference between a luminance value of a first measurement area in the first image frame and a luminance value of a second measurement area in the second image frame; A pulse wave estimation device comprising:
2. 2. The pulse wave estimation device according to claim 1, wherein the detection section and the tracking section respectively detect the face organ points and set the tracking points for each image frame.
3. 2. The pulse wave estimation device according to claim 1, wherein the detection section and the tracking section extract the face organ points and set the tracking points, respectively, for every predetermined number of image frames.
4. an acquisition unit acquires a plurality of consecutive image frames including a first image frame and a second image frame which are successively obtained from a subject whose pulse wave is to be estimated; a detection unit detecting facial organ points indicative of positions of facial organs of the subject within the first image frame; a tracking unit, by referring to the first image frame, sets a tracking point indicating a position of the facial organ in the second image frame after the movement of the facial organ occurs; a first setting unit sets a position of a first measurement area in the first image frame, in which a luminance value for estimating the pulse wave is to be measured, based on the detected face organ points in the first image frame; a second setting unit sets a position of a second measurement area, different from the first measurement area, in the second image frame based on the tracking point in the set second image frame; a measurement unit measures luminance values of pixels in a first measurement area in a first image frame set by the first setting unit and in a second measurement area in a second image frame set by the second setting unit; an estimation unit estimates a pulse wave of the subject based on a luminance difference that is a difference between a luminance value of a first measurement area in the measured first image frame and a luminance value of a second measurement area in the measured second image frame; Pulse wave estimation method.
Citation Information
Patent Citations
Heart rate detection method and apparatus, computer equipment and readable storage medium
CN111513701A
Biological information measuring device, biological information measuring method, biological information display device and biological information display method
JP2016190022A
Pulse wave detection device and pulse wave detection program
JP2016193022A
Biological information detection device and biological information detection method
JP2019170868A
Pulse wave detection method, pulse wave detection program, and pulse wave detection device
WO2016006027A1
