Method and apparatus for estimating biological information

Abstract

To specify a pulse wave from an image including noise.SOLUTION: A biological information estimation method comprises: a setting step of setting a plurality of regions in an image IM obtained by imaging a driver D; a calculation step of calculating an average luminance value for each region for the continuously input images IM to obtain time-series data of the average luminance value; an analysis step of performing frequency analysis to the time-series data; a detection step of detecting a peak frequency for each region from the result of frequency analysis; and a specification step of searching for a common frequency from the peak frequency in the plurality of regions to specify the frequency of the pulse wave of the driver D.SELECTED DRAWING: Figure 13

Description

Technical Field

[0001] The present invention relates to a biological information estimation method and a biological information estimation device.

Background Art

[0002] When the physical condition of a vehicle driver deteriorates during driving, it may lead to an accident. In order to reduce the risk of accidents, it has been proposed to estimate biological information such as blood pressure indicating the physical condition of the driver. Patent Document 1 discloses a method of identifying a subject's pulse wave from the change in the luminance of an image of the subject and estimating the change in blood pressure from the pulse wave.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, an image of a vehicle driver contains a lot of noise due to various factors such as vehicle vibration, movement of sunlight, entry and exit of a tunnel, etc. It is difficult to accurately identify the pulse wave of the driver from such an image.

[0005] There is a need to improve the accuracy of identifying a pulse wave from an image containing noise in a biological information estimation method and a biological information estimation device.

Means for Solving the Problems

[0006] The biological information estimation method according to the present invention includes a setting step of setting a plurality of regions in an image of a user, and a calculation step of calculating a luminance value for each of the regions for the continuously input images and obtaining time-series data of the luminance values. An analysis step of performing frequency analysis on the aforementioned time series data, A detection step in which peak frequencies are detected for each region based on the results of the frequency analysis, The method includes a selection step of searching for a common frequency from the peak frequencies in the multiple regions to identify the frequency of the user's pulse wave.

[0007] The biological information estimation device according to the present invention is A region setting unit that sets multiple regions in an image of a user, A brightness calculation unit calculates brightness values ​​for each region of the continuously input images and obtains time-series data of the brightness values, The system includes a pulse wave identification unit that calculates a brightness value for each of the aforementioned regions, performs frequency analysis on the time-series data of the brightness values, detects a peak frequency for each of the aforementioned regions from the results of the frequency analysis, searches for a common frequency from the peak frequencies in the multiple regions, and identifies the frequency of the user's pulse wave. [Effects of the Invention]

[0008] According to the present invention, it is possible to improve the accuracy of identifying pulse waves from images containing noise. [Brief explanation of the drawing]

[0009] [Figure 1] (a) is a diagram showing a vehicle equipped with a biometric information estimation device, and (b) is a diagram showing an example of an image captured by an infrared camera. [Figure 2] This is a block diagram showing the schematic configuration of a biological information estimation device. [Figure 3] This is a diagram illustrating the overall area of ​​an image. [Figure 4] This diagram illustrates the multiple regions set within the overall area. [Figure 5] This graph shows an example of the results of frequency analysis. [Figure 6](a) is a table showing the peak frequencies detected from each region, and (b) of FIG. 6 is a table showing the rounded peak frequencies. [Figure 7] It is a table showing the search results of common peak frequencies. [Figure 8] It is a diagram showing a pulse wave detection region. [Figure 9] It is a diagram showing an example of a distance table. [Figure 10] It is a diagram showing the results of frequency analysis of regions A20 and A63 to which a band-pass filter is applied. [Figure 11] It is a diagram showing the pulse wave data of regions A20 and A63. [Figure 12] It is a graph showing the relationship between the peak time difference and blood pressure. [Figure 13] It is a flowchart for explaining the blood pressure estimation process. [Figure 14] It is a flowchart for explaining the details of the region setting process. [Figure 15] It is a flowchart for explaining the details of the pulse wave identification process. [Figure 16] It is a diagram for explaining an example of setting a plurality of regions according to Modification 1. [Figure 17] It is a flowchart for explaining the region setting process according to Modification 1.

Mode for Carrying Out the Invention

[0010] Hereinafter, embodiments of the present invention will be described. In the embodiment, an example of mounting an apparatus (biological information estimation apparatus) for implementing a biological information estimation method on a vehicle will be described. The biological information estimation apparatus estimates the blood pressure of a driver (user) of the vehicle as biological information. (a) of FIG. 1 is a diagram showing a vehicle on which the biological information estimation apparatus 1 is mounted, and (b) of FIG. 1 is a diagram showing an example of an image IM taken by the infrared camera 2. FIG. 2 is a block diagram showing a schematic configuration of the biological information estimation apparatus 1.

[0011] As shown in Figure 1(a), the biometric information estimation device 1 is mounted on the vehicle V. The biometric information estimation device 1 receives a continuous stream of images IM captured by the infrared camera 2 (imaging device). The biometric information estimation device 1 uses these images IM to estimate the pulse wave of the driver D, and then estimates the blood pressure from the pulse wave. The biometric information estimation device 1 can output information based on the estimation results to a display 3 or speaker 4 installed inside the vehicle.

[0012] The infrared camera 2 can be any known infrared camera 2 capable of detecting wavelengths in the near-infrared region. The infrared camera 2 can be the same infrared camera 2 used for monitoring driver D. The infrared camera 2 captures a grayscale image IM by continuously irradiating it with near-infrared light. The infrared camera 2 can capture images by irradiating it with near-infrared light, regardless of whether it is day or night. This suppresses the influence of ambient light on the image compared to color images (RGB images) obtained with an RGB camera.

[0013] In the vehicle interior, infrared camera 2 is installed in a location on the front side of the driver's seat DS, such as the instrument panel IP. Infrared camera 2 is positioned facing the driver's seat DS. Infrared camera 2 photographs the vehicle interior, including the driver's seat DS. As shown in Figure 1(b), the field of view of infrared camera 2 includes a part of the vehicle interior, including the driver's seat DS, the upper body of driver D sitting in the driver's seat DS, and the steering wheel SW held by driver D.

[0014] As shown in Figure 2, the biological information estimation device 1 comprises a storage unit 10 and a control unit 11. Although not shown in the figure, the storage unit 10 is composed of memory such as ROM and RAM. Various programs executed by the biological information estimation device 1 are stored in the memory. The control unit 11 is composed of a processor such as a CPU (Central Processing Unit). The functional configuration of the control unit 11 shown in Figure 2 is realized when the processor executes the programs stored in memory. The memory unit 10 also stores values ​​calculated during the processing of the control unit 11, reference values ​​used in the processing of the control unit 11, tables, etc. As an example, the memory unit 10 stores a distance table 101 and a blood pressure table 102.

[0015] The control unit 11 includes a region setting unit 12, a brightness calculation unit 13, a pulse wave identification unit 14, a blood pressure estimation unit 15, and a notification unit 16.

[0016] <Area setting section: Setting step> Figure 3 illustrates the overall area WA in image IM. Figure 4 is a diagram illustrating the multiple regions A1 to A64 that are set within the overall region WA. As shown in Figure 3, the region setting unit 12 sets the overall region WA in the image IM input from the infrared camera 2. As shown in Figure 4, the region setting unit 12 divides the overall region WA and sets multiple regions A.

[0017] As shown in Figure 3, the overall region WA refers to the area within the image IM that includes driver D. The overall region WA is set, for example, as a rectangular area. The overall region WA is set to include at least driver D's face and driver D's hands gripping the steering wheel SW. The position on the image IM can be determined by the XY coordinate system, which indicates the position of the pixels that make up the image IM. The X coordinate indicates the horizontal position of the image IM as shown in Figure 3, and the Y coordinate indicates the vertical position of the image IM as shown in Figure 3.

[0018] The region setting unit 12 extracts the face of driver D from the image IM using known face recognition processing and sets the overall region WA based on the face of driver D. The region setting unit 12 extracts two feature points from the driver D's face from the image IM, for example, using a known face recognition process. Feature points are points located at characteristic positions on each of the parts that make up the face, such as the eyes, eyebrows, nose, and lips. The region setting unit 12 calculates the distance between the two feature points. The region setting unit 12 extracts the eye and nose feature points of driver D from the image IM, for example as shown in Figure 3, and calculates the distance D1 between the nose feature point and the feature point of either the left or right eye.

[0019] The region setting unit 12 determines the size of the overall region WA based on the distance D1. The distance D1 between the eyes and nose of the driver D as seen in the image IM is not constant and changes depending on the driver D's build, seating position, etc. It is thought that the larger the distance D1, the larger the size of the driver D as seen in the image IM. In other words, the larger the distance D1, the larger the size of the overall region WA is set to be.

[0020] When the overall region WA is a rectangle, the size of the overall region WA can be expressed by its width L (Length) and height W (Width), as shown in Figure 3. The width L represents the length of the X coordinate system on the image IM. The height W represents the length of the Y coordinate system on the image IM.

[0021] The memory unit 10 stores, for example, an algorithm for determining the size of the overall area WA from the distance D1 between the eyes and the nose. The area setting unit 12 can use this algorithm to calculate the size of the overall area WA from the distance D1 between the eyes and the nose. Alternatively, a table showing the size of the overall area WA corresponding to the distance D1 between the eyes and the nose may be created in advance and stored in the memory unit 10. The area setting unit 12 can refer to the table to obtain the size of the overall area WA corresponding to the calculated distance D1.

[0022] The region setting unit 12 further sets the position of the overall region WA in the image IM. If the overall region WA is a rectangle, the position of the overall region WA can be defined by the coordinates (endpoint coordinates) of the four corners P1, P2, P3, and P4. For example, the region setting unit 12 can set the endpoint coordinates using the position of the nose of the driver D detected by face recognition processing as the center of the overall region WA.

[0023] Figure 3 shows examples of setting the coordinates of four endpoints when the coordinates of the center point of the nose are (x,y). The endpoint coordinates of the upper right corner P1 are set to (x+L / 2, yW / 2), the endpoint coordinates of the upper left corner P2 are set to (xL / 2, yW / 2), the endpoint coordinates of the lower right corner P3 are set to (x+L / 2, y+W / 2), and the endpoint coordinates of the lower left corner P4 are set to (xL / 2, y+W / 2). The examples of setting endpoint coordinates are not limited to these; for example, the center of the overall region WA may be set to a position other than the center point of the nose.

[0024] As shown in Figure 4, the region setting unit 12 divides the overall region WA into multiple regions. For example, the region setting unit 12 divides the rectangular overall region WA into a grid. The number of divisions of the overall region WA is predetermined. The number of divisions is not limited, but Figure 4 shows an example in which the overall region WA is divided into 64 regions A1 to A64 (8 units wide x 8 units high). Each region A1 to A64 has the same size.

[0025] The region setting unit 12 stores information for regions A1 to A64 in the storage unit 10. As information for regions A1 to A64, the region setting unit 12 stores, for example, the identifier and endpoint coordinates of each region A1 to A64 in the storage unit 10.

[0026] <Brightness Calculation Unit: Calculation Step> The luminance calculation unit 13 calculates time-series data of luminance values ​​for each region A1 to A64 set by the region setting unit 12. The luminance calculation unit 13 calculates the average luminance value by dividing the sum of the luminance values ​​of the pixels constituting each region by the total number of pixels in each region. The luminance calculation unit 13 stores the calculated average luminance values ​​for each region A1 to A64 in the storage unit 10. Each time an image IM is input from the infrared camera 2, the luminance calculation unit 13 calculates the average luminance value for each region A1 to A64 and stores it sequentially in the storage unit 10. As the average luminance value data is accumulated in the storage unit 10, time-series data of the average luminance values ​​for each region A1 to A64 is obtained.

[0027] <Pulse wave identification section: Analysis step, detection step, identification step> The pulse wave identification unit 14 identifies the pulse wave of driver D from the time-series data of average brightness values, and further identifies the region where the pulse wave is detected (pulse wave detection region) from among multiple regions A1 to A64. The pulse wave identification unit 14 acquires time-series data of the average brightness value for a certain period of time T1 minutes stored in the memory unit 10 for each region A1 to A64. The pulse wave identification unit 14 performs frequency analysis on the time-series data using the fast Fourier transform (FFT).

[0028] Figure 5 is a graph showing an example of the results of frequency analysis. As shown in Figure 5, the change in brightness in each region is shown as a frequency amplitude spectrum through frequency analysis. Figure 5(a) shows the results of the frequency analysis of region A18 shown in Figure 4, Figure 5(b) shows the results of the frequency analysis of region A20, and Figure 5(c) shows the results of the frequency analysis of region A63. In all of these regions, a peak frequency, which is a frequency with a high amplitude, can be found.

[0029] Here, the image IM, which shows the skin of driver D, changes in brightness according to driver D's pulse wave. A pulse wave is a waveform that captures the changes in blood vessel volume that occur as the heart pumps blood. These changes in blood vessel volume fluctuate in a cycle determined by the heart rate. Therefore, the blood flow through driver D's blood vessels increases or decreases in conjunction with these changes in blood vessel volume.

[0030] Hemoglobin in the blood absorbs near-infrared light. Therefore, the image IM of driver D's skin tends to have lower brightness when blood flow is high and higher brightness when blood flow is low. In other words, the brightness changes periodically in accordance with the periodically changing pulse wave of driver D. When the image is taken under conditions with little change in the environment, frequency analysis of the time-series data of the average brightness value generally detects a peak frequency that can be estimated as the pulse wave frequency in the range of 0.7 to 2.0 Hz.

[0031] However, during the operation of vehicle V, various environmental changes occur, such as vibrations of vehicle V, movement of sunlight, and entering and exiting tunnels. Although the influence of ambient light can be reduced by taking images with infrared camera 2, a lot of noise still occurs in the image IM. Therefore, as shown in Figure 5, the frequency analysis of the image IM detects many peak frequencies that appear to be caused by noise. It is thought that pulse wave frequencies are included among these peak frequencies. However, since multiple peak frequencies are also detected in the typical pulse wave frequency range of 0.7 to 2.0 Hz, it is difficult to identify which peak frequencies are pulse wave frequencies.

[0032] In this embodiment, as shown in Figure 4, the entire region WA is divided into a grid, separating it into regions that include the driver D and regions that do not. Furthermore, the region that includes the driver D is further divided into regions that capture the driver's exposed skin and regions that do not. For example, region A18 is a region that does not include driver D. On the other hand, region A20 shows the skin of driver D's forehead, and region A63 shows the skin of driver D's left hand. As shown in Figure 5(a), the frequency analysis results for region A18, which does not include driver D, show that many peak frequencies are included in ranges other than 0.7 to 2.0 Hz, suggesting that it is significantly affected by noise. On the other hand, as shown in Figures 5(b) and 5(c), the frequency analysis results for regions A20 and A63, which capture the driver's skin, show multiple peak frequencies, but the majority fall within the range of 0.7 to 2.0 Hz. These regions are thought to be relatively less affected by noise and contain pulse wave frequencies.

[0033] Thus, even with noisy image IMs, by setting the overall region WA that includes driver D, and further dividing it into multiple regions A1 to A64, it is possible to isolate regions where the noise has less impact and where the pulse wave frequency is easily detectable.

[0034] The pulse wave identification unit 14 searches for common peak frequencies from the analysis results of the frequencies in regions A1 to A64. Peak frequencies caused by noise are generated by random factors and are therefore unlikely to be the same. Therefore, by searching for common frequencies among multiple peak frequencies in regions A1 to A64, the pulse wave frequency can be identified.

[0035] Figure 6(a) is a table showing the peak frequencies detected in each region, and Figure 6(b) is a table showing the rounded peak frequencies. Figures 6(a) and 6(b) show the peak frequencies of some regions within regions A1 to A64. Figure 7 is a table showing the search results for common peak frequencies. As shown in Figure 6(a), the pulse wave identification unit 14 detects peak frequencies for each region A1 to A64 based on the results of frequency analysis in each region. In each region, the pulse wave identification unit 14 detects multiple frequencies as peak frequencies in descending order of power (amplitude). For example, the pulse wave identification unit 14 can detect three frequencies as peak frequencies. Note that the first, second, and third peaks in Figure 6(a) do not represent the order of decreasing power, but rather the order in which they were detected randomly. A lower limit (predetermined value) may be set for the peak frequency. The pulse wave identification unit 14 will not detect a frequency as a peak frequency, even if it is among the top three frequencies in the region, if that frequency is below the lower limit.

[0036] The pulse wave identification unit 14 further searches for a common peak frequency among the peak frequencies detected in each region A1 to A64. Here, since errors may occur in the results of frequency analysis, common peak frequencies may include not only perfectly matching frequencies but also approximate frequencies. The pulse wave identification unit 14 may, for example, extract common peak frequencies if the error is within the range of ±0.05. Alternatively, as shown in Figure 6(b), the pulse wave identification unit 14 may round the detected peak frequency value to the nearest 0.1 so that the approximate frequencies have the same value.

[0037] If multiple common peak frequencies are found, the pulse wave identification unit 14 counts the number of times each peak frequency is found, as shown in Figure 7. The pulse wave identification unit 14 identifies the peak frequency with the highest number of occurrences as the pulse wave frequency. In the example in Figure 7, 1.3 Hz, which had the highest number of occurrences, is identified as the pulse wave frequency. The pulse wave identification unit 14 stores the identified pulse wave frequency in the storage unit 10.

[0038] Figure 8 shows the pulse wave detection area. The pulse wave identification unit 14 selects the region where the pulse wave frequency is detected as the pulse wave detection region. In Figure 8, the region selected as the pulse wave detection region is hatched. In the example in Figure 8, regions A20, A21, A28, A29, A36, and A37, which include the face of driver D, regions A44 and A45, which include the neck of driver D, and regions A58, A59, A62, and A63, which show the hands of driver D, are selected.

[0039] The pulse wave identification unit 14 calculates the distance D2 between each pulse wave detection region and other pulse wave detection regions. As shown in Figure 8, the pulse wave identification unit 14 calculates the distance D2 as, for example, the distance between the endpoint coordinates of the pulse wave detection regions.

[0040] The pulse wave identification unit 14 creates a distance table 101 that records the calculated distance D2 and stores it in the storage unit 10. Figure 9 shows an example of the distance table 101. As shown in Figure 9, the distance table 101 records the combination of two pulse wave detection regions and the calculated distance D2 values, sorted in descending order of distance D2. In the example in Figure 9, the distance D2 between area A20, which captures driver D's face as shown in Figure 8, and area A63, which captures driver D's left hand, is the largest. The second largest distance D2 is between area A21, which captures driver D's face, and area A58, which captures driver D's right hand. This distance D2 is calculated to select two areas from the pulse wave detection area that are suitable for blood pressure estimation, as described later.

[0041] <Blood pressure estimation unit: Selection step, filtering step, estimation step> The blood pressure estimation unit 15 refers to the distance table 101 and selects the combination of two pulse wave detection regions with the largest distance D2. The blood pressure estimation unit 15 uses the time difference that occurs in the pulse waves of the two pulse wave estimation regions to estimate the blood pressure.

[0042] As mentioned above, the pulse wave shows the change in blood volume in the blood vessels caused by the blood pumped from the heart. Here, the time it takes for the blood pumped from the heart to reach the vessels varies depending on the distance from the heart. In other words, it is thought that the time difference in the pulse wave will be larger in areas where there is a difference in distance from the heart.

[0043] Here, pulse wave estimation regions with a large distance D2 on the image IM are considered to have a large difference in distance from the heart. Therefore, the blood pressure estimation unit 15 selects the combination of the two pulse wave detection regions with the largest distance D2 to estimate blood pressure.

[0044] The blood pressure estimation unit 15 acquires time-series data of the average brightness values ​​of regions A20 and A63 with the largest distance D2 from the memory unit 10 and performs frequency analysis using the Fast Fourier Transform. The blood pressure estimation unit 15 applies a bandpass filter when performing frequency analysis on the time-series data of the average brightness values ​​in regions A20 and A63. The blood pressure estimation unit 15 sets the frequency band of the bandpass filter based on the frequency of the pulse wave identified by the pulse wave identification unit 14. The frequency band of the bandpass filter can be set, for example, to ±10% of the pulse wave frequency.

[0045] Figure 10 shows the results of frequency analysis in regions A20 and A63 after applying a bandpass filter. Figure 10(a) shows the results of frequency analysis in region A20, and Figure 10(b) shows the results of frequency analysis in region A63. Figure 11 shows the time-series data for regions A20 and A63 after applying a bandpass filter. Figure 12 is a graph showing the relationship between peak time difference and blood pressure.

[0046] In Figure 10, the frequency analysis results for the bandpass filtered regions A20 and A63 are displayed as amplitude spectra, similar to Figure 5. As shown in Figures 5(b) and 5(c), the frequency analysis results for regions A20 and A63 showed that although the noise effect was relatively small, multiple peak frequencies due to the noise were observed. On the other hand, as shown in Figures 10(a) and 10(b), by applying a bandpass filter, peak frequencies outside the frequency band within ±10% of the pulse wave frequency of 1.3 Hz are removed from the frequency analysis results in regions A20 and A63. In other words, peak frequencies caused by noise are removed from the frequency analysis results in regions A20 and A63. As a result, as shown in Figure 11, the time-series data of luminance values ​​in regions A20 and A63, to which the bandpass filter has been applied, shows periodically occurring frequency peaks, reflecting the waveform of driver D's pulse wave. In other words, by applying a bandpass filter based on the pulse wave frequency to the time-series data of luminance values ​​in regions A20 and A63, driver D's pulse wave data can be obtained. Furthermore, since the bandpass filter is set to a narrow frequency band of ±10% of the pulse wave frequency, peak frequencies caused by noise are easily removed. This results in highly accurate pulse wave data.

[0047] As shown in Figure 11, the waveforms in regions A20 and A63 show roughly the same period, but there is a time difference in the timing of the peak appearance (peak time difference PTD). As mentioned above, the face in region A20 and the hand in region A63 are at different distances from the heart, so the time it takes for the blood pumped from the heart to reach them is different. This is why such a peak time difference PTD occurs. As shown in Figure 12, the peak time difference PTD varies with blood pressure. Higher blood pressure results in a smaller peak time difference PTD, while lower blood pressure results in a larger peak time difference PTD.

[0048] The blood pressure estimation unit 15 detects peaks in regions A20 and A63. For each peak in region A20, the blood pressure estimation unit 15 calculates the time difference between it and the nearest peak in region A63. The blood pressure estimation unit 15 calculates the average value of the peak time difference PTD by dividing the sum of the calculated peak time differences PTD by the number of detected peaks.

[0049] The memory unit 10 stores a blood pressure table 102 (see Figure 2) that shows the correspondence between peak time difference PTD and blood pressure values. The blood pressure estimation unit 15 refers to the blood pressure table 102 and obtains the blood pressure value corresponding to the average value of the calculated peak time difference PTD. The blood pressure estimation unit 15 stores the obtained blood pressure value in the memory unit 10 as an estimated value of driver D's blood pressure.

[0050] The blood pressure estimation unit 15 acquires time-series data from areas A20 and A63 stored in the storage unit 10 at regular intervals T2, performs the blood pressure estimation process described above, and sequentially stores the estimated blood pressure values ​​in the storage unit 10. This provides data showing the changes in driver D's blood pressure. The blood pressure estimation unit 15 continues the blood pressure estimation process while the vehicle V is in motion.

[0051] Furthermore, while the blood pressure estimation unit 15 is processing, it may become impossible to detect pulse waves from regions A20 and A63. For example, if driver D moves the position of their left hand and the left hand is no longer visible in region A63, pulse waves will no longer be detectable from region A63. Alternatively, if sunlight shines locally into region A20, a peak that is not due to a pulse wave may be detected in region A20. In such cases, an appropriate peak time difference PTD cannot be calculated from the combination of regions A20 and A63.

[0052] If an appropriate peak time difference PTD is not calculated, the blood pressure estimation unit 15 can change the combination of pulse wave detection regions used for blood pressure estimation. A reference value for determining an appropriate peak time difference PTD may be set in advance and stored in the memory unit 10. The reference value can be an acceptable value that can be estimated to be caused by blood pressure. If the average value of the calculated peak time difference PTD exceeds the reference value, the blood pressure estimation unit 15 can change the combination of pulse wave detection areas.

[0053] The blood pressure estimation unit 15 refers to the distance table 101 and selects the combination of pulse wave detection regions with the second largest distance D2. In the example in Figure 9, region A21 (see Figure 8) which captures the face of driver D and region A58 (see Figure 8) which captures the right hand of driver D are selected. The blood pressure estimation unit 15 calculates the peak time difference PTD of regions A21 and A58 and estimates the blood pressure. Furthermore, if an appropriate peak time difference PTD cannot be calculated from regions A21 and A58, the blood pressure estimation unit 15 may select the next combination of pulse wave detection regions with the largest distance D2 and continue calculating the peak time difference PTD until an appropriate peak time difference PTD is calculated.

[0054] <Notification section> The notification unit 16 may, for example, display the estimated blood pressure value on the display 3 (see Figure 2) installed inside the vehicle. The blood pressure value may be displayed as a number or as a graph. The notification unit 16 may also monitor changes in blood pressure and output an alarm if there is a sudden rise or fall in blood pressure. The notification unit 16 may display text on the display 3 as an alarm, for example, recommending that the vehicle be stopped. The notification unit 16 may also output a voice or buzzer sound recommending that the vehicle be stopped from the speaker 4 inside the vehicle (see Figure 2) as an alarm. In that case, the memory unit 10 stores a threshold value indicating a rapid rise or fall in blood pressure. The notification unit 16 can refer to the threshold value to determine whether or not an alarm should be issued. Alternatively, the notification unit 16 may transmit the estimated blood pressure value to the driver D's smartphone or the like.

[0055] The following describes the processes performed by the control unit 11. Figure 13 is a flowchart illustrating the blood pressure estimation process. Figure 14 is a flowchart illustrating the details of the area setting process. Figure 15 is a flowchart illustrating the details of the pulse wave identification process.

[0056] As shown in Figure 13, the control unit 11 performs region setting processing (step S01) and pulse wave identification processing (step S02) in order to perform blood pressure estimation processing. The region setting processing may be performed, for example, when the ignition switch of the vehicle V is operated and the infrared camera 2 starts taking pictures. Alternatively, if a seating sensor or seat belt sensor is provided in the driver's seat DS, the region setting processing may be performed at the timing of sensor detection.

[0057] As shown in Figure 14, when an image IM is input from the infrared camera 2 (step S101: Yes), the area setting unit 12 of the control unit 11 performs face recognition processing (step S102) and extracts the eyes and nose of driver D from the image IM. If the region setting unit 12 cannot extract the eyes and nose of driver D from the image IM (step S103: No), it returns to step S101 and performs face recognition processing on the image IM of the next frame. If the eye and nose are extracted from the image IM (step S103: Yes), the region setting unit 12 calculates the distance D1 between the eye and nose (step S104).

[0058] The region setting unit 12 determines the size of the overall region WA based on the distance D1 between the eyes and nose (step S105). The region setting unit 12 calculates the width L and height W of the overall region WA from the distance D1, for example, using an algorithm stored in the memory unit 10.

[0059] The region setting unit 12 sets the position of the overall region WA in the image IM (step S106). For example, if the overall region WA is a rectangle, the region setting unit 12 sets the endpoint coordinates of the four corners P1 to P4, with the position of the driver D's nose as the center.

[0060] The region setting unit 12 divides the overall region WA and sets up multiple regions A1 to A64 (step S107). For example, the region setting unit 12 divides the rectangular overall region WA into a grid using a predetermined number of divisions. The region setting unit 12 stores the endpoint coordinates of each region A1 to A64 in the storage unit 10.

[0061] As shown in Figure 15, once the region setting process is complete, the luminance calculation unit 13 calculates the average luminance value for each region A1 to A64 for the image IM input from the infrared camera 2 (step S201). The luminance calculation unit 13 sequentially calculates the average luminance value for each region A1 to A64 for the image IM continuously input from the infrared camera 2 and stores the calculation results in the storage unit 10. The luminance calculation unit 13 continues this process. That is, each time an image IM is input from the infrared camera 2, the luminance calculation unit 13 calculates the average luminance value and stores it in the storage unit 10. As a result, time-series data of the average luminance values ​​for each region A1 to A64 is accumulated in the storage unit 10.

[0062] When a certain period of time T1 has elapsed, as measured by the timer provided by the control unit 11 (step S202: Yes), the pulse wave identification unit 14 acquires time-series data of the average brightness value for that period of time T1 stored in the memory unit 10. The pulse wave identification unit 14 performs frequency analysis on the time-series data using the Fast Fourier Transform (step S203).

[0063] The pulse wave identification unit 14 detects peak frequencies from the results of frequency analysis in each region A1 to A64 (step S204). For example, in each region A1 to A64, the pulse wave identification unit 14 extracts the top three peak frequencies (first peak, second peak, third peak) in descending order of power.

[0064] The pulse wave identification unit 14 searches for a common peak frequency among the peak frequencies of each region A1 to A64 (step S205). The pulse wave identification unit 14 identifies the common peak frequency as the pulse wave frequency (step S206). If, for example, multiple common peak frequencies are found, the pulse wave identification unit 14 identifies the common peak frequency that was found most frequently as the pulse wave frequency (see Figure 7). The pulse wave identification unit 14 stores the identified pulse wave frequency in the storage unit 10.

[0065] As shown in Figure 15, the pulse wave identification unit 14 calculates the distance D2 between regions from which the pulse wave frequency has been extracted (pulse wave detection regions) (step S207). The pulse wave detection regions are stored in the storage unit 10 in a distance table 101 which records the combination of two pulse wave detection regions and the calculated distance D2 values, in descending order of distance D2.

[0066] Returning to Figure 13, when a certain time T2 has elapsed according to the timer measurement (Step S03: Yes), the blood pressure estimation unit 15 refers to the distance table 101. The blood pressure estimation unit 15 selects the combination of two regions A20 and A63 from the regions recorded as pulse wave detection regions that have the largest distance D2 (Step S04). For the sake of clarity, we have used the notation "two regions A20 and A63" as shown in the example in Figure 8, but it goes without saying that the "two regions" selected are not limited to regions A20 and A63.

[0067] The blood pressure estimation unit 15 acquires time-series data of the average brightness values ​​of two regions A20 and A63 over a fixed period of time T2 minutes, which is stored in the memory unit 10 (step S05). The blood pressure estimation unit 15 acquires the pulse wave frequency stored in the memory unit 10. The blood pressure estimation unit 15 applies a bandpass filter set based on the pulse wave frequency and performs frequency analysis of the time series data in two regions A20 and A63 (step S06) to acquire pulse wave data. The blood pressure estimation unit 15 can, for example, apply a bandpass filter set to ±10% of the pulse wave frequency.

[0068] The blood pressure estimation unit 15 detects frequency peaks in the pulse wave data of regions A20 and A63. The blood pressure estimation unit 15 calculates the time difference between each peak in regions A20 and A63 and calculates the average value of the peak time difference PTD in the time series data (step S07).

[0069] If an appropriate average value of peak time difference PTD is calculated (step S08: Yes), the blood pressure estimation unit 15 estimates the blood pressure based on the average value of the peak time difference PTD (step S09). For example, the blood pressure estimation unit 15 can compare the calculated average value of peak time difference PTD with a preset reference value and determine that the peak time difference PTD is appropriate if it is below the reference value. The blood pressure estimation unit 15 can also, for example, refer to the blood pressure table 102 in the memory unit 10 to obtain the blood pressure value corresponding to the average value of the peak time difference PTD.

[0070] If the blood pressure estimation unit 15 cannot calculate an appropriate average value for the peak time difference PTD (step S08: No), it refers to the distance table 101 and selects the combination of the two regions A21 and A58 with the next largest distance D2 (step S10), and returns to step S05.

[0071] After storing the blood pressure value estimated in step S09 in the storage unit 10, the blood pressure estimation unit 15 returns to step S03, and thereafter, every time a certain period of time T2 has elapsed, it acquires time-series data of the accumulated average brightness value and estimates the blood pressure.

[0072] As described above, the method for estimating biological information includes the following processes. (1) Methods for estimating biological information are: The setting steps (steps S101 to S107) involve setting multiple regions A1 to A64 in the image IM taken of driver D (user), For continuously input image IMs, a calculation step (step S201) is performed to calculate the average brightness value (brightness value) for each region A1 to A64 and obtain time-series data of the average brightness value. An analysis step (step S203) in which frequency analysis is performed on time series data, Based on the results of the frequency analysis, a detection step (step S204) is performed to detect the peak frequency for each region A1 to A64, The system includes a determination step (steps S205-S206) which searches for a common frequency from the peak frequencies in multiple regions A1-A64 to identify the frequency of driver D's pulse wave.

[0073] This biometric information estimation method can improve the accuracy of identifying pulse waves from noisy image imaging (IM). Specifically, by dividing the image IM into multiple regions A1 to A64, it is possible to isolate the region from the image IM that captures driver D's skin and where the pulse wave frequency is easily detectable. The region containing driver D's skin contains the same pulse wave frequency. On the other hand, peak frequencies due to noise are generated by random factors and are therefore unlikely to be the same. Therefore, by searching for common peak frequencies from multiple regions A1 to A64, the accuracy of identifying the pulse wave frequency can be improved even when using an image IM containing noise.

[0074] (2) Methods for estimating biological information are: A selection step (step S04) to select two regions A20 and A63 in which the pulse wave frequency is detected, A filtering step (steps S05-S06) is performed to obtain the pulse wave data of driver D by applying a bandpass filter of a frequency band set based on the pulse wave frequency to the time series data of two regions A20 and A63. The system includes an estimation step (steps S07-S08) which involves determining the peak time difference (PTD) of pulse waves in pulse wave data from two regions A20 and A63, and estimating the blood pressure of driver D based on the peak time difference (PTD).

[0075] In the biomedical information estimation device 1, blood pressure is estimated from the peak time difference (PTD) of pulse waves occurring in pulse wave data from two different regions A20 and A63. Here, since the peak time difference (PTD) is not a large value, if the accuracy of the pulse wave data obtained from the two regions A20 and A63 is low, it may affect the accuracy of the blood pressure estimation. When low-noise image IM can be obtained, pulse wave data can be obtained by applying a bandpass filter in the 0.7-2.0 Hz frequency range when performing frequency analysis on the time-series data of average brightness values. However, in the case of image IM containing a lot of noise, the 0.7-2.0 Hz frequency range also contains peak frequencies due to factors other than pulse waves, so even if a bandpass filter is applied in a general range, highly accurate pulse wave data cannot be obtained.

[0076] In this embodiment, the frequency band of the bandpass filter is set based on the frequency of the pulse wave identified by the pulse wave identification unit 14. By setting the frequency band to a narrow range, for example, ±10% of the pulse wave frequency, peak frequencies caused by factors other than the pulse wave can be removed, and highly accurate pulse wave data can be obtained. This improves the accuracy of calculating the peak time difference PTD for the two regions A20 and A63, thereby improving the accuracy of blood pressure estimation.

[0077] (3) The setup step is: From the image IM, two feature points are extracted from the face of driver D, and the size of the overall region WA encompassing driver D is set based on the distance D1 between the two feature points (steps S102-S105), Setting the position of the endpoint of the entire region WA in the image IM with one of the two feature points mentioned above as the center (step S106), This includes dividing the entire region WA and setting up multiple regions A1 to A64 (step S107).

[0078] By setting a whole region WA encompassing driver D, and then defining multiple regions A1 to A64 within that whole region WA, the proportion of regions A1 to A64 containing driver D's skin increases, making it easier to detect a common peak frequency indicating the pulse wave frequency. This improves the accuracy of identifying the pulse wave frequency. Furthermore, by extracting two feature points from driver D's face, such as the eye and nose feature points, determining the size of the overall area WA based on their distance D1, and determining the endpoint coordinates of the overall area WA with driver D's nose feature point as the center, it becomes easier to set the overall area WA in the image IM where driver D is visible.

[0079] (4) The detection step includes detecting multiple peak frequencies having power above a predetermined value (lower limit) for each region A1 to A64 based on the results of the frequency analysis (step S204).

[0080] Even in a region containing the pulse wave frequency, peak frequencies due to noise higher than the pulse wave frequency may be detected. Detecting multiple peak frequencies in the detection step makes it easier to include the pulse wave frequency. In the embodiment, an example of detecting three peak frequencies in descending order of power (amplitude) was described, but the number of peak frequencies to be detected is not limited to three; for example, two may be used.

[0081] (5) The selection step is, This includes selecting the two regions A20 and A63 with the largest distance D2 between them (step S04), which are pulse wave detection regions (regions where pulse wave frequencies are detected).

[0082] As mentioned above, the peak time difference PTD is not a large value and is therefore susceptible to noise. Here, the greater the difference in distance from the heart between the two regions, the larger the peak time difference PTD tends to be. For example, the difference in distance from the heart between driver D's face and hands tends to be large. In the selection step, if two regions A20 and A63 with a large distance D2 on the image IM are selected, the combination of the region containing driver D's face and the region containing D's hands is likely to be selected. In other words, since two regions A20 and A63 with a large difference in distance from the heart are selected, the peak time difference PTD becomes large, and the influence of noise can be reduced. This can improve the accuracy of blood pressure estimation.

[0083] The biological information estimation device 1 that performs the aforementioned processing can also achieve the same effect.

[0084] [Example 1] Figure 16 is a diagram illustrating an example of setting multiple regions related to Modification Example 1. In the embodiment described above, the region setting unit 12 sets an overall region WA that encompasses the entire driver D in the image IM. In the modified example 1, the region setting unit 12 sets a first overall region WA1 that encompasses the face of the driver D and a second overall region WA2 that includes the steering wheel SW.

[0085] The region setting unit 12 extracts the face of driver D from the image IM using known face recognition processing. As shown in Figure 16, the region setting unit 12 sets the region encompassing the extracted face of driver D as the first overall region WA1. The region setting unit 12 further extracts the steering wheel SW from the image IM using known image processing. The region setting unit 12 sets the region encompassing the extracted steering wheel SW as the second overall region WA2. Driver D holds the steering wheel SW for a long time while driving the vehicle V. Therefore, by setting a region encompassing the steering wheel SW, it is possible to set a region that is highly likely to contain driver D's hands.

[0086] As illustrated in the example of the embodiment, pulse wave frequencies are easily detected in the region of the image IM that includes the face and hands of driver D. Furthermore, because the face and hands are at different distances from the heart, a peak time difference PTD is likely to occur, making them suitable regions for use in blood pressure estimation. In other words, by setting a first overall region WA1 and a second overall region WA2, the region in which pulse wave frequencies are easily detected can be narrowed down in advance. This reduces the processing load when the pulse wave identification unit 14 searches for common peak frequencies.

[0087] The first overall region WA1 and the second overall region WA2 can be, for example, rectangular regions. The region setting unit 12 sets the size of the first overall region WA1 and sets the endpoint coordinates so that the face of the driver D is included. The region setting unit 12 sets the size of the second overall region WA2 and sets the endpoint coordinates so that the steering wheel SW is included.

[0088] The region setting unit 12 divides the first overall region WA1 and the second overall region WA2, respectively, to set up multiple regions. For example, the region setting unit 12 divides the rectangular first overall region WA1 and the second overall region WA2 into a grid using a predetermined number of divisions. The region setting unit 12 stores the endpoint coordinates of the multiple regions in the storage unit 10.

[0089] Figure 17 is a flowchart illustrating the region setting process for Modification Example 1. Since the processes other than the region setting process in Modification Example 1 are the same as in the embodiment, a detailed explanation is omitted. As shown in Figure 17, when an image IM is input from the infrared camera 2 (step S111: Yes), the region setting unit 12 of the control unit 11 performs known face recognition processing and image processing (step S112). The region setting unit 12 extracts the face of driver D from the image IM and sets the position of the first overall region WA1 that includes the face of driver D (step S113). The region setting unit 12 sets the position of the second overall region WA2 that includes the steering wheel SW from the image IM through image processing (step S114).

[0090] The region setting unit 12 divides the first overall region WA1 and the second overall region WA2, respectively, to set up multiple regions (step S115). The region setting unit 12 stores the identifier and endpoint coordinates of each region in the storage unit 10.

[0091] As described above, the biometric information estimation method of Modification 1 includes the following processes. (3) Image IM was taken by an infrared camera 2 (imaging device) installed in vehicle V, capturing the user, who is the driver of vehicle V, D. In methods for estimating biological information, The setup steps are: Extracting the user's face from the image IM and setting a first overall region WA1 that encompasses the user's face (steps S112, S113), Step S114 involves extracting the steering wheel SW of vehicle V from image IM and setting the position of the second overall region WA2 that encompasses the steering wheel SW, This includes dividing the first overall region WA1 and the second overall region WA2 to set up multiple regions (step S115).

[0092] In the image IM of driver D, the face and hands are areas where skin is likely to be exposed, making them regions where pulse waves are easily detected. By setting a first overall region WA1 encompassing driver D's face and a second overall region WA2 encompassing the hands, the areas where pulse waves are easily detected can be narrowed down, thereby improving the accuracy of pulse wave identification.

[0093] Furthermore, driver D holds the steering wheel switch for a long time while driving. Also, the fixed, ring-shaped steering wheel switch is easy to extract using image processing. Therefore, by defining the region encompassing the steering wheel switch extracted by image processing as the second overall region WA2, the overall region WA encompassing the hand can be set.

[0094] Furthermore, since driver D's face and hands are at different distances from the heart, it is easier to calculate the peak time difference PTD. When a first overall region WA1 and a second overall region WA2 are set, the combination of regions included in the first overall region WA1 and regions included in the second overall region WA2 tends to result in a large distance D2 between the regions. In other words, the blood pressure estimation unit 15 is more likely to select two regions in which a peak time difference PTD is likely to occur, thereby improving the accuracy of blood pressure estimation.

[0095] [Other variations] In the above-described embodiment, the biometric information estimation device 1 was shown to estimate the blood pressure of driver D, but it may also estimate the blood pressure of other occupants of vehicle V. In that case, the position or shooting range of the infrared camera 2 is changed so that other occupants can be photographed. The biometric information estimation device 1 can estimate the blood pressure of other occupants from the image IM taken of them.

[0096] The location where the biometric information estimation device 1 is installed is not limited to vehicle V. The biometric information estimation device 1 can improve the accuracy of estimating the user's blood pressure even in shooting environments other than vehicle V where the image IM contains a lot of noise. The biometric information estimation device 1 may be installed, for example, on trains, ships, motorcycles, bicycles, etc.

[0097] In the above-described embodiment, an example was given in which the biological information estimation device 1 estimates blood pressure as biological information, but the device is not limited to this embodiment. The biological information estimation device 1 may output pulse waves as biological information, or it may estimate and output biological information other than blood pressure from pulse waves.

[0098] In the above-described embodiment, an example was explained in which the biometric information estimation device 1 extracts the feature points of the eyes and nose as the facial feature points of driver D, and determines the size of the overall region WA from the distance D1 between them, but the invention is not limited to this. The biometric information estimation device 1 may, for example, calculate the distance between the feature points of driver D's right and left eyes and determine the size of the overall WA region from that distance. Alternatively, the biometric information estimation device 1 may extract the contour of driver D's face as feature points and determine the size of the overall WA region from the size of the contour of the face. If the distance D1 between the eyes and nose cannot be calculated, the biometric information estimation device 1 may determine the size of the overall area WA by these other methods. Furthermore, if the driver D is wearing a mask, sunglasses, hat, etc., and the size of the overall area WA cannot be determined by any method, the device may determine the size of the overall area WA to a pre-set size.

[0099] The embodiments and modifications of the present invention have been described above. The present invention is not limited to the embodiments and modifications shown above. It can be modified as appropriate within the scope of the technical idea of ​​the present invention. [Explanation of Symbols]

[0100] 1. Biological Information Estimation Device 2. Infrared camera 3 displays 4 speakers 10 Storage section 11 Control Unit 12 Area setting section 13. Brightness calculation unit 14. Pulse wave identification area 15. Blood pressure estimation unit 16 Notification Department V Vehicle IM image WA whole area WA1 First Overall Domain WA2 2nd overall area A1~A64 area

Claims

1. A setting step to set multiple areas in an image of the user, A calculation step of calculating brightness values ​​for each region of the continuously input images and obtaining time-series data of the brightness values, An analysis step of performing frequency analysis on the aforementioned time series data, A detection step in which peak frequencies are detected for each region based on the results of the frequency analysis, The system includes a identification step of searching for a common frequency from the peak frequencies in the plurality of regions to identify the frequency of the user's pulse wave, A selection step in which two regions are selected from the regions in which the frequency of the pulse wave identified in the above-mentioned specific step is detected, A filtering step to obtain the user's pulse wave data by applying a bandpass filter of a frequency band set based on the pulse wave frequency to the time series data of the two regions, A method for estimating biological information, comprising: an estimation step of determining the peak time difference of pulse waves in pulse wave data of the two regions, and estimating the user's blood pressure based on the peak time difference.

2. The aforementioned setup step is, Extracting two feature points from the user's face from the aforementioned image, and determining the size of the entire region encompassing the user based on the distance between the two feature points, Setting the position of the endpoint of the entire region in the image with respect to one of the two feature points mentioned above, The method for estimating biological information according to claim 1, comprising dividing the overall area and setting the plurality of areas.

3. The aforementioned image was taken by an imaging device installed in the vehicle, capturing the user, who is the driver of the vehicle. The aforementioned setup step is, Extract the user's face from the aforementioned image and set a first overall region that includes the user's face. The steering wheel of the vehicle is extracted from the aforementioned image, and the position of the second overall region encompassing the steering wheel is set. The method for estimating biological information according to claim 1, comprising dividing the first overall region and the second overall region to set up the plurality of regions.

4. The method for estimating biological information according to claim 1, wherein the detection step includes detecting a plurality of peak frequencies having a power of a predetermined value or more for each region based on the results of the frequency analysis.

5. The aforementioned selection step is, The method for estimating biological information according to claim 1, comprising selecting the two regions where the distance between the regions in which the pulse wave frequency is detected is the largest.

6. A region setting unit that sets multiple regions in an image of a user, A brightness calculation unit calculates brightness values ​​for each region of the continuously input images and obtains time-series data of the brightness values, A pulse wave identification unit performs frequency analysis on the time-series data of the brightness values, detects peak frequencies for each region from the results of the frequency analysis, searches for a common frequency from the peak frequencies in the multiple regions, and identifies the frequency of the user's pulse wave. A biological information estimation device comprising: a blood pressure estimation unit that selects two regions from the aforementioned plurality of regions in which the frequency of the pulse wave is detected, applies a bandpass filter of a frequency band set based on the frequency of the pulse wave to the time series data of the two regions to acquire the user's pulse wave data, determines the peak time difference of the pulse wave in the pulse wave data of the two regions, and estimates the user's blood pressure based on the peak time difference.

Images