Heart rate detection device, heart rate detection system, heart rate detection method, and program

Abstract

The system accurately detects heart rate signals from imaging signals obtained by capturing images of the subject. [Solution] The heart rate detection device includes a signal extraction unit that extracts image components in the skin-colored region from an RGB format moving image signal obtained by imaging a subject with a camera, and a heart rate signal generation unit that generates one or more heart rate signals, each representing a heart rate component, from the image components in the skin-colored region. Furthermore, the heart rate detection device may also include a differential moving average unit that repeats a set of differential processing and moving average processing a predetermined number of times for each of the one or more heart rate signals, and a heart rate information generation unit that generates heart rate information representing the subject's heart rate based on the one or more heart rate signals obtained by repeating the set of differential processing and moving average processing a predetermined number of times.

Description

Technical Field

[0001] The present invention relates to a heartbeat detection device, a heartbeat detection system, a heartbeat detection method, and a program.

Background Art

[0002] In Society 5.0, which is one of the science and technology policies in Japan, by utilizing IoT (Internet of Things) technology, in a society where all things are connected to the Internet, new value creation is realized by analyzing data, aiming to achieve both a comfortable, vibrant, and high-quality life, economic development, and the resolution of social issues. Also, IoT technology has begun to be applied in the field of people. For example, Patent Documents 1 to 9 describe technologies for sensing the biological signals of a subject. In recent years, for the realization of Society 5.0, a technology for non-contact heartbeat measurement of a subject without the subject's awareness, which leads to the analysis of human health and emotions, has attracted attention.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Patent Document 3

Patent Document 4

Patent Document 5

Patent Document 6

Patent Document 7

Patent Document 8

Patent Document 9

[0004] A known method for detecting a subject's heart rate non-contact is to use images captured by a camera. This method has the advantage of being easy to handle because it uses a camera. However, this method is susceptible to optical disturbances, and due to the principle of heart rate detection, the detection signal is very weak, making it difficult to achieve high accuracy. For example, with this method, detecting a heart rate using images captured by a camera was difficult due to variations in brightness from frame to frame, the subject's movement, and noise generated during skin tone detection processing, making it difficult to accurately detect the heart rate signal.

[0005] The present invention aims to provide a heart rate detection device, a heart rate detection system, a heart rate detection method, and a program that accurately detect heart rate signals from imaging signals of a subject. [Means for solving the problem]

[0006] To solve the above-mentioned problems and achieve the objective, the heart rate detection device according to the present invention comprises a heart rate signal generation unit that generates one or more heart rate signals, each representing a heart rate component, from an RGB format moving image signal obtained by imaging a subject with a camera. [Effects of the Invention]

[0007] According to the present invention, a heart rate signal can be accurately detected from an imaging signal obtained by capturing an image of a subject. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 shows the configuration of the heart rate detection system according to the embodiment. [Figure 2]Figure 2 shows the functional configuration of the heart rate detection device. [Figure 3] Figure 3 shows the functional configuration of the heart rate signal generation unit. [Figure 4] Figure 4 is a flowchart showing the processing flow of the heart rate signal generation unit. [Figure 5] Figure 5 shows the hardware configuration of the information processing device. [Modes for carrying out the invention]

[0009] The embodiments will be described below with reference to the drawings.

[0010] Figure 1 shows the configuration of the heart rate detection system 10 according to the embodiment. The heart rate detection system 10 detects heart rate information that represents information about the heart rate of the person being measured, such as heart rate interval data (RRI: RR Interval).

[0011] The heart rate detection system 10 according to this embodiment includes a camera 20, a heart rate detection device 24, and a display device 26.

[0012] Camera 20 captures images of the subject without contact and outputs the captured image signal. Camera 20 is optical and generates moving images at a resolution of 2K or higher and a frame rate of 30fps or higher. If the subject moves, the camera 20 may be controlled to move in order to follow the subject so that the subject is included in the field of view. The captured signal output from camera 20 includes, for example, a component of heartbeat in the component of change in light intensity in areas where the subject's skin is exposed.

[0013] The heart rate detection device 24 outputs heart rate information of the person being measured based on the imaging signal output from the camera 20. The heart rate information is, for example, heart rate interval data. In addition to or in addition to heart rate interval data, the heart rate information may also include, for example, heart rate and a heart rate data sequence representing the time of the heartbeat.

[0014] The heartbeat detection device 24 may output the heartbeat information of the subject at each unit time which is a predetermined time interval. When outputting the heartbeat information at each unit time, the heartbeat detection device 24 may output error information indicating that the heartbeat information could not be detected in the unit time when the heartbeat information could not be detected from the imaging signal.

[0015] The heartbeat detection device 24 is constituted by a computer. The heartbeat detection device 24 may be a server device on a network, or may be a cloud or the like in which a plurality of server devices on the network cooperate to operate. When the heartbeat detection device 24 is a server device or the like on a network, the camera 20 is connected to the heartbeat detection device 24 via the network. The computer and the server device function as the heartbeat detection device 24 by executing a program.

[0016] The display device 26 acquires the heartbeat information from the heartbeat detection device 24 and displays it on the monitor. Thereby, the display device 26 can cause the measurer or the like to recognize the heartbeat information of the subject. Further, the display device 26 acquires the error information from the heartbeat detection device 24 and displays it on the monitor. Thereby, the display device 26 can cause the measurer or the like to recognize that an error has occurred in the detection of the heartbeat information of the subject. Note that the display device 26 may display the heartbeat information and the error information on an LED (Light Emitting Diode) or the like.

[0017] The heart rate detection system 10 may also include, in place of or in addition to the display device 26, at least one of a printer, an audio output device, a storage device, or a communication device. The printer acquires heart rate information and error information from the heart rate detection device 24 and prints it on paper or the like. The audio output device acquires heart rate information and error information from the heart rate detection device 24 and outputs it as audio. The storage device acquires heart rate information and error information from the heart rate detection device 24 and stores it in a storage medium. The communication device acquires heart rate information and error information from the heart rate detection device 24 and transmits it to other devices via a network. By including a printer, an audio output device, a storage device, or a communication device, such a heart rate detection system 10 can also allow the person taking the measurement to recognize the heart rate information of the person being measured and whether an error has occurred.

[0018] Figure 2 shows the functional configuration of the heart rate detection device 24.

[0019] The heart rate detection device 24 comprises a heart rate signal generation unit 32 and a heart rate information generation unit 34.

[0020] The heart rate signal generation unit 32 acquires imaging signals from the camera 20 and generates a heart rate signal representing the heart rate component of the person being measured. For example, the heart rate signal generation unit 32 extracts the image component of the skin-colored region, which is the spatial area where the person's skin is exposed, from the imaging signal output from the camera 20, and generates a heart rate signal based on the image component of the extracted spatial area.

[0021] The heart rate signal output from the heart rate signal generation unit 32 is time-series data representing the magnitude of the heartbeat. For example, the heart rate signal generation unit 32 outputs a heart rate signal for a predetermined measurement period.

[0022] Furthermore, the heart rate signal generation unit 32 sets imaging conditions for the camera 20. For example, the heart rate signal generation unit 32 sets the resolution and frame rate for the camera 20. In addition, the heart rate signal generation unit 32 sets the white balance adjustment function to be enabled or disabled for the camera 20.

[0023] The heart rate information generation unit 34 acquires heart rate signals for the measurement period from the heart rate signal generation unit 32 and generates heart rate information based on the acquired heart rate signals. The heart rate information generation unit 34 then outputs the generated heart rate information to the display device 26.

[0024] Figure 3 shows the functional configuration of the heart rate signal generation unit 32.

[0025] The heart rate signal generation unit 32 includes an imaging condition setting unit 40, an imaging signal acquisition unit 42, a conversion unit 44, a skin tone detection unit 46, a signal extraction unit 48, a signal generation unit 50, a differential moving average unit 52, an output unit 54, and a storage unit 56.

[0026] Prior to generating the heart rate signal, the imaging condition setting unit 40 sets imaging conditions for the camera 20. For example, the heart rate signal generation unit 32 sets the resolution to 2K or higher and the frame rate to 30fps or higher. Also, for example, the heart rate signal generation unit 32 disables image quality adjustment functions such as white balance adjustment for the camera 20.

[0027] The imaging signal acquisition unit 42 acquires the imaging signal output from the camera 20.

[0028] The conversion unit 44 acquires the imaging signal from the imaging signal acquisition unit 42. The conversion unit 44 converts the imaging signal into an RGB format video signal and an HSV format video signal. For example, the conversion unit 44 includes an RGB conversion unit 58 and an HSV conversion unit 60. The RGB conversion unit 58 generates an RGB format video signal by converting the imaging signal to RGB format frame by frame. The HSV conversion unit 60 generates an HSV format video signal by converting the RGB format video signal to HSV format frame by frame.

[0029] The skin tone detection unit 46 acquires a video signal in HSV format from the conversion unit 44. For each frame in the HSV video signal, the skin tone detection unit 46 detects a region of the HSV color space, which is a predetermined range of hue, saturation, and brightness, as a skin tone region.

[0030] For example, the skin tone detection unit 46 may detect a region of the first HSV color space, which is a predetermined first hue range, a first saturation range, and a first brightness range, as a first skin tone region for each frame of the HSV format video signal. Alternatively, the skin tone detection unit 46 may detect a region of the second HSV color space, which is a predetermined second hue range included in the first hue range, a second saturation range included in the first saturation range, and a second brightness range included in the first brightness range, as a second skin tone region.

[0031] Here, the second HSV color space is narrower than the first HSV color space. Therefore, by detecting the first skin tone region, the skin tone detection unit 46 sets a wider range of values ​​in the HSV color space corresponding to skin tone, enabling detection of skin tone regions in environments that are too dark or too bright. Alternatively, by detecting the second skin tone region, the skin tone detection unit 46 sets a narrower range of values ​​in the HSV color space corresponding to skin tone, enabling detection of skin tone regions regardless of the brightness of the image.

[0032] Furthermore, the skin tone detection unit 46 may also detect the overlapping area between the first skin tone region and the second skin tone region as a skin tone region. This allows the skin tone detection unit 46 to accurately detect skin tone regions even when the brightness changes from frame to frame.

[0033] The skin tone detection unit 46 may receive skin tone conditions for detecting skin tone regions from the user or an external device. This allows the skin tone detection unit 46 to detect skin tone regions under different conditions, for example, in response to changes in the environment.

[0034] The signal extraction unit 48 acquires a video signal in RGB format from the conversion unit 44. The signal extraction unit 48 also acquires information from the skin tone detection unit 46 indicating the position of skin-colored regions within the image for each frame. The signal extraction unit 48 extracts image components of skin-colored regions from the RGB video signal for each frame. The signal extraction unit 48 outputs the image components of skin-colored regions for each frame.

[0035] The signal generation unit 50 acquires image components in the skin tone region for each frame from the signal extraction unit 48. The signal generation unit 50 then generates one or more heart rate signals representing the heart rate component from the image components in the skin tone region of the RGB format video signal for each frame. For example, the signal generation unit 50 generates one or more heart rate signals for each frame based on the average light intensity of the R component, the average light intensity of the G component, and the average light intensity of the B component of the RGB format video signal in the skin tone region.

[0036] For example, the signal generation unit 50 generates one or more heartbeat signals based on different calculation formulas. For example, the G component contains more heartbeat components than the B and R components. Therefore, for example, the signal generation unit 50 may generate one of the one or more heartbeat signals based on the average light intensity of the G component. Alternatively, for example, the signal generation unit 50 may generate one of the one or more heartbeat signals based on the difference obtained by subtracting the average light intensity of the B component from the average light intensity of the G component. Alternatively, for example, the signal generation unit 50 may generate one of the one or more heartbeat signals based on the difference obtained by subtracting the average light intensity of the R component from the average light intensity of the G component. Furthermore, for each of the one or more heartbeat signals, the signal generation unit 50 may detect peaks or bottoms, remove unwanted high-frequency components higher than the heartbeat frequency, or remove DC components.

[0037] The differential moving average unit 52 acquires one or more heartbeat signals from the signal generation unit 50. For each of the acquired one or more heartbeat signals, the differential moving average unit 52 repeats the set of differentiation and moving average processing a predetermined number of times. The differential moving average unit 52 may perform the set of differentiation and moving average processing once, or it may perform the set of differentiation and moving average processing two or more times.

[0038] The output unit 54 outputs one or more heart rate signals, which have been processed by the differential moving average unit 52 by repeating the differential processing and moving average processing a predetermined number of times, to the heart rate information generation unit 34. The output unit 54 may also store one or more heart rate signals in the storage unit 56. In this case, the heart rate information generation unit 34 acquires a portion of a specified measurement period from each of the one or more heart rate signals stored in the storage unit 56 and generates heart rate information representing the heart rate of the person being measured. The output unit 54 may also display one or more heart rate signals on the display device 26, print them on paper using a printer, or transmit them to an external device via a network using a communication device.

[0039] Figure 4 is a flowchart showing the processing flow of the heart rate signal generation unit 32. Next, the processing flow of the heart rate signal generation unit 32 will be explained with reference to the flowchart in Figure 4.

[0040] First, in S11, the heart rate signal generation unit 32 sets imaging conditions for the camera 20. For example, the heart rate signal generation unit 32 sets the resolution to 2K or higher, the frame rate to 30fps or higher, and disables image quality adjustment functions such as white balance.

[0041] Next, in S12, the heart rate signal generation unit 32 acquires the imaging signal output from the camera 20.

[0042] Next, in S13, the heart rate signal generation unit 32 generates an RGB format video signal by converting the imaging signal to RGB format frame by frame.

[0043] Next, in S14, the heart rate signal generation unit 32 generates an HSV format video signal by converting the RGB format video signal to HSV format frame by frame.

[0044] Next, the heart rate signal generation unit 32 executes the processes from S16 to S19 for each frame (loop processing between S15 and S20).

[0045] In S16, the heart rate signal generation unit 32 detects a region of the HSV color space, which is a predetermined range of hue, saturation, and brightness, in the frame to be processed in the HSV format video signal, as a skin tone region.

[0046] For example, the heart rate signal generation unit 32 detects a region of the first HSV color space, which is a predetermined first hue range, first saturation range, and first brightness range, as the first skin tone region for a frame of HSV-formatted video signal to be processed. Furthermore, the heart rate signal generation unit 32 detects a region of the second HSV color space, which is a predetermined second hue range included in the first hue range, a second saturation range included in the first saturation range, and a second brightness range included in the first brightness range, as the second skin tone region for a frame of processed. Then, the heart rate signal generation unit 32 detects the overlapping region between the first skin tone region and the second skin tone region as the skin tone region for a frame of processed. In this case, the second HSV color space is narrower than the first HSV color space.

[0047] Next, in S17, the heart rate signal generation unit 32 extracts image components in the skin tone region from the frame of the RGB format video signal to be processed.

[0048] Next, in S18, the heart rate signal generation unit 32 calculates the average light intensity of the R component, the average light intensity of the G component, and the average light intensity of the B component in the skin tone region based on the frame of the RGB format video signal to be processed.

[0049] Next, in S19, the heart rate signal generation unit 32 calculates one or more heart rate values ​​using one or more different calculation formulas based on the average light intensity of the R component, the average light intensity of the G component, and the average light intensity of the B component in the skin tone region. Each of the one or more heart rate values ​​represents the value of the corresponding heart rate signal at the time of the frame being processed.

[0050] Then, once the heart rate signal generation unit 32 has completed processing S16 to S19 for all frames, it proceeds to processing S21. By performing processing S16 to S19 for all frames, the heart rate signal generation unit 32 can generate one or more heart rate signals. Each of the one or more heart rate signals is time-series data of heart rate values.

[0051] In S21, the heart rate signal generation unit 32 repeats the differential processing and moving average processing combination a predetermined number of times for each of the one or more heart rate signals it has generated. The heart rate signal generation unit 32 may perform the differential processing and moving average processing combination once, or it may perform the differential processing and moving average processing combination two or more times.

[0052] Next, in S22, the heart rate signal generation unit 32 outputs one or more heart rate signals, obtained by repeating the differential processing and moving average processing combination a predetermined number of times, to the heart rate information generation unit 34. The heart rate signal generation unit 32 may also store one or more heart rate signals.

[0053] The heart rate signal generation unit 32 terminates this flow after completing the processing in S22.

[0054] As described above, the heart rate signal generation unit 32 extracts image components in the skin-colored region from the RGB format video signal and generates one or more heart rate signals from the RGB signal image components corresponding to the skin-colored region. As a result, the heart rate signal generation unit 32 can generate one or more heart rate signals that accurately represent the heart rate components contained in the exposed skin of the subject in the RGB format video signal.

[0055] Furthermore, the heart rate signal generation unit 32 generates heart rate information representing the subject's heart rate by repeating a set of differential processing and moving average processing for each of one or more heart rate signals a predetermined number of times. In this way, the heart rate signal generation unit 32 can remove disturbances such as optical disturbances or the subject's movements, and generate heart rate information with high accuracy.

[0056] Figure 5 shows the hardware configuration of the information processing device. The heart rate detection device 24 is implemented, for example, by a device with a hardware configuration similar to that of a general information processing device, as shown in Figure 5. The information processing device comprises a CPU (Central Processing Unit) 201, an operating device 202, a main memory 203, an auxiliary memory 204, a communication device 205, and a bus 206. Each part is connected by the bus 206.

[0057] The CPU 201 uses a predetermined area of ​​the main memory 203 as a working area and, in cooperation with various programs pre-stored in the auxiliary memory 204, etc., executes various processes and comprehensively controls the operation of each part that constitutes the heart rate detection device 24. In addition, the CPU 201 operates the operating device 202 and the communication device 205, etc., in cooperation with the programs.

[0058] The operating device 202 is an input device such as a touch panel, mouse, or keyboard, which receives information input from the user as an instruction signal and outputs that instruction signal to the CPU 201.

[0059] The main memory 203 is a volatile storage medium such as SDRAM (Synchronous Dynamic Random Access Memory). The main memory 203 functions as a workspace for the CPU 201.

[0060] The auxiliary storage device 204 is a rewritable recording device such as a semiconductor storage medium like flash memory, or a magnetically or optically recordable storage medium. The auxiliary storage device 204 stores the program used for control. The communication device 205 transmits and receives data with other devices.

[0061] The program executed by the heart rate detection device 24 may be stored on a computer connected to a network such as the Internet and provided by downloading it via the network. Alternatively, the program executed by the heart rate detection device 24 may be pre-installed on a portable storage medium or the like and provided in that form.

[0062] The program executed by the heart rate detection device 24 has a modular configuration including an imaging condition setting module, an imaging signal acquisition module, a conversion module, a skin tone detection module, a signal extraction module, a signal generation module, a differential moving average module, and an output module. The CPU 201 reads such a program from a storage medium or the like and loads each of the above modules into the main memory 203. Then, by executing such a program, the CPU 201 functions as the imaging condition setting unit 40, the imaging signal acquisition unit 42, the conversion unit 44, the skin tone detection unit 46, the signal extraction unit 48, the signal generation unit 50, the differential moving average unit 52, and the output unit 54. Also, by the CPU 201 executing such a program, the main memory 203 or auxiliary memory 204 functions as the storage unit 56. Note that some or all of the imaging condition setting unit 40, the imaging signal acquisition unit 42, the conversion unit 44, the skin tone detection unit 46, the signal extraction unit 48, the signal generation unit 50, the differential moving average unit 52, and the output unit 54 may be configured by hardware.

[0063] Although embodiments of the present invention have been described above, these embodiments are presented as examples only and are not intended to limit the scope of the invention. Various modifications can be made to the embodiments. [Explanation of Symbols]

[0064] 10 Heart rate detection system 20 cameras 24 Heart rate detection device 26 Display device 32 Heart rate signal generation unit 34 Heart rate information generation unit 40 Imaging Condition Setting Unit 42 Image signal acquisition unit 44 Conversion section 46 Skin tone detection unit 48 Signal Extraction Section 50 Signal generation unit 52 Differential moving average section 54 Output section 56 Memory section 58 RGB conversion section 60 HSV conversion section

Claims

1. A heart rate signal generation unit generates one or more heart rate signals, each representing a heart rate component, from RGB format video signals obtained by imaging a subject with a camera. A heart rate detection device equipped with the following features.

2. The system further comprises a signal extraction unit that extracts image components in the skin-color region from the aforementioned moving image signal. The heart rate signal generation unit generates one or more heart rate signals from the image components in the skin-colored region. The heart rate detection device according to claim 1.

3. A differential moving average unit that repeats a set of differential processing and moving average processing a predetermined number of times for each of the one or more heart rate signals, A heart rate information generation unit generates heart rate information representing the heart rate of the person being measured, based on one or more heart rate signals obtained by repeating the combination of the differential processing and the moving average processing a predetermined number of times. To further enhance The heart rate detection device according to claim 2.

4. The camera further includes an imaging condition setting unit that disables the white balance adjustment function, which adjusts the image quality of the white balance. A heart rate detection device according to any one of claims 1 to 3.

5. A conversion unit that converts the imaging signal output from the camera into an RGB format video signal and an HSV format video signal, A skin tone detection unit detects the skin tone region based on the HSV format video signal, To further prepare. The heart rate detection device according to claim 2 or 3.

6. The system further includes a skin tone detection unit that detects a region of the HSV color space, which is a predetermined hue range, saturation range, and brightness range, as the skin tone region for each frame of the HSV format video signal. The heart rate detection device according to claim 5.

7. The skin tone detection unit, for each frame of the HSV format video signal, A region of the first HSV color space, which consists of a predetermined first hue range, first saturation range, and first lightness range, is detected as the first skin tone region. The region of the second HSV color space, which is a predetermined second hue range included in the first hue range, a second chroma range included in the first chroma range, and a second lightness range included in the first lightness range, is detected as the second skin tone region. The overlapping region between the first skin-color region and the second skin-color region is detected as the skin-color region. The second HSV color space is narrower than the first HSV color space. The heart rate detection device according to claim 6.

8. The heart rate signal generation unit generates one or more heart rate signals based on the average light intensity of the R component, the average light intensity of the G component, and the average light intensity of the B component in the skin-color region. The heart rate detection device according to claim 2 or 3.

9. A camera that images the person being measured, Heart rate detection device, Equipped with, The aforementioned heart rate detection device is A heart rate signal generation unit generates one or more heart rate signals, each representing a heart rate component, from an RGB format video signal obtained by imaging a subject with the aforementioned camera, A heart rate detection system having the following features.

10. The information processing device generates one or more heart rate signals, each representing a heart rate component, from RGB format video signals obtained by imaging a subject with a camera. Heart rate detection method.

11. A program that makes a computer function as a heart rate detection device, The aforementioned computer, A heart rate signal generation unit generates one or more heart rate signals, each representing a heart rate component, from RGB format video signals obtained by imaging a subject with a camera. A program that makes something work.

Images