Heartbeat detection device, heartbeat detection system, heartbeat detection method, and program

Abstract

To accurately detect heartbeat information from a moving image signal.SOLUTION: A heartbeat detection device includes: a detection signal generation unit that generates a first detection signal including a heartbeat component of a measurement subject and a second detection signal having a small heartbeat component on the basis of a signal component of a skin exposure region where the skin of the measurement subject is exposed in a moving image signal obtained by imaging the measurement subject; a difference signal generation unit that generates a difference signal indicating a difference between the first detection signal and the second detection signal; a heartbeat signal generation unit that generates a heartbeat signal on the basis of the difference signal; a vertex data generation unit that generates a plurality of pieces of vertex data in the heartbeat signal; an error correction unit that generates, as a heartbeat data sequence, the plurality of pieces of valid vertex data obtained by performing error correction on missing or erroneous valid vertex data in a case where a part of a set of the plurality of pieces of valid vertex data is missing or erroneous; and an output unit that outputs heartbeat information based on the heartbeat data sequence.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a heartbeat detection device, a heartbeat detection system, a heartbeat detection method, and a program. [Background technology]

[0002] Society 5.0, one of Japan's science and technology policies, aims to utilize IoT (Internet of Things) technology in a society where everything is connected to the Internet, thereby creating new value through data analysis and achieving a comfortable, vibrant, and high-quality life, as well as economic development and the resolution of social issues. IoT technology is also beginning to be applied to the human field. For example, Patent Documents 1 to 12 describe technologies for contactless sensing of a subject's biosignals. In recent years, in order to realize Society 5.0, technology for contactless heart rate measurement without the subject's awareness has been attracting attention, leading to the analysis of human health and emotions. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2023-139294 [Patent Document 2] Japanese Patent Application Publication No. 2023-036691 [Patent Document 3] Patent No. 7336696 [Patent Document 4] Japanese Patent Publication No. 2022-084884 [Patent Document 5] Japanese Patent Application Publication No. 2018-089369 [Patent Document 6] Patent No. 7203314 [Patent Document 7] Japanese Patent Application Publication No. 2018-008039 [Patent Document 8] Patent No. 7308564 [Patent Document 9] Special Publication No. 2023-502295 [Patent Document 10] Patent No. 7131709 [Patent Document 11] Special Publication No. 2022-518751 [Patent Document 12] International Publication No. 23 / 090429 Summary of the Invention [Problem to be solved by the invention]

[0004] A known method for detecting the heart rate of a person being measured without contact is to use an image captured by a camera. The method of detecting the heart rate from an image captured by a camera has the advantage of being easy to handle because it uses a camera. However, the heart rate detection method using an image captured by a camera is vulnerable to light disturbances, and the detection signal is very weak due to the principles of heart rate detection, making it difficult to improve accuracy.

[0005] The present invention has been made in consideration of the above, and aims to provide a heart rate detection device, a heart rate detection system, a heart rate detection method, and a program that accurately detect heart rate information from a moving image signal obtained by capturing an image of a person being measured. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems and achieve the object, the heartbeat detection device of the present invention comprises: a detection signal generation unit that generates a first detection signal including a heartbeat component of the subject and a second detection signal that includes less of the heartbeat component than the first detection signal based on signal components of an exposed skin area of ​​the subject in a moving image signal obtained by capturing an image of the subject; a differential signal generation unit that generates a differential signal representing the difference between the first detection signal and the second detection signal; a heartbeat signal generation unit that generates a heartbeat signal representing the heartbeat component based on the differential signal; a vertex data generation unit that generates a plurality of vertex data representing the times of vertices that are peak points or bottom points in the heartbeat signal; a valid data extraction unit that extracts from the plurality of vertex data sets a plurality of valid vertex data sets that satisfy the periodicity of the heartbeat; an error correction unit that, if some of the plurality of valid vertex data sets are missing or incorrect, generates the plurality of valid vertex data sets as a heartbeat data sequence by error-correcting the missing or incorrect valid vertex data; and an output unit that outputs heartbeat information based on the heartbeat data sequence. [Effects of the Invention]

[0007] According to the present invention, heart rate information can be detected with high accuracy from a moving image signal obtained by capturing an image of a subject. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing the configuration of a heartbeat detection system according to the first embodiment. [Figure 2] FIG. 2 is a diagram showing the functional configuration of the heartbeat detecting device according to the first embodiment. [Figure 3] FIG. 3 is a diagram illustrating a first example of the configuration of the detection signal generating unit. [Figure 4] FIG. 4 is a diagram illustrating a second example of the configuration of the detection signal generating unit. [Figure 5] FIG. 5 is a diagram illustrating a third example of the configuration of the detection signal generating unit. [Figure 6] FIG. 6 is a flowchart showing an example of the flow of processing by the heartbeat signal generating unit. [Figure 7]FIG. 7 is a diagram for explaining the process of joining vertices in the heartbeat signal generating unit. [Figure 8] FIG. 8 is a flowchart showing an example of the flow of a process for generating vertex data. [Figure 9] FIG. 9 is a flowchart showing an example of the flow of processing by the valid data extraction unit. [Figure 10] FIG. 10 is a diagram showing a first example for explaining the process of extracting a plurality of sets of valid vertex data. [Figure 11] FIG. 11 is a diagram illustrating a second example for explaining the process of extracting a set of multiple valid vertex data. [Figure 12] FIG. 12 is a flowchart showing an example of the flow of processing by the error correction unit. [Figure 13] FIG. 13 is a flowchart illustrating an example of the processing flow of the phase difference calculation unit and the output unit. [Figure 14] FIG. 14 is a diagram showing the configuration of a heartbeat detecting device according to the second embodiment. [Figure 15] FIG. 15 is a flowchart showing an example of the flow of processing by the valid data extraction unit according to the second embodiment. [Figure 16] FIG. 16 is a diagram for explaining the process of extracting a plurality of sets of valid vertex data according to the second embodiment. [Figure 17] FIG. 17 is a diagram illustrating a hardware configuration of an information processing device. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an embodiment will be described with reference to the drawings.

[0010] (First embodiment) 1 is a diagram showing the configuration of a heartbeat detection system 10 according to the first embodiment. The heartbeat detection system 10 detects heartbeat information representing information relating to the heartbeat of a subject without contacting the subject.

[0011] The heartbeat detection system 10 includes a camera 20, a light emitting device 22, a heartbeat detection device 24, and a display device 26.

[0012] Camera 20 captures an image of the subject and outputs a video signal of the subject. Camera 20 is optical and generates a video signal at a frame rate of 30 fps or higher with a resolution of 2K or higher. When the subject moves, camera 20 may be controlled to move its imaging direction to follow the subject so as to include the subject within its field of view.

[0013] The light emitting device 22 irradiates the subject with light. This allows the camera 20 to output a moving image signal that makes it easy to detect the subject's heartbeat. The light emitting device 22 irradiates light that includes at least one wavelength of visible light, near-infrared light, and infrared light. The light emitting device 22 may irradiate continuous light or pulsed light. Note that if the optical environment allows the camera 20 to generate a moving image signal that makes it possible to detect the subject's heartbeat, the heartbeat detection system 10 may not include the light emitting device 22.

[0014] The heartbeat detector 24 acquires the moving image signal output from the camera 20, performs image processing on the acquired moving image signal, and outputs heartbeat information of the subject. The heartbeat information may be, for example, the heart rate, the heartbeat period, or a heartbeat data string including multiple data representing the time of the heartbeat.

[0015] The heartbeat detecting device 24 may output the heartbeat information of the subject at each unit time, which is a predetermined time interval. When the heartbeat information is output at each unit time, the heartbeat detecting device 24 may output error information indicating that the heartbeat information could not be detected during the unit time during which the heartbeat information could not be detected from the moving image signal.

[0016] Furthermore, the heart rate detection device 24 controls the light emitting device 22 to control the light irradiated from the light emitting device 22 to the person being measured. Furthermore, if the imaging direction of the camera 20 can track the person being measured, the heart rate detection device 24 detects the position of the person being measured and controls the imaging direction of the camera 20.

[0017] The heartbeat detection device 24 is configured by a computer. The heartbeat detection device 24 may be a server device on a network, or may be a cloud or the like in which multiple server devices on a network operate in cooperation with each other. When the heartbeat detection device 24 is a server device or the like on a network, the camera 20 and the light-emitting device 22 are connected to the heartbeat detection device 24 via the network. The computer and the server device function as the heartbeat detection device 24 by executing a program.

[0018] The display device 26 acquires heart rate information from the heart rate detection device 24 and displays it on a monitor. This allows the person taking the measurement to recognize the heart rate information of the person being measured. The display device 26 also acquires error information from the heart rate detection device 24 and displays it on a monitor. This allows the person taking the measurement to recognize that an error has occurred in detecting the heart rate information of the person being measured. The display device 26 may also display the heart rate information and error information on an LED (Light Emitting Diode) or the like.

[0019] The heart rate detection system 10 may include at least one of a printing device, an audio output device, a storage device, or a communication device instead of or in addition to the display device 26. The printing device acquires heart rate information and error information from the heart rate detection device 24 and prints them on paper or the like. The audio output device acquires heart rate information and error information from the heart rate detection device 24 and outputs them as audio. The storage device acquires heart rate information and error information from the heart rate detection device 24 and stores them in a storage medium. The communication device acquires heart rate information and error information from the heart rate detection device 24 and transmits them to another device via a network. By including such a printing device, audio output device, storage device, or communication device, the heart rate detection system 10 can also notify the person being measured of the heart rate information and that an error has occurred.

[0020] FIG. 2 is a diagram showing the functional configuration of the heartbeat detection device 24 according to the first embodiment.

[0021] The heartbeat detection device 24 includes a light emission control unit 32, an acquisition unit 34, an area identification unit 36, a detection signal generation unit 38, a difference signal generation unit 40, a heartbeat signal generation unit 42, a vertex data generation unit 44, a valid data extraction unit 46, an error correction unit 48, a memory unit 50, a phase difference calculation unit 52, an error output unit 54, and an output unit 56.

[0022] The light emission control unit 32 controls the light emitting device 22 so that the light emitting device 22 irradiates the subject with light.

[0023] The acquisition unit 34 acquires a video signal of the subject captured by the camera 20. The acquisition unit 34 performs image processing on the video signal to adjust sharpness, brightness, etc., so that subsequent processing can be performed appropriately. The acquisition unit 34 may also perform frame synthesis, composite processing, contrast correction, etc. on the video signal.

[0024] The region identifying unit 36 ​​acquires the video signal from the acquiring unit 34. The region identifying unit 36 ​​performs image analysis on the acquired video signal, and identifies a first partial region and a second partial region for each frame in the video signal.

[0025] The first partial region is a region in the exposed skin region where the subject's skin is exposed, where the heartbeat component is easily detected.

[0026] The second partial region is a region in the exposed skin region where the subject's skin is exposed, which is different from the first partial region, and is a region in which it is more difficult to detect a heartbeat component than the first partial region.

[0027] For example, the region identification unit 36 ​​may detect the contours of the subject's face and features such as the eyes, nose, and mouth based on the video signal to identify a predetermined portion of the subject's body where the skin is exposed and identify the identified portion as a first partial region. Alternatively, the region identification unit 36 ​​may identify a predetermined portion of the subject's body where the skin is exposed, different from the first partial region, and identify the identified portion as a second partial region. Alternatively, the region identification unit 36 ​​may detect blood flow at each position in the exposed skin region of the subject based on the video signal using a predetermined algorithm, and identify a portion with a high blood flow as a first partial region and a portion with a blood flow that is a predetermined amount or a predetermined percentage less than the first partial region as a second partial region.

[0028] The detection signal generation unit 38 acquires the moving image signal from the acquisition unit 34. The detection signal generation unit 38 acquires, from the region identification unit 36, information identifying the first partial region and information identifying the second partial region.

[0029] The detection signal generating unit 38 generates a first detection signal including the subject's heartbeat component and a second detection signal including less of the heartbeat component than the first detection signal, based on the signal component of the exposed skin region of the moving image signal. The first detection signal and the second detection signal are signals whose levels include the heartbeat component. Note that the second detection signal does not necessarily need to include any heartbeat component.

[0030] For example, the detection signal generation unit 38 extracts the signal component of the first partial region from the moving image signal acquired from the acquisition unit 34 based on information identifying the first partial region acquired from the region identification unit 36, and generates a first detection signal and a second detection signal based on the extracted signal component.

[0031] Alternatively, the detection signal generation unit 38 cuts out the signal component of the first partial region in the video signal acquired from the acquisition unit 34 based on the information identifying the first partial region acquired from the region identification unit 36, and generates a first detection signal based on the cut-out signal component. In this case, the detection signal generation unit 38 further cuts out the signal component of the second partial region in the video signal acquired from the acquisition unit 34 based on the information identifying the second partial region acquired from the region identification unit 36, and generates a second detection signal based on the cut-out signal component. A more detailed configuration of the detection signal generation unit 38 will be described later with reference to FIGS. 3, 4, and 5.

[0032] The differential signal generating section 40 acquires the first detection signal and the second detection signal from the detection signal generating section 38. The differential signal generating section 40 generates a differential signal that represents the difference between the first detection signal and the second detection signal.

[0033] Here, optical noise contained in the moving image signal capturing the exposed skin area is included as in-phase noise in both the first detection signal and the second detection signal. In contrast, the first detection signal contains more heartbeat components than the second detection signal. Therefore, the differential signal representing the difference between the first and second detection signals has the optical noise contained in the signal components of the exposed skin area canceled out, leaving the heartbeat component. Therefore, the differential signal, captured by the camera 20, contains less noise and is a signal from which the heartbeat component can be easily detected.

[0034] The heartbeat signal generating unit 42 acquires the differential signal from the differential signal generating unit 40. Based on the differential signal, the heartbeat signal generating unit 42 generates a heartbeat signal representing the heartbeat component. The heartbeat signal generating unit 42 generates the heartbeat signal by detecting vertices, which are peak points or bottom points, from the differential signal, removing unnecessary high-frequency components higher than the frequency of the heartbeat, and removing DC components. An example of the processing flow in the heartbeat signal generating unit 42 will be described later with reference to FIG. 6.

[0035] The vertex data generation unit 44 acquires the heartbeat signal from the heartbeat signal generation unit 42. The vertex data generation unit 44 generates a plurality of vertex data from the heartbeat signal. Each of the plurality of vertex data represents the time and level of a vertex, which is a peak point or a bottom point in the heartbeat signal. In this embodiment, the vertex data generation unit 44 divides the heartbeat signal by unit time, for example, S time (S is a positive integer), and generates a plurality of vertex data for each unit time.

[0036] The cardiac waveform includes one waveform in which the level rises or falls sharply in a short period of time within one cycle. Therefore, the times included in each of the plurality of vertex data generated by the vertex data generating unit 44 are candidates for the times at which the waveform in which the level rises or falls sharply in the cardiac waveform occurs. An example of the processing flow in the vertex data generating unit 44 will be described later with reference to FIG. 8.

[0037] The valid data extraction unit 46 acquires multiple vertex data sets from the vertex data generation unit 44. The valid data extraction unit 46 extracts multiple sets of valid vertex data sets that satisfy the periodicity of a heartbeat from the multiple vertex data sets. In this embodiment, the valid data extraction unit 46 extracts multiple sets of valid vertex data sets for each unit time.

[0038] The plurality of vertex data may include vertex data that does not represent the time when a waveform with abrupt increases or decreases in level occurred in the heartbeat waveform due to noise or the like. A set of multiple valid vertex data is a data group obtained by removing vertex data caused by noise or the like from the plurality of vertex data. The range of a human heartbeat cycle and the range of heartbeat fluctuation are known in advance. Therefore, the valid data extraction unit 46 can extract, from the plurality of vertex data, a set of multiple valid vertex data that satisfies the periodicity of a heartbeat using a predetermined algorithm that takes into account the range of a human heartbeat cycle and the range of heartbeat fluctuation.

[0039] Furthermore, due to the influence of errors and the like, the plurality of vertex data may not include vertex data representing the time at which a waveform with a steep increase or decrease in level occurred in some cardiac cycles. In such cases, the set of valid vertex data may lack valid vertex data representing the time at which some cardiac cycles occurred. In other words, the plurality of cardiac cycles represented by the set of valid vertex data may include blank cardiac cycles that do not include valid vertex data.

[0040] Furthermore, in this embodiment, the valid data extraction unit 46 determines, for each unit time, whether the number of valid vertex data included in the extracted set of valid vertex data is equal to or greater than a predetermined number. If the number of valid vertex data included in the extracted set of valid vertex data is less than the predetermined number, the valid data extraction unit 46 outputs a notification indicating an error to the error output unit 54. Furthermore, if the number of valid vertex data included in the extracted set of valid vertex data is less than the predetermined number, the valid data extraction unit 46 terminates subsequent processing for that unit time.

[0041] An example of the processing flow in the valid data extraction unit 46 will be described later with reference to FIG.

[0042] The error correction unit 48 acquires multiple sets of valid vertex data from the valid data extraction unit 46. In this embodiment, the valid data extraction unit 46 acquires multiple sets of valid vertex data from the valid data extraction unit 46 if the number of valid vertex data included in the extracted sets of valid vertex data per unit time is equal to or greater than a predetermined number.

[0043] If some of the sets of valid vertex data are missing or erroneous, the error correction unit 48 generates a heartbeat data sequence by correcting the errors in the missing or erroneous valid vertex data. For example, the error correction unit 48 determines whether the acquired sets of valid vertex data are missing valid vertex data representing some of the times in the heartbeat cycle. If the acquired sets of valid vertex data are missing valid vertex data representing some of the times in the heartbeat cycle, the error correction unit 48 generates new valid vertex data representing the missing valid vertex data by interpolating from the sets of valid vertex data. Then, the error correction unit 48 generates a heartbeat data sequence by adding the new valid vertex data generated by interpolation to the sets of valid vertex data.

[0044] In this embodiment, the error correction unit 48 generates such a heartbeat data sequence for each unit time. An example of the processing flow in the error correction unit 48 will be described later with reference to FIG.

[0045] The storage unit 50 acquires and stores the heartbeat data sequence from the error correction unit 48. In this embodiment, the storage unit 50 acquires and stores the heartbeat data sequence for each unit time.

[0046] The phase difference calculation unit 52 calculates, for each unit time, the phase difference between the heartbeat data sequence stored in the storage unit 50 and the heartbeat data sequence newly output from the error correction unit 48. For example, if the unit time of the heartbeat data sequence stored in the storage unit 50 is a first unit time and the unit time of the heartbeat data sequence newly output from the error correction unit 48 is a second unit time, the second unit time is a unit time that comes later than the first unit time. In this case, the phase difference calculation unit 52 calculates the phase difference between the heartbeat data sequence of the first unit time and the heartbeat data sequence of the second unit time.

[0047] The phase difference calculation unit 52 determines, for each unit time, whether the phase difference between the heartbeat data sequence for the first unit time stored in the storage unit 50 and the heartbeat data sequence for the first unit time newly output from the error correction unit 48 is equal to or greater than a predetermined value. The phase difference calculation unit 52 outputs the determination result to the error output unit 54 and the output unit 56 for each unit time.

[0048] The error output unit 54 determines, for each unit time, whether or not it has received an error notification from the valid data extraction unit 46, indicating that the number of valid vertex data included in the set of extracted valid vertex data is less than a predetermined number. When the error output unit 54 receives an error notification from the valid data extraction unit 46, it outputs error information to the display device 26, indicating that heart rate information could not be detected during that unit time.

[0049] Furthermore, the error output unit 54 obtains, for each unit time, from the phase difference calculation unit 52, a determination result as to whether or not the phase difference is equal to or greater than a predetermined value. If the phase difference is equal to or greater than the predetermined value, the error output unit 54 outputs error information indicating that heartbeat information could not be detected for the first unit time to the display device 26. That is, if the phase difference is equal to or greater than the predetermined value, the error output unit 54 outputs error information indicating that heartbeat information could not be detected for the heartbeat data sequence stored in the storage unit 50 to the display device 26.

[0050] The output unit 56 generates heartbeat information based on the heartbeat data sequence stored in the storage unit 50 and outputs the generated heartbeat information to the display device 26. The output unit 56 may calculate at least one of the heart rate and the heartbeat period based on the heartbeat data sequence stored in the storage unit 50 and output the calculated at least one of the heart rate and the heartbeat period as heartbeat information. Alternatively, the output unit 56 may output the heartbeat data sequence stored in the storage unit 50 as heartbeat information.

[0051] In this embodiment, the output unit 56 obtains, for each unit time, from the phase difference calculation unit 52, a determination result as to whether the phase difference is equal to or greater than a predetermined value. If, for each unit time, the phase difference is not equal to or greater than the predetermined value, that is, if the phase difference is smaller than the predetermined value, the output unit 56 outputs heartbeat information based on the heartbeat data sequence for the first unit time that is stored in the storage unit 50. Furthermore, if, for each unit time, the phase difference is equal to or greater than the predetermined value, the output unit 56 deletes the heartbeat data sequence for the first unit time that is stored in the storage unit 50, and does not output the heartbeat information.

[0052] An example of the processing flow in the phase difference calculation section 52 and the output section 56 will be described later with reference to FIG.

[0053] FIG. 3 is a diagram showing a first example of the configuration of the detection signal generating unit 38. As shown in FIG.

[0054] For example, the detection signal generation section 38 may have a configuration as shown in Fig. 3. The detection signal generation section 38 according to the first example includes a first detection section 62 and a second detection section 64.

[0055] The first detection unit 62 acquires the moving image signal from the acquisition unit 34 and acquires information identifying the first partial region from the region identification unit 36. Based on the information identifying the first partial region, the first detection unit 62 detects, from the moving image signal, a signal component of the first partial region in which a heartbeat component in an exposed skin region can be easily detected. Then, the first detection unit 62 outputs a signal based on the signal component of the first partial region as a first detection signal.

[0056] The second detection unit 64 acquires the moving image signal from the acquisition unit 34 and acquires information identifying the second partial region from the region identification unit 36. Based on the information identifying the second partial region, the second detection unit 64 detects, from the moving image signal, a signal component of the second partial region in which it is more difficult to detect a heartbeat component than in the first partial region in the exposed skin region. Then, the second detection unit 64 outputs a signal based on the signal component of the second partial region as a second detection signal.

[0057] The first detection signal and the second detection signal both contain noise present in the exposed skin area, but the levels of the heartbeat components contained therein are different. Therefore, the detection signal generating unit 38 according to this first example can convert the differential signal calculated by the differential signal generating unit 40 at the subsequent stage into a signal in which the noise present in phase with both the first detection signal and the second detection signal has been removed, thereby enhancing the heartbeat component and making it easier to detect the heartbeat component.

[0058] FIG. 4 is a diagram showing a second example of the configuration of the detection signal generating unit 38. As shown in FIG.

[0059] For example, the detection signal generating section 38 may have a configuration as shown in Fig. 4. The detection signal generating section 38 according to the second example includes a first detecting section 62, a component generating section 66, and a separating section 68.

[0060] The first detection unit 62 executes the same processing as in Example 1. The component generation unit 66 generates an RGB signal representing the signal component of the first partial region.

[0061] The separation unit 68 outputs a signal based on a first component of the three components that make up the RGB signal of the first partial region as a first detection signal, and outputs a signal based on a second component that is different from the first component of the three components that make up the RGB signal of the first partial region as a second detection signal.

[0062] For example, the separator 68 may output the brightness of the G component (green component) of the first partial region multiplied by a predetermined coefficient (α) as the first detection signal, and the brightness of the B component (blue component) of the first partial region as the second detection signal. Note that the predetermined coefficient (α) is an arbitrary positive real number. In other words, in this case, the first detection signal represents the brightness of the G component in the first partial region multiplied by the predetermined coefficient. The second detection signal represents the brightness of the B component in the first partial region. Note that the brightness represents a value equivalent to the average light amount of the corresponding component.

[0063] Alternatively, the separator 68 may output the brightness of the G component of the first partial region multiplied by a predetermined coefficient (α) as the first detection signal, and the brightness of the R component (red component) of the first partial region as the second detection signal. In other words, in this case, the first detection signal represents the brightness of the G component of the first partial region multiplied by the predetermined coefficient, and the second detection signal represents the brightness of the R component of the first partial region.

[0064] The G component contains more heartbeat components than the B and R components. Therefore, the detection signal generating section 38 according to the second example can convert the differential signal calculated in the differential signal generating section 40 at the subsequent stage into a signal in which noise contained in phase with both the first detection signal and the second detection signal is removed and the heartbeat component is emphasized, making it easier to detect the heartbeat component.

[0065] The component generating section 66 may also perform HSV conversion on the RGB signal representing the signal component of the first partial region to generate an HSV signal representing the signal component of the first partial region.

[0066] In this case, the separation unit 68 outputs a signal based on the first component of the three components that make up the hue signal of the HSV signal as the first detection signal, and outputs a signal based on a second component that is different from the first component of the three components that make up the hue signal of the HSV signal as the second detection signal.

[0067] For example, the separator 68 may output, as the first detection signal, an area obtained by multiplying the total area of ​​the hue range of the HB component in the hue signal included in the HSV signal of the first partial region by a predetermined coefficient (α), and output, as the second detection signal, the total area of ​​the hue range of the HG component in the hue signal included in the HSV signal of the first partial region. In other words, in this case, the first detection signal represents the area obtained by multiplying the total area of ​​the hue range of the HB component in the hue signal included in the HSV signal of the first partial region by the predetermined coefficient (α). The second detection signal represents the total area of ​​the hue range of the HG component in the hue signal included in the HSV signal of the first partial region.

[0068] For example, the separator 68 may output, as the first detection signal, an area obtained by multiplying the total area of ​​the hue range of the HB component in the hue signal included in the HSV signal of the first partial region by a predetermined coefficient (α), and output, as the second detection signal, the total area of ​​the hue range of the HR component in the hue signal included in the HSV signal of the first partial region. In other words, in this case, the first detection signal represents the area obtained by multiplying the total area of ​​the hue range of the HB component in the hue signal included in the HSV signal of the first partial region by the predetermined coefficient (α). The second detection signal represents the total area of ​​the hue range of the HR component in the hue signal included in the HSV signal of the first partial region.

[0069] The total area of ​​the hue range of the HG component represents the total area of ​​the range of 120°±a in the hue signal. The total area of ​​the hue range of the HR component represents the total area of ​​the range of 360°±a in the hue signal. The total area of ​​the hue range of the HB component represents the total area of ​​the range of 240°±a in the hue signal. a is a value of 60° or less.

[0070] The HB component contains more heartbeat components than the HG component and the HR component. Therefore, the detection signal generating section 38 according to the second example can convert the differential signal calculated by the differential signal generating section 40 in the subsequent stage into a signal in which noise contained in phase with both the first detection signal and the second detection signal is removed and the heartbeat component is emphasized, making it easier to detect the heartbeat component.

[0071] FIG. 5 is a diagram showing a third example of the configuration of the detection signal generating unit 38. In FIG.

[0072] For example, the detection signal generation unit 38 may have a configuration as shown in Fig. 5. The detection signal generation unit 38 according to the third example includes a first detection unit 62, a first component generation unit 72, a first separation unit 74, a second detection unit 64, a second component generation unit 76, and a second separation unit 78.

[0073] The first detection unit 62 performs the same processing as in the first example. The first component generation unit 72 generates a first RGB signal representing the signal component of the first partial region. The first separation unit 74 outputs a signal based on the first component of the three components that make up the first RGB signal of the first partial region as the first detection signal. For example, the first separation unit 74 outputs the brightness of the G component (green component) of the first partial region as the first detection signal.

[0074] The second detection unit 64 performs the same processing as in the first example. The second component generation unit 76 generates a second RGB signal representing the signal component of the second partial region. The second separation unit 78 outputs, as the second detection signal, a signal based on the first component, which is the same as the component separated by the first separation unit 74, out of the three components constituting the second RGB signal of the second partial region. For example, the second separation unit 78 outputs the brightness of the G component (green component) of the second partial region as the second detection signal.

[0075] The first component generation unit 72 may also perform HSV conversion on the first RGB signal representing the signal component of the first partial region to generate a first HSV signal representing the signal component of the first partial region. In this case, the second component generation unit 76 performs HSV conversion on the second RGB signal representing the signal component of the second partial region to generate a second HSV signal representing the signal component of the second partial region. In this case, for example, the first separation unit 74 outputs the total area of ​​the hue range of the HB component in the hue signal of the first partial region as the first detection signal. Also, for example, the second separation unit 78 outputs the total area of ​​the hue range of the HB component in the hue signal of the second partial region as the second detection signal.

[0076] The detection signal generating section 38 according to this third example can convert the differential signal calculated in the subsequent differential signal generating section 40 into a signal in which noise contained in phase in both the first detection signal and the second detection signal is removed and the heartbeat component is emphasized, making it easier to detect the heartbeat component.

[0077] Fig. 6 is a flowchart showing an example of the processing flow of the heartbeat signal generating unit 42. Fig. 7 is a diagram for explaining the processing of joining vertices in the heartbeat signal generating unit 42. The heartbeat signal generating unit 42 executes the processing, for example, according to the flow shown in Fig. 6.

[0078] First, in S11, the heartbeat signal generator 42 generates a moving average signal by taking a moving average of the differential signal, thereby enabling the heartbeat signal generator 42 to generate a signal with reduced high-frequency noise and the like.

[0079] Next, in S12, the heart rate signal generating unit 42 detects vertices, which are peak points or bottom points, from the moving average signal, and generates a vertex data string that indicates the level and time of the vertex.

[0080] Next, in S13, the heartbeat signal generation unit 42 performs first-order differentiation on the vertex data sequence to generate a first-order differentiated data sequence. This allows the heartbeat signal generation unit 42 to generate a data sequence from which DC components have been removed. Note that the heartbeat signal generation unit 42 may perform other processing, such as second-order differentiation, instead of first-order differentiation to remove DC components.

[0081] Next, in S14, the heartbeat signal generating unit 42 detects peaks in the signal representing the first-order differentiated data string.

[0082] Next, in S15, the heartbeat signal generating unit 42 generates a signal by connecting the vertices detected in S14. For example, as shown in Fig. 7, the heartbeat signal generating unit 42 generates a signal by connecting the vertices with straight lines. Instead of using straight lines, the heartbeat signal generating unit 42 may generate a signal by smoothly connecting the vertices using a predetermined function.

[0083] Next, in S16, the heartbeat signal generating unit 42 performs band-pass filtering on the signal obtained by joining the peaks.

[0084] Next, in S17, the heartbeat signal generating unit 42 removes the DC component from the band-pass filtered signal by transversal filtering.

[0085] By performing the above-described processes from S11 to S17, the heartbeat signal generating unit 42 can generate a heartbeat signal from which noise and the like have been removed from the differential signal.

[0086] 8 is a flowchart showing an example of the processing flow of the vertex data generation unit 44. The vertex data generation unit 44 executes processing according to the flow shown in FIG.

[0087] First, in S21, the vertex data generation unit 44 divides the heartbeat signal into unit times, for example, time S. Then, the vertex data generation unit 44 executes the loop process between S22 and S24 for each unit time.

[0088] In the loop processing for each unit time, in S23, the vertex data generation unit 44 detects vertices, which are peak points or bottom points, from the heartbeat signal. Then, the vertex data generation unit 44 generates a plurality of vertex data, each of which represents the time and level of the vertex. The vertex data generation unit 44 repeatedly executes the loop processing between S22 and S24 until measurement of the heartbeat of the subject is completed.

[0089] By executing the above-described processes from S21 to S24, the vertex data generating unit 44 can generate a plurality of vertex data from the heartbeat signal for each unit time.

[0090] Fig. 9 is a flowchart showing an example of the processing flow of valid data extraction unit 46. Fig. 10 is a diagram showing a first example for explaining the processing for extracting a plurality of sets of valid vertex data. Fig. 11 is a diagram showing a second example for explaining the processing for extracting a plurality of sets of valid vertex data.

[0091] The valid data extraction unit 46 executes the process according to the flow shown in FIG. 9, for example.

[0092] First, the valid data extraction unit 46 repeatedly executes the loop process between S31 and S39 for each unit time.

[0093] In the loop processing between S31 and S39, first, in S32, the valid data extraction unit 46 determines whether or not there is a plurality of vertex data for the next unit time. If there is not a plurality of vertex data for the next unit time (No in S32), the valid data extraction unit 46 ends this flow. If there is a plurality of vertex data for the next unit time (Yes in S32), the valid data extraction unit 46 advances the processing to S33.

[0094] In S33, the valid data extraction unit 46 selects the N vertex data items with the highest amplitude from the plurality of vertex data items for the corresponding unit time generated by the vertex data generation unit 44. If the vertex is a peak point, the valid data extraction unit 46 selects the N vertex data items with the highest level. If the vertex is a bottom point, the valid data extraction unit 46 selects the N vertex data items with the lowest level.

[0095] Here, N is a predetermined integer equal to or greater than 2. As an example, N is a value that allows extraction of the maximum amount of valid vertex data that may include a heartbeat during the unit time S.

[0096] Next, in S34, the valid data extraction unit 46 extracts a set of valid vertex data that satisfies the periodicity of a heartbeat from the selected N pieces of vertex data.

[0097] For example, as shown in FIG. 10, the valid data extraction unit 46 calculates the time difference (T x Then, the valid data extraction unit 46 calculates the average period (T xsn ) (n is an integer equal to or greater than 1) plus or minus a predetermined margin. In this case, the valid data extraction unit 46 extracts a plurality of sets of valid vertex data so that the range of the valid vertex data is within the range of n times (n is an integer equal to or greater than 1) the average period (T xsn ) is within the range of the human heartbeat cycle. This allows the valid data extraction unit 46 to extract, from the selected N pieces of vertex data, a set of valid vertex data that satisfies the periodicity of the heartbeat.

[0098] 11, the valid data extraction unit 46 may extract a plurality of valid vertex data sets that are missing valid vertex data that represent the times of some cardiac cycles. That is, the valid data extraction unit 46 may extract a plurality of valid vertex data sets that include blank cardiac cycles that do not include valid vertex data.

[0099] Subsequently, in S35, the valid data extraction unit 46 determines whether the number of valid vertex data included in the set of multiple valid vertex data is M or more.

[0100] Note that M is a predetermined integer equal to or greater than 2 and smaller than N. In this embodiment, M is N / 2. As an example, M is a value obtained by subtracting a predetermined margin from the minimum value at which a heartbeat may be included in the unit time S.

[0101] If the number of valid vertex data included in the set of multiple valid vertex data is not equal to or greater than M (=N / 2) (No in S35), the valid data extraction unit 46 advances the process to S36.

[0102] In S36, the valid data extraction unit 46 discards the data for the current unit time. That is, the valid data extraction unit 46 deletes the set of multiple valid vertex data for the current unit time, and ends the subsequent processing for that unit time.

[0103] Next, in S37, the valid data extraction unit 46 outputs an error notification indicating that the number of valid vertex data included in the set of multiple valid vertex data is less than M (=N / 2) to the error output unit 54. This allows the error output unit 54 to output error information indicating that heartbeat information could not be detected in the current unit time. After completing S37, the valid data extraction unit 46 repeats the process from S32 for the next unit time.

[0104] If the number of valid vertex data included in the set of valid vertex data is M (=N / 2) or more (Yes in S35), the valid data extraction unit 46 advances the process to S38.

[0105] In S38, the valid data extraction unit 46 calculates the average period of the heartbeats represented by the sets of valid vertex data. For example, the valid data extraction unit 46 may acquire the average period used to extract the sets of valid vertex data, or may calculate the average period from the sets of valid vertex data after extraction. After completing S38, the valid data extraction unit 46 repeats the process from S32 for the next unit time.

[0106] By performing the above processes from S31 to S39, the valid data extraction unit 46 can extract, from the plurality of vertex data, a set of multiple valid vertex data that satisfies the periodicity of a heartbeat. Furthermore, if the number of valid vertex data included in the extracted set of multiple valid vertex data is smaller than a predetermined number, the valid data extraction unit 46 can output error information indicating that heartbeat information could not be detected in the current unit time to the error output unit 54. Furthermore, if the number of valid vertex data included in the extracted set of multiple valid vertex data is smaller than a predetermined number, the valid data extraction unit 46 can terminate subsequent processing for the current unit time.

[0107] FIG. 12 is a flowchart showing an example of the flow of processing by the error correction unit 48.

[0108] The error correction unit 48 executes the process according to the flow shown in FIG. 12, for example.

[0109] First, the error correction unit 48 repeatedly executes the loop process between S41 and S49 for each unit time.

[0110] In the loop processing between S41 and S49, first, in S42, the error correction unit 48 determines whether or not there are multiple sets of valid vertex data for the current unit time. That is, the error correction unit 48 determines whether or not multiple sets of valid vertex data for the current unit time have been output from the valid data extraction unit 46. If there are not multiple sets of valid vertex data for the current unit time (No in S42), the error correction unit 48 repeats the processing from S42 for the next unit time. If there are multiple sets of valid vertex data for the current unit time (Yes in S42), the error correction unit 48 proceeds with the processing to the loop processing between S43 and S47.

[0111] In the loop processing between S43 and S47, the error correction unit 48 repeatedly executes the processing for each pair of valid vertex data that are adjacent in time and are included in a set of multiple valid vertex data. Two pairs of valid vertex data that are adjacent in time are pairs of valid vertex data that have no other valid vertex data between them.

[0112] In the loop process between S43 and S47, first, in S44, the error correction unit 48 calculates the time difference between two pieces of valid vertex data that are adjacent in time to be processed.

[0113] Next, in S45, the error correction unit 48 determines whether the time difference is within a first time range. The first time range ranges from a lower limit obtained by reducing a predetermined first margin amount (α1) by twice the average period to an upper limit obtained by increasing a predetermined second margin amount (α2) by twice the average period. More specifically, the first time range is a time range that is equal to or greater than (average period × 2 - α1) and equal to or less than (average period × 2 + α2). Note that the first margin amount (α1) and the second margin amount (α2) are values ​​equal to or greater than the maximum value of human heartbeat fluctuation. The first margin amount (α1) and the second margin amount (α2) may be the same. By executing the process of S45, the error correction unit 48 can detect the position of the missing valid vertex data.

[0114] If the time difference is not within the first time range (No in S45), the error correction unit 48 returns the process to S44 and proceeds with the process on the two valid vertex data whose times are adjacent to each other and which will be the next processing targets. If the time difference is within the first time range (Yes in S45), the error correction unit 48 proceeds with the process to S46.

[0115] In S46, the error correction unit 48 generates new valid vertex data that represents the midpoint time of the two valid vertex data that are adjacent in time to be processed. By executing the process of S46, the error correction unit 48 can generate new valid vertex data that represents the missing valid vertex data. After completing the process of S46, the error correction unit 48 returns the process to S44 and proceeds with the process on the next two valid vertex data that are adjacent in time to be processed.

[0116] When the error correction unit 48 has completed the processes from S44 to S46 for all two pieces of valid vertex data that are adjacent in time, the process proceeds to S48.

[0117] In S48, the error correction unit 48 generates a heartbeat data sequence by adding all of the generated new valid vertex data to a plurality of sets of valid vertex data, and then outputs the generated heartbeat data sequence.

[0118] By performing the above processes from S41 to S49, when valid vertex data representing some cardiac cycle times is missing from a set of multiple valid vertex data, the error correction unit 48 can generate new valid vertex data representing the missing valid vertex data by interpolating from the sets of multiple valid vertex data. Then, the error correction unit 48 can generate a cardiac data sequence in which the new valid vertex data generated by interpolation is added to the sets of multiple valid vertex data.

[0119] FIG. 13 is a flowchart showing an example of the processing flow of the phase difference calculation section 52 and the output section 56.

[0120] The phase difference calculation section 52 and the output section 56 execute the process according to the flow shown in FIG. 13, for example.

[0121] First, the phase difference calculation unit 52 and the output unit 56 repeatedly execute the loop process between S51 and S60 for each unit time.

[0122] In the loop process between S51 and S60, first, in S52, the phase difference calculation unit 52 determines whether or not a heartbeat data sequence for the current unit time exists. That is, the phase difference calculation unit 52 determines whether or not a heartbeat data sequence for the current unit time has been output from the error correction unit 48. If a heartbeat data sequence for the current unit time does not exist (No in S52), the phase difference calculation unit 52 repeats the process from S52 for the next unit time. If a heartbeat data sequence for the current unit time exists (Yes in S52), the phase difference calculation unit 52 proceeds to S53.

[0123] In S53, the phase difference calculation unit 52 determines whether or not there is a heartbeat data sequence for a past unit time stored in the storage unit 50. If there is no heartbeat data sequence for a past unit time stored in the storage unit 50 (No in S53), the phase difference calculation unit 52 proceeds to S59. If there is a heartbeat data sequence for a past unit time stored in the storage unit 50 (Yes in S53), the phase difference calculation unit 52 proceeds to S54.

[0124] In S54, the phase difference calculation unit 52 calculates, for each unit time, the phase difference between the heartbeat data sequence for the past unit time stored in the memory unit 50 and the heartbeat data sequence for the current unit time output from the error correction unit 48.

[0125] For example, the phase difference calculation unit 52 calculates the time difference between the latest valid vertex data in the heartbeat data sequence for the past unit time stored in the storage unit 50, and the earliest valid vertex data in the heartbeat data sequence for the current unit time output from the error correction unit 48, that is, the time of the oldest valid vertex data. Next, the phase difference calculation unit 52 calculates the remainder of the calculated time difference and the average period of the heartbeat data sequence for the current unit time output from the error correction unit 48. Then, the phase difference calculation unit 52 calculates the phase difference based on the calculated remainder and the average period of the heartbeat data sequence. For example, the phase difference calculation unit 52 calculates the phase difference based on the ratio of the remainder to the average period.

[0126] Subsequently, in S55, the phase difference calculation unit 52 determines whether the calculated phase difference is equal to or greater than a predetermined value (β). If the phase difference is equal to or greater than the predetermined value (β) (Yes in S55), the phase difference calculation unit 52 proceeds to S56.

[0127] In S56, the output unit 56 deletes the heartbeat data sequence for the past unit time stored in the storage unit 50 from the storage unit 50.

[0128] Subsequently, in S57, the phase difference calculation unit 52 outputs an error notification indicating that the phase difference of the heartbeat data sequence for the past unit time stored in the storage unit 50 is equal to or greater than a predetermined value (β) to the error output unit 54. This enables the error output unit 54 to output error information indicating that heartbeat information could not be detected for the past unit time stored in the storage unit 50. After completing S57, the phase difference calculation unit 52 advances the process to S59.

[0129] If the phase difference is not equal to or greater than the predetermined value (β) (No in S55), the phase difference calculation unit 52 advances the process to S58.

[0130] In S58, the output unit 56 calculates and outputs heartbeat information based on the heartbeat data sequence for a past unit time stored in the storage unit 50. For example, the output unit 56 calculates the heart rate based on the heartbeat data sequence and outputs it as heartbeat information. Alternatively, for example, the output unit 56 calculates the heartbeat period based on the heartbeat data sequence and outputs it as heartbeat information. Alternatively, for example, the output unit 56 may output the heartbeat data sequence as is as heartbeat information. After completing the process of S58, the output unit 56 proceeds to S59.

[0131] In S59, the output unit 56 stores the heartbeat data string for the current unit time output from the error correction unit 48 in the storage unit 50 as the next processing target.

[0132] After completing the process of S59, the phase difference calculation section 52 and the output section 56 repeat the process for the next unit time from S52.

[0133] By executing the above processes from S51 to S60, the phase difference calculation unit 52 and the output unit 56 can output heartbeat information for each unit time. Furthermore, if the phase difference between the heartbeat data sequence for the past unit time stored in the storage unit 50 and the heartbeat data sequence for the current unit time output from the error correction unit 48 is equal to or greater than a predetermined value, the phase difference calculation unit 52 and the output unit 56 delete the heartbeat data sequence for the past unit time stored in the storage unit 50 and cause the error output unit 54 to output error information.

[0134] As described above, the heartbeat detection system 10 according to the first embodiment generates a first detection signal including a heartbeat component of the subject and a second detection signal including a smaller heartbeat component than the first detection signal, based on signal components of an exposed skin region of the subject in a moving image signal obtained by capturing an image of the subject.The heartbeat detection system 10 then generates a heartbeat signal based on a differential signal representing the difference between the first detection signal and the second detection signal.

[0135] As a result, the heartbeat detection system 10 according to the first embodiment can remove noise from the moving image signal obtained by capturing an image of the subject, and can detect heartbeat information with high accuracy.

[0136] Furthermore, the heartbeat detection system 10 according to the first embodiment extracts, from the heartbeat signal, a plurality of sets of valid vertex data that satisfy the periodicity of the heartbeat. If a set of the plurality of valid vertex data is missing valid vertex data that represents a part of the time of the heartbeat cycle, the heartbeat detection system 10 generates new valid vertex data representing the missing valid vertex data by interpolating from the plurality of sets of valid vertex data, and generates a heartbeat data sequence in which the new valid vertex data is added to the plurality of sets of valid vertex data.

[0137] As a result, the heartbeat detection system 10 according to the first embodiment can detect heartbeat information even if the heartbeat signal component contained in the differential signal is weak.

[0138] Furthermore, the heartbeat detection system 10 according to the first embodiment outputs error information when it is not possible to extract a set of multiple valid peak data that satisfies the periodicity of a heartbeat from the heartbeat signal, and when the phase difference between the heartbeat data sequence of the first unit time and the heartbeat data sequence of the second unit time preceding the first unit time is equal to or greater than a predetermined value.

[0139] As a result, when the heartbeat information cannot be detected, the heartbeat detection system 10 according to the first embodiment can make the person taking the measurement or the like aware that the heartbeat information cannot be detected.

[0140] (Second embodiment) Next, a heartbeat detection system 10 according to a second embodiment will be described. The heartbeat detection system 10 according to the second embodiment has substantially the same functions and configuration as the heartbeat detection system 10 according to the first embodiment described with reference to Figs. 1 to 13. Therefore, in the heartbeat detection system 10 according to the second embodiment, components having substantially the same functions and configurations as those in the first embodiment are denoted by the same reference numerals, and detailed description thereof will be omitted except for the differences.

[0141] FIG. 14 is a diagram showing the configuration of a heartbeat detecting device 24 according to the second embodiment.

[0142] The detection signal generator 38 according to the second embodiment generates P types of pairs of first and second detection signals, where P is an integer of 2 or greater.

[0143] Each of the P sets of first and second detection signals has at least one of the first and second detection signals different from the other sets. For example, the detection signal generator 38 calculates each of the P sets of first and second detection signals using any of the configurations described with reference to Figures 3 to 5. In this case, each of the P sets of first and second detection signals is calculated using a configuration different from the other sets.

[0144] The differential signal generating circuit 40 according to the second embodiment generates a differential signal for each of P pairs of first and second detection signals. Therefore, the differential signal generating circuit 40 generates P differential signals. Each of the P differential signals differs from the other differential signals in at least one of the first and second detection signals.

[0145] The heartbeat signal generating section 42 according to the second embodiment generates a heartbeat signal for each of the P differential signals. Therefore, the heartbeat signal generating section 42 generates P heartbeat signals.

[0146] The vertex data generating section 44 according to the second embodiment generates a plurality of vertex data for each of the P heartbeat signals for each unit time.

[0147] The valid data extracting section 46 according to the second embodiment extracts a plurality of sets of valid vertex data based on all of the plurality of vertex data of the P heartbeat signals for each unit time.

[0148] Fig. 15 is a flowchart showing an example of the flow of processing by the valid data extraction unit 46 according to the second embodiment. Fig. 16 is a diagram for explaining processing for extracting a plurality of sets of valid vertex data.

[0149] In the second embodiment, the valid data extraction unit 46 executes processing according to the flow shown in FIG. 15, for example.

[0150] First, the valid data extraction unit 46 repeatedly executes the loop process between S71 and S83 for each unit time.

[0151] In the loop processing between S71 and S83, first, in S72, the valid data extraction unit 46 determines whether or not multiple vertex data for the next unit time exists. If multiple vertex data for the next unit time does not exist (No in S72), the valid data extraction unit 46 ends this flow. If multiple vertex data for the next unit time exists (Yes in S72), the valid data extraction unit 46 advances the processing to the loop processing between S73 and S76.

[0152] The valid data extraction unit 46 executes the loop process between S73 and S76 for each of the P heartbeat signals.

[0153] In the loop process between S73 and S76, first, in S74, the valid data extraction unit 46 selects the top N vertex data in descending order of amplitude from the plurality of vertex data detected from the target heartbeat signal.

[0154] Subsequently, in S75, the valid data extraction unit 46 extracts a plurality of sets of intermediate vertex data that satisfy the periodicity of a heartbeat from the selected N pieces of vertex data.

[0155] When the loop process between S73 and S76 is completed for all P heartbeat signals, the valid data extraction unit 46 advances the process to S77.

[0156] In S77, the valid data extraction unit 46 selects a plurality of sets of Q or more intermediate peak data whose time difference is within a predetermined value from the plurality of intermediate peak data for each of the P heartbeat signals. In this embodiment, Q is an integer equal to or greater than P / 2.

[0157] For example, as shown in Fig. 16, suppose that the heartbeat signal generator 42 generates five types of heartbeat signals based on five types of difference signals. In such a case, the valid data extractor 46 selects a set of three or more intermediate vertex data whose time difference is within a predetermined value. The predetermined value represents the error range within which the heartbeat timings can be considered to be synchronized.

[0158] Next, in S78, for each of the selected sets, the valid data extraction unit 46 generates one valid vertex data item from among the valid vertex data items based on the Q or more pieces of intermediate vertex data items. For example, as shown in FIG. 16 , the valid data extraction unit 46 generates valid vertex data for each set by averaging the Q or more pieces of valid vertex data items included in that set.

[0159] Subsequently, in S79, the valid data extraction unit 46 determines whether the number of valid vertex data included in the set of multiple valid vertex data is M or more.

[0160] If the number of valid vertex data included in the set of valid vertex data is not equal to or greater than M (=N / 2) (No in S79), the valid data extraction unit 46 advances the process to S80.

[0161] In S80, the valid data extraction unit 46 deletes the set of valid vertex data for the current unit time. Subsequently, in S81, the valid data extraction unit 46 outputs an error notification indicating that the number of valid vertex data included in the set of valid vertex data is less than M (=N / 2) to the error output unit 54. This allows the error output unit 54 to output error information indicating that heartbeat information could not be detected in the current unit time. After completing S81, the valid data extraction unit 46 repeats the process from S72 for the next unit time.

[0162] If the number of valid vertex data included in the set of valid vertex data is M (=N / 2) or more (Yes in S79), the valid data extraction unit 46 advances the process to S82.

[0163] In S82, the valid data extraction unit 46 calculates the average period of the heartbeat represented by a set of multiple valid vertex data. After completing S82, the valid data extraction unit 46 repeats the process from S72 for the next unit time.

[0164] By performing the above processes from S71 to S83, the valid data extraction unit 46 can extract, from the plurality of vertex data, a set of multiple valid vertex data that satisfies the periodicity of a heartbeat. Furthermore, if the number of valid vertex data included in the extracted set of multiple valid vertex data is smaller than a predetermined number, the valid data extraction unit 46 can output error information indicating that heartbeat information could not be detected in the current unit time to the error output unit 54. Furthermore, if the number of valid vertex data included in the extracted set of multiple valid vertex data is smaller than a predetermined number, the valid data extraction unit 46 can terminate subsequent processing for the current unit time.

[0165] As described above, the heartbeat detection system 10 according to the second embodiment, like the first embodiment, can remove noise from a moving image signal capturing an image of the subject and detect heartbeat information with high accuracy. Also, like the first embodiment, the heartbeat detection system 10 according to the second embodiment can detect heartbeat information even if the heartbeat signal component included in the differential signal is weak. Also, like the first embodiment, the heartbeat detection system 10 according to the second embodiment can inform the subject or the like that heartbeat information cannot be detected when heartbeat information cannot be detected. Furthermore, the heartbeat detection system 10 according to the second embodiment uses P differential signals, and therefore can detect heartbeat information with higher accuracy.

[0166] Fig. 17 is a diagram showing the hardware configuration of an information processing device. As an example, the heartbeat detection device 24 is realized by a device having the same hardware configuration as a general information processing device as shown in Fig. 17. The information processing device includes a CPU (Central Processing Unit) 301, an operation device 302, a display device 303, a main memory device 305, an auxiliary memory device 306, a communication device 307, and a bus 309. Each unit is connected by the bus 309.

[0167] The CPU 301 executes various processes in cooperation with various programs stored in advance in the auxiliary storage device 306, etc., using a predetermined area of ​​the main storage device 305 as a working area, and comprehensively controls the operation of each component constituting the heartbeat detection device 24. The CPU 301 also operates the operation device 302, the display device 303, the communication device 307, etc. in cooperation with the programs.

[0168] The operation device 302 is an input device such as a touch panel, a mouse, or a keyboard, and receives information input by a user as an instruction signal, and outputs the instruction signal to the CPU 301. The display device 303 displays various information based on a display signal from the CPU 301.

[0169] The main storage device 305 is a volatile storage medium such as a Synchronous Dynamic Random Access Memory (SDRAM), etc. The main storage device 305 functions as a work area for the CPU 301.

[0170] The auxiliary storage device 306 is a rewritable storage device such as a semiconductor storage medium such as a flash memory, or a magnetically or optically recordable storage medium. The auxiliary storage device 306 stores programs used for control. The communication device 307 transmits and receives data to and from other devices.

[0171] The program executed by the heartbeat detection device 24 may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Alternatively, the program executed by the heartbeat detection device 24 may be provided in advance in a portable storage medium or the like.

[0172] The program executed by the heartbeat detection device 24 has a modular configuration including a light-emission control module, an acquisition module, a region identification module, a detection signal generation module, a difference signal generation module, a heartbeat signal generation module, a vertex data generation module, a valid data extraction module, an error correction module, a phase difference calculation module, an error output module, and an output module. The CPU 301 reads such program from a storage medium or the like and loads each of the above modules into the main memory device 305. By executing such program, the CPU 301 functions as the light-emission control unit 32, the acquisition unit 34, the region identification unit 36, the detection signal generation unit 38, the difference signal generation unit 40, the heartbeat signal generation unit 42, the vertex data generation unit 44, the valid data extraction unit 46, the error correction unit 48, the phase difference calculation unit 52, the error output unit 54, and the output unit 56. By executing such program, the CPU 301 also causes the main memory device 305 or the auxiliary memory device 306 to function as the storage unit 50. In addition, some or all of the light emission control unit 32, acquisition unit 34, area identification unit 36, detection signal generation unit 38, difference signal generation unit 40, heartbeat signal generation unit 42, vertex data generation unit 44, valid data extraction unit 46, error correction unit 48, phase difference calculation unit 52, error output unit 54, and output unit 56 may be configured using hardware.

[0173] Although the embodiments of the present invention have been described above, these embodiments are presented as examples and are not intended to limit the scope of the invention. Various modifications can be made to the embodiments. [Explanation of symbols]

[0174] 10 heartbeat detection system, 20 camera, 22 light emitting device, 24 heartbeat detection device, 26 display device, 32 light emitting control unit, 34 acquisition unit, 36 area identification unit, 38 detection signal generation unit, 40 difference signal generation unit, 42 heartbeat signal generation unit, 44 vertex data generation unit, 46 valid data extraction unit, 48 error correction unit, 50 memory unit, 52 phase difference calculation unit, 54 error output unit, 56 output unit

Claims

1. a detection signal generating unit that generates a first detection signal including a heartbeat component of the subject and a second detection signal including a smaller amount of the heartbeat component than the first detection signal, based on a signal component of an exposed skin region of the subject in a moving image signal obtained by capturing an image of the subject; a differential signal generating unit that generates a differential signal representing a difference between the first detection signal and the second detection signal; a heartbeat signal generating unit that generates a heartbeat signal representing the heartbeat component based on the differential signal; a vertex data generating unit that generates a plurality of vertex data representing times of vertices that are peak points or bottom points in the heartbeat signal; a valid data extraction unit that extracts a set of valid vertex data that satisfies the periodicity of a heartbeat from the plurality of vertex data; an error correction unit that, when a part of the sets of the plurality of valid vertex data is missing or erroneous, corrects the error in the missing or erroneous valid vertex data to generate the plurality of valid vertex data as a heartbeat data sequence; an output unit that outputs heartbeat information based on the heartbeat data sequence; A heart rate detection device comprising:

2. The detection signal generation unit a first detection unit that detects a signal component of a first partial region in which the heartbeat component in the exposed skin region is easily detected from the moving image signal, and outputs the signal component of the first partial region as the first detection signal; a second detection unit that detects, from the moving image signal, a signal component of a second partial region in which the heartbeat component is more difficult to detect than the first partial region in the exposed skin region, and outputs the signal component of the second partial region as the second detection signal; Contains 2. The heart rate detection device according to claim 1.

3. The detection signal generation unit a first detection unit that detects, from the moving image signal, a signal component of a first partial region in which the heartbeat component in the exposed skin region is easily detected; a component generation unit that generates an RGB signal representing a signal component of the first partial region or an HSV signal obtained by HSV-converting the RGB signal; a separation unit that outputs, as the first detection signal, a signal based on a first component of the three components that constitute the RGB signal or the three components that constitute a hue signal of the HSV signal, and outputs, as the second detection signal, a signal based on a second component that is different from the first component of the three components that constitute the RGB signal or the HSV signal; Contains 2. The heart rate detection device according to claim 1.

4. The detection signal generation unit a first detection unit that detects, from the moving image signal, a signal component of a first partial region in which the heartbeat component in the exposed skin region is easily detected; a first component generator that generates a first RGB signal representing a signal component of the first partial region, or a first HSV signal obtained by HSV-converting the first RGB signal; a first separation unit that outputs, as the first detection signal, a signal based on a first component of three components that constitute the first RGB signal or three components that constitute a hue signal of the first HSV signal; a second detection unit that detects, from the moving image signal, a signal component of a second partial region in the exposed skin region in which the heartbeat component is more difficult to detect than the first partial region; a second component generator configured to generate a second RGB signal representing a signal component of the second partial region or a second HSV signal obtained by HSV-converting the second RGB signal; a second separation unit that outputs, as the second detection signal, a signal based on the first component of three components that constitute the second RGB signal or three components that constitute a hue signal of the second HSV signal; Contains 2. The heart rate detection device according to claim 1.

5. the first detection signal represents an amount obtained by multiplying the brightness of the G component in the RGB signal by a predetermined coefficient; The second detection signal represents the brightness of the B component in the RGB signal.

4. The heart rate detection device according to claim 3.

6. the first detection signal represents an amount obtained by multiplying the brightness of the G component in the RGB signal by a predetermined coefficient; The second detection signal represents the brightness of the R component in the RGB signal.

4. The heart rate detection device according to claim 3.

7. the first detection signal represents an area obtained by multiplying a total area of ​​a hue range of an HB component in a hue signal included in the HSV signal by a predetermined coefficient; The second detection signal represents a total area of ​​the hue range of the HG component in the hue signal included in the HSV signal.

4. The heart rate detection device according to claim 3.

8. the first detection signal represents an area obtained by multiplying a total area of ​​a hue range of an HB component in a hue signal included in the HSV signal by a predetermined coefficient; The second detection signal represents a total area of ​​the hue range of the HR component in the hue signal included in the HSV signal.

4. The heart rate detection device according to claim 3.

9. When the set of the plurality of valid vertex data is missing valid vertex data that represents a part of the time of the cardiac cycle, the error correction unit generates new valid vertex data that represents the missing valid vertex data by interpolating from the set of the plurality of valid vertex data, and generates the cardiac data sequence in which the new valid vertex data is added to the set of the plurality of valid vertex data.

2. The heart rate detection device according to claim 1.

10. the vertex data generation unit generates the plurality of vertex data for each predetermined unit time; The valid data extraction unit, for each unit time, Selecting N vertex data (N is an integer equal to or greater than 2) having the largest amplitude from the plurality of vertex data; A set of the plurality of valid vertex data satisfying the periodicity of a heartbeat is extracted from the N pieces of vertex data.

10. The heart rate detection device according to claim 9.

11. If the number of valid vertex data included in the set of the plurality of valid vertex data per unit time is M or more (M is an integer equal to or greater than 2 and smaller than N), the valid data extraction unit calculates an average period of heartbeats represented by the plurality of sets of valid vertex data; When a time difference between two adjacent valid vertex data pieces included in the set of the plurality of valid vertex data pieces is within a first time range from a lower limit value obtained by reducing a predetermined first margin amount by twice the average period to an upper limit value obtained by increasing a predetermined second margin amount by twice the average period, the error correction unit generates the new valid vertex data piece representing the time between the two adjacent valid vertex data pieces, and generates the heartbeat data sequence by adding the new valid vertex data piece to the set of the plurality of valid vertex data pieces. The heart rate detection device according to claim 10.

12. M is N / 2 12. The heart rate detection device according to claim 11.

13. The method further includes an error output unit that outputs error information indicating that the heartbeat information could not be detected when the number of valid vertex data included in the set of valid vertex data is less than M.

13. The heart rate detection device according to claim 12.

14. a phase difference calculation unit that calculates a phase difference between the heartbeat data sequence for a first unit time that is the unit time and the heartbeat data sequence for a second unit time that is a unit time before the first unit time; an error output unit that outputs error information indicating that the heartbeat information could not be detected for the first unit time when the phase difference is equal to or greater than a predetermined value; Furthermore, When the phase difference is smaller than the predetermined value, the output unit outputs the heartbeat information based on the heartbeat data sequence for the first unit time.

13. The heart rate detection device according to claim 12.

15. the differential signal generation unit generates P differential signals (P is an integer equal to or greater than 2) each representing a difference between the first detection signal and the second detection signal; Each of the P differential signals is different from the other differential signals in that at least one of the first detection signal and the second detection signal is different from the other differential signals, the heartbeat signal generating unit generates P heartbeat signals corresponding to the P difference signals; each of the P cardiac signals is the cardiac signal generated based on the corresponding differential signal; the vertex data generation unit generates the plurality of vertex data for each of the P heartbeat signals; The valid data extraction unit extracts the plurality of valid vertex data sets based on the plurality of vertex data for each of the P heartbeat signals.

2. The heart rate detection device according to claim 1.

16. the vertex data generation unit generates the plurality of vertex data for each unit time for each of the P heartbeat signals; For each unit time, the valid data extraction unit performs the following for each of the P heartbeat signals: Selecting N vertex data (N is an integer equal to or greater than 2) having the largest amplitude from the plurality of vertex data; extracting a plurality of intermediate vertex data that satisfy the periodicity of a heartbeat from the N vertex data; The valid data extraction unit, for each unit time, selecting a plurality of sets of Q or more intermediate peak data whose time difference is within a predetermined value from the plurality of intermediate peak data for each of the P heartbeat signals; Q is an integer equal to or greater than P / 2, generating one valid vertex data item from the plurality of valid vertex data items based on the Q or more intermediate vertex data items for each of the selected plurality of sets; 16. The heart rate detection device according to claim 15.

17. a camera that captures an image of the subject and outputs the moving image signal; A heartbeat detection device according to any one of claims 1 to 16; A heart rate detection system comprising:

18. A heartbeat detection method for generating heartbeat information representing a heartbeat of a subject by an information processing device, comprising: The information processing device, generating a first detection signal including a heartbeat component of the subject and a second detection signal including a smaller amount of the heartbeat component than the first detection signal, based on a signal component of an exposed skin region of the subject in a moving image signal obtained by capturing an image of the subject; generating a differential signal representing a difference between the first detection signal and the second detection signal; generating a heartbeat signal representing the heartbeat component based on the difference signal; generating a plurality of apex data representing times of apexes, which are peak points or bottom points in the heartbeat signal; extracting a set of a plurality of valid vertex data that satisfy the periodicity of a heartbeat from the plurality of vertex data; If a part of the set of the plurality of effective vertex data is missing or erroneous, error correction is performed on the missing or erroneous effective vertex data to generate the plurality of effective vertex data as a heartbeat data string; outputting the heartbeat information based on the heartbeat data sequence; Heart rate detection method.

19. A program for causing a computer to function as a heartbeat detection device, The computer a detection signal generating unit that generates a first detection signal including a heartbeat component of the subject and a second detection signal including a smaller amount of the heartbeat component than the first detection signal, based on a signal component of an exposed skin region of the subject in a moving image signal obtained by capturing an image of the subject; a differential signal generating unit that generates a differential signal representing a difference between the first detection signal and the second detection signal; a heartbeat signal generating unit that generates a heartbeat signal representing the heartbeat component based on the differential signal; a vertex data generating unit that generates a plurality of vertex data representing times of vertices that are peak points or bottom points in the heartbeat signal; a valid data extraction unit that extracts a set of valid vertex data that satisfies the periodicity of a heartbeat from the plurality of vertex data; an error correction unit that, when the set of the plurality of valid vertex data is missing valid vertex data representing a part of the time of the cardiac cycle, generates new valid vertex data representing the missing valid vertex data by interpolating from the set of the plurality of valid vertex data, and generates a cardiac data sequence in which the new valid vertex data is added to the set of the plurality of valid vertex data; an error correction unit that, when a part of the sets of the plurality of valid vertex data is missing or erroneous, corrects the error in the missing or erroneous valid vertex data to generate the plurality of valid vertex data as a heartbeat data sequence; an output unit that outputs heartbeat information based on the heartbeat data sequence; A program that makes it work.

Images