Heartbeat detection system, heartbeat detection method, and program

Abstract

To accurately detect heartbeat information in a non-contact manner even when a subject is in the state of moving.SOLUTION: A heartbeat detection system includes: a sensor device that detects a sensor signal including a heartbeat component of a person to be measured without contacting the person to be measured; a person-to-be-measured detection camera that generates an image signal obtained by imaging the person to be measured; a person-to-be-measured position specification unit that specifies a person-to-be-measured position which is a position of the person to be measured from the image signal; a detection direction control unit that controls a detection direction of the sensor device according to the person-to-be-measured position; and a heartbeat detection device that generates a heartbeat signal indicating a heartbeat component on the basis of the sensor signal and outputs heartbeat information on the basis of the heartbeat signal.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a heartbeat detection system, a heartbeat detection method, and a program. [Background technology]

[0002] Society 5.0, one of Japan's science and technology policies, aims to utilize IoT (Internet of Things) technology in a society where everything is connected to the Internet, thereby creating new value through data analysis and achieving a comfortable, vibrant, and high-quality life, as well as economic development and the resolution of social issues. IoT technology is also beginning to be applied to the human field. In recent years, in order to realize Society 5.0, attention has been focused on technology that measures heart rate without contact, without the subject's awareness, which can lead to the analysis of human health and emotions. For example, Patent Document 1 describes a system that uses a camera to observe a user without contact. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6134607 Summary of the Invention [Problem to be solved by the invention]

[0004] A known method for detecting the heartbeat of a person being measured without contact is to use an image captured by a camera. Detecting the heartbeat from an image captured by a camera has the advantage of being easy to handle because it uses a camera. Another known method for detecting the heartbeat of a person being measured without contact is to use a distance signal measured by a millimeter-wave radar device. Detecting the heartbeat from a distance signal measured by a millimeter-wave radar device does not require capturing an image of the person being measured, so it has the advantage of being able to detect the heartbeat without identifying the individual and being able to take into consideration the invasion of the person's privacy.

[0005] However, in non-contact methods of detecting the heartbeat of a person being measured, the detection signal is very weak due to the principles of heartbeat detection, making it difficult to detect the heartbeat when the person being measured is walking or moving around.

[0006] The present invention has been made in consideration of the above, and aims to provide a heart rate detection system, a heart rate detection method, and a program that can detect heart rate information non-contact and accurately even when the person being measured is moving. [Means for solving the problem]

[0007] In order to solve the above-mentioned problems and achieve the object, the heart rate detection system of the present invention comprises a sensor device that detects a sensor signal containing a heart rate component of the subject without contacting the subject, a subject detection camera that generates an image signal by capturing an image of the subject, a subject position identification unit that identifies the subject's position from the image signal, which is the position of the subject, a detection direction control unit that controls the detection direction of the sensor device depending on the subject's position, and a heart rate detection device that generates a heart rate signal representing the heart rate component based on the sensor signal and outputs heart rate information based on the heart rate signal. [Effects of the Invention]

[0008] According to the present invention, heart rate information can be detected accurately in a non-contact manner even when the subject is moving. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing the configuration of a heartbeat detection system according to the first embodiment. [Figure 2] FIG. 2 is a diagram showing an example of a housing that houses a millimeter wave radar device. [Figure 3] FIG. 3 is a diagram showing the configuration of a heartbeat detection system according to the second embodiment. [Figure 4] FIG. 4 is a diagram showing the configuration of a heartbeat detection system according to the third embodiment. [Figure 5] FIG. 5 is a diagram showing the functional configuration of a heartbeat detecting device according to the fourth embodiment. [Figure 6] FIG. 6 is a flowchart showing an example of the flow of processing by the heartbeat signal generating unit. [Figure 7] FIG. 7 is a diagram for explaining the process of joining vertices in the heartbeat signal generating unit. [Figure 8] FIG. 8 is a flowchart showing an example of the flow of a process for generating vertex data. [Figure 9] FIG. 9 is a flowchart showing an example of the flow of processing by the valid data extraction unit. [Figure 10] FIG. 10 is a diagram showing a first example for explaining the process of extracting a plurality of sets of valid vertex data. [Figure 11] FIG. 11 is a diagram illustrating a second example for explaining the process of extracting a set of multiple valid vertex data. [Figure 12] FIG. 12 is a flowchart showing an example of the flow of processing by the error correction unit. [Figure 13] FIG. 13 is a flowchart illustrating an example of the processing flow of the phase difference calculation unit and the output unit. [Figure 14] FIG. 14 is a diagram showing the configuration of a heartbeat detecting device according to the fifth embodiment. [Figure 15] FIG. 15 is a flowchart showing an example of the flow of processing by the valid data extraction unit according to the fifth embodiment. [Figure 16] FIG. 16 is a diagram for explaining the process of extracting a plurality of sets of valid vertex data according to the fifth embodiment. [Figure 17] FIG. 17 is a diagram showing the functional configuration of a heartbeat detecting device according to the sixth embodiment. [Figure 18] FIG. 18 is a diagram showing the configuration of a heartbeat detecting device according to the seventh embodiment. [Figure 19] FIG. 19 is a diagram illustrating a hardware configuration of an information processing device. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments will be described with reference to the drawings.

[0011] (First embodiment) 1 is a diagram showing the configuration of a heartbeat detection system 10 according to the first embodiment. The heartbeat detection system 10 detects heartbeat information representing information relating to the heartbeat of a subject without contacting the subject.

[0012] The heartbeat detection system 10 according to the first embodiment includes a millimeter wave radar device 11, a movement control mechanism 12, a subject detection camera 13, an information processing device 14, and a display device 15.

[0013] The millimeter-wave radar device 11 outputs a distance signal that measures the distance to the subject for each position. The distance signal from the millimeter-wave radar device 11 contains a vibration component corresponding to the subject's heartbeat. Therefore, the millimeter-wave radar device 11 is an example of a sensor device that detects a sensor signal containing the subject's heartbeat component without contacting the subject.

[0014] The millimeter-wave radar device 11 measures the distance to the subject by emitting millimeter-wave electromagnetic waves to the subject and detecting the waves reflected from the subject. In this embodiment, the millimeter-wave radar device 11 has a distance resolution capable of detecting the vibrations of a human heartbeat. For example, the millimeter-wave radar device 11 includes at least one MIMO (Multi Input Multi Output) millimeter-wave radar that transmits and receives radio waves at frequencies of 24 GHz or higher. Note that the millimeter-wave radar device 11 is not limited to the MIMO type and may include multiple millimeter-wave radars of other types.

[0015] The millimeter-wave radar device 11 also outputs distance signals representing distances to each position within a certain spatial range at a predetermined time rate, and therefore outputs time-series distance signals representing distances to each position within a certain range on the subject's body.

[0016] Furthermore, the millimeter wave radar device 11 is provided so that the emission direction of the electromagnetic waves can be changed by, for example, a rotation mechanism that rotates the main body.

[0017] The movement control mechanism 12 moves the emission direction of electromagnetic waves from the millimeter-wave radar device 11 in accordance with the control of the information processing device 14. For example, the movement control mechanism 12 moves the emission direction of electromagnetic waves from the millimeter-wave radar device 11 by driving a rotation mechanism.

[0018] Subject detection camera 13 generates a wide-angle image signal, which is an image signal obtained by capturing an image of the subject. Subject detection camera 13 has a relatively wide-angle imaging range. As an example, subject detection camera 13 may have a horizontal angle of view of 100° or more.

[0019] The information processing device 14 is, for example, a computer. The information processing device 14 may be a server device on a network, or may be a cloud or the like in which multiple server devices on a network operate in cooperation with each other. When the information processing device 14 is a server device or the like on a network, it is connected to the millimeter-wave radar device 11, the subject detection camera 13, and the movement control mechanism 12 via the network.

[0020] The information processing device 14 includes a control device 22 and a heartbeat detection device 24 .

[0021] The control device 22 is realized by the information processing device 14 as the information processing device 14 executes a control program. That is, the information processing device 14 functions as the control device 22 by executing the control program.

[0022] The control device 22 includes a subject position specifying unit 25 and a detection direction control unit 26 .

[0023] Subject position identifying unit 25 acquires a wide-angle image signal from subject detection camera 13. For example, subject position identifying unit 25 acquires the wide-angle image signal at a predetermined frame rate. Subject position identifying unit 25 performs image analysis on the wide-angle image signal to identify the position of the subject within the angle of view of the wide-angle image signal.

[0024] The detection direction control unit 26 drives the movement control mechanism 12 in accordance with the subject's position within the angle of view of the wide-angle image signal to control the direction in which electromagnetic waves are emitted by the millimeter-wave radar device 11. The direction in which electromagnetic waves are emitted by the millimeter-wave radar device 11 is an example of the detection direction of a sensor device that detects a sensor signal containing a heartbeat component of the subject without contacting the subject.

[0025] More specifically, the detection direction control unit 26 shifts the emission direction of electromagnetic waves from the millimeter-wave radar device 11 so that the emission direction of electromagnetic waves from the millimeter-wave radar device 11 is directed toward the person being measured. The correspondence between the direction of the angle of view in the wide-angle image signal captured by the person being measured detection camera 13 and the emission direction of electromagnetic waves from the millimeter-wave radar device 11 is adjusted in advance. Therefore, the detection direction control unit 26 can identify the position of the emission direction of electromagnetic waves from the millimeter-wave radar device 11 that corresponds to the position of the person being measured within the angle of view in the wide-angle image signal. This allows the detection direction control unit 26 to shift the emission direction of electromagnetic waves from the millimeter-wave radar device 11 so that the emission direction of electromagnetic waves from the millimeter-wave radar device 11 is directed toward the person being measured.

[0026] The detection direction control unit 26 may change the emission direction of electromagnetic waves from the millimeter-wave radar device 11 every time the person being measured moves a certain distance, or may change the emission direction of electromagnetic waves from the millimeter-wave radar device 11 at regular time intervals. This allows the millimeter-wave radar device 11 to continue irradiating the person being measured with electromagnetic waves, following the person's movement, when the person is located within a range where electromagnetic waves can be emitted.

[0027] The heartbeat detection device 24 acquires the distance signal output from the millimeter-wave radar device 11, performs signal processing on the acquired distance signal, and outputs heartbeat information of the subject. The heartbeat information may be, for example, the heartbeat rate, the heartbeat period, or a heartbeat data string including multiple data representing the time of the heartbeat.

[0028] The heartbeat detecting device 24 may output the heartbeat information of the subject at each unit time, which is a predetermined time interval. When the heartbeat information is output at each unit time, the heartbeat detecting device 24 may output error information indicating that the heartbeat information could not be detected during the unit time during which the heartbeat information could not be detected from the time-series distance signal.

[0029] The display device 15 acquires heart rate information from the heart rate detection device 24 and displays it on a monitor. This allows the person taking the measurement to recognize the heart rate information of the person being measured. The display device 15 also acquires error information from the heart rate detection device 24 and displays it on a monitor. This allows the person taking the measurement to recognize that an error has occurred in detecting the heart rate information of the person being measured. The display device 15 may also display the heart rate information and error information on an LED (Light Emitting Diode) or the like.

[0030] The heart rate detection system 10 may include at least one of a printing device, an audio output device, a storage device, or a communication device instead of or in addition to the display device 15. The printing device acquires heart rate information and error information from the heart rate detection device 24 and prints them on paper or the like. The audio output device acquires heart rate information and error information from the heart rate detection device 24 and outputs them as audio. The storage device acquires heart rate information and error information from the heart rate detection device 24 and stores them in a storage medium. The communication device acquires heart rate information and error information from the heart rate detection device 24 and transmits them to another device via a network. By including a printing device, an audio output device, a storage device, or a communication device, such heart rate detection system 10 can also notify the person being measured of the heart rate information and that an error has occurred.

[0031] FIG. 2 is a diagram showing an example of a housing 17 that houses the millimeter wave radar device 11. As shown in FIG.

[0032] The millimeter-wave radar device 11, movement control mechanism 12, and subject detection camera 13 may be housed integrally in a housing 17, as shown in Fig. 2. For example, the housing 17 is attached to an indoor ceiling, etc. By being housed integrally in such a housing 17, the millimeter-wave radar device 11 and subject detection camera 13 can adjust in advance the correspondence between the emission direction of the electromagnetic waves and the direction of the angle of view of the wide-angle image signal.

[0033] According to the heart rate detection system 10 of the first embodiment described above, the direction of emission of electromagnetic waves from the millimeter wave radar device 11 is moved to follow the subject, so that heart rate information can be detected accurately and without contact even when the subject is moving.

[0034] (Second embodiment) Next, a heartbeat detection system 10 according to a second embodiment will be described. The heartbeat detection system 10 according to the second embodiment has substantially the same functions and configuration as the heartbeat detection system 10 according to the first embodiment. In the heartbeat detection system 10 according to the second embodiment, components having substantially the same functions and configuration as those in the first embodiment are denoted by the same reference numerals, and detailed description thereof will be omitted except for differences.

[0035] FIG. 3 is a diagram showing the configuration of a heartbeat detection system 10 according to the second embodiment.

[0036] The heartbeat detection system 10 according to the second embodiment includes a heartbeat detection camera 18 instead of the millimeter wave radar device 11.

[0037] Heartbeat detection camera 18 captures an image of the subject and outputs a moving image signal of the subject. The moving image signal capturing an image of the subject's exposed skin contains a vibration component corresponding to the subject's heartbeat. Therefore, heartbeat detection camera 18 is an example of a sensor device that detects a sensor signal containing the subject's heartbeat component without contacting the subject.

[0038] Heartbeat detection camera 18 is optical and generates a moving image signal at a frame rate of 30 fps or more and a resolution of 2K or more. Heartbeat detection camera 18 also has a narrower angle of view than subject detection camera 13.

[0039] Furthermore, the heartbeat detection camera 18 is provided so that the direction of the angle of view, that is, the direction of the optical axis, can be moved by, for example, a rotation mechanism that rotates the main body.

[0040] The movement control mechanism 12 moves the direction of the angle of view of the heartbeat detection camera 18 in accordance with the control of the information processing device 14. For example, the movement control mechanism 12 moves the direction of the angle of view of the heartbeat detection camera 18 by driving a rotation mechanism.

[0041] When information processing device 14 is a server device or the like on a network, it is connected to heart rate detection camera 18, subject detection camera 13, and movement control mechanism 12 via the network.

[0042] In this embodiment, the detection direction control unit 26 drives the movement control mechanism 12 in accordance with the subject's position within the angle of view of the wide-angle image signal to control the direction of the angle of view of the heartbeat detection camera 18. The direction of the angle of view of the heartbeat detection camera 18 is an example of the detection direction of the sensor device that detects a sensor signal containing the subject's heartbeat component without contacting the subject.

[0043] More specifically, the detection direction control unit 26 moves the direction of the angle of view of the heartbeat detection camera 18 so that the subject is included in the angle of view of the heartbeat detection camera 18. The correspondence between the direction of the angle of view of the subject detection camera 13 and the direction of the angle of view of the heartbeat detection camera 18 is adjusted in advance. Therefore, the detection direction control unit 26 can identify the direction of the angle of view of the heartbeat detection camera 18 that corresponds to the position of the subject within the angle of view in the wide-angle image signal. This allows the detection direction control unit 26 to move the direction of the angle of view of the heartbeat detection camera 18 so that the subject is included in the angle of view of the heartbeat detection camera 18.

[0044] The detection direction control unit 26 may move the direction of the angle of view of the heartbeat detection camera 18 every time the person being measured moves a certain distance, or may move the direction of the angle of view of the heartbeat detection camera 18 every certain time period. In this way, when the person being measured is located within a range where electromagnetic waves can be emitted, the heartbeat detection camera 18 can follow the movement of the person being measured and keep the person being measured included in the angle of view of the heartbeat detection camera 18.

[0045] In this embodiment, the heart rate detection device 24 acquires the moving image signal output from the heart rate detection camera 18, and identifies an exposed skin region in the acquired moving image signal where the subject's skin is exposed. The heart rate detection device 24 then performs signal processing on the signal component of the exposed skin region and outputs heart rate information of the subject.

[0046] As in the first embodiment, heartbeat detection camera 18, movement control mechanism 12, and subject detection camera 13 may be housed integrally in housing 17. By being housed integrally in housing 17, heartbeat detection camera 18 and subject detection camera 13 can adjust in advance the correspondence between the direction of the angle of view in the moving image signal and the direction of the angle of view in the wide-angle image signal.

[0047] According to the heart rate detection system 10 of the first embodiment described above, the direction of emission of electromagnetic waves from the millimeter wave radar device 11 is moved to follow the subject, so that heart rate information can be detected accurately and without contact even when the subject is moving.

[0048] According to the heart rate detection system 10 of the second embodiment described above, the direction of the angle of view of the heart rate detection camera 18 is moved so that the subject is included within the angle of view of the heart rate detection camera 18, so that heart rate information can be detected accurately and without contact even when the subject is moving.

[0049] (Third embodiment) Next, a heartbeat detection system 10 according to a third embodiment will be described. The heartbeat detection system 10 according to the third embodiment has substantially the same functions and configuration as the heartbeat detection system 10 according to the second embodiment. In the heartbeat detection system 10 according to the third embodiment, components having substantially the same functions and configurations as those in the first and second embodiments are denoted by the same reference numerals, and detailed description thereof will be omitted except for the differences.

[0050] FIG. 4 is a diagram showing the configuration of a heartbeat detection system 10 according to the third embodiment.

[0051] The control device 22 according to the third embodiment further includes a cutout unit .

[0052] The cropping unit 28 acquires a moving image signal from the heartbeat detection camera 18. The cropping unit 28 crops out a signal of a part of the angle of view from the acquired moving image signal, and provides the signal to the heartbeat detection device 24.

[0053] The cropping unit 28 changes the angle of view from which the signal is cropped based on the subject's position, which is the subject's position within the angle of view calculated based on the wide-angle image signal, and the sensor direction, which is the direction of the angle of view of the heartbeat detection camera 18. More specifically, the cropping unit 28 determines horizontal and vertical ranges to crop from the video signal so that at least the subject's exposed skin area is included, and outputs the video signal of the determined range. This allows the heartbeat detection device 24 to efficiently extract the signal component of the exposed skin area from the video signal.

[0054] The heartbeat detection system 10 according to the third embodiment may be configured without the movement control mechanism 12. In this case, the cropping unit 28 determines the horizontal and vertical ranges to be cropped from the moving image signal based only on the subject's position.

[0055] According to the heartbeat detection system 10 of the third embodiment described above, similar to the second embodiment, it is possible to detect heartbeat information in a non-contact manner with high accuracy even when the subject is moving.

[0056] (Fourth embodiment) Next, a heartbeat detection system 10 according to a fourth embodiment will be described. The heartbeat detection system 10 according to the fourth embodiment has substantially the same functions and configuration as the heartbeat detection system 10 according to the first embodiment. Therefore, in the heartbeat detection system 10 according to the fourth embodiment, components having substantially the same functions and configurations as those in the first to third embodiments are denoted by the same reference numerals, and detailed description thereof will be omitted except for the differences.

[0057] 5 is a diagram showing the functional configuration of the heartbeat detecting device 24 according to the fourth embodiment. The heartbeat detecting device 24 according to the fourth embodiment has the functional configuration shown in FIG.

[0058] The heartbeat detection device 24 includes a distance signal acquisition unit 32, a position identification unit 34, a first detection signal generation unit 36, a second detection signal generation unit 38, a difference signal generation unit 40, a heartbeat signal generation unit 42, a vertex data generation unit 44, a valid data extraction unit 46, an error correction unit 48, a memory unit 50, a phase difference calculation unit 52, an error output unit 54, and an output unit 56.

[0059] The distance signal acquisition unit 32 acquires distance signals that measure the distance to the subject for each position within a certain spatial range from the millimeter-wave radar device 11. The distance signal acquisition unit 32 performs predetermined signal processing on the distance signals so that subsequent processing can be performed appropriately.

[0060] The position specifying unit 34 acquires the distance signal from the distance signal acquiring unit 32. The position specifying unit 34 performs signal analysis on the acquired distance signal, and specifies a first position and a second position in the distance signal.

[0061] The first position is a position on the subject where the heartbeat component is easily detected.

[0062] The second position is a position on the subject that is different from the first position and is a position where it is more difficult to detect a heartbeat component than the first position.

[0063] For example, the position identifying unit 34 analyzes the vibration component for each position based on the distance signal and extracts the vibration corresponding to the heartbeat of the person being measured for each position. Then, the position identifying unit 34 identifies a position having vibrations including the heartbeat component of the person being measured as a first position. Furthermore, the position identifying unit 34 identifies a position having less vibrations including the heartbeat component of the person being measured than the first position as a second position. The position identifying unit 34 may identify a position having less vibrations including the heartbeat component than the first position by a predetermined amount or a predetermined percentage as the second position. Note that the vibrations at the second position may not have any heartbeat component at all.

[0064] The first detection signal generating unit 36 ​​acquires a distance signal from the distance signal acquiring unit 32. The first detection signal generating unit 36 ​​also acquires information identifying the first position from the position identifying unit 34. Based on the distance signal, the first detection signal generating unit 36 ​​generates a first detection signal representing the distance at a first position where vibrations including a heartbeat component of the subject occur. The first detection signal is a signal whose level includes the heartbeat component. For example, based on the information identifying the first position acquired from the position identifying unit 34, the first detection signal generating unit 36 ​​extracts a signal component at the first position from the distance signal acquired from the distance signal acquiring unit 32, and generates a first detection signal based on the extracted signal component.

[0065] The second detection signal generation unit 38 acquires the distance signal from the distance signal acquisition unit 32. The second detection signal generation unit 38 also acquires information identifying the second position from the position identification unit 34. The second detection signal generation unit 38 generates a second detection signal representing the distance at the second position based on the distance signal. The second detection signal is a signal whose level contains fewer heartbeat components than the first detection signal. Note that the second detection signal may not contain any heartbeat components at all. For example, the second detection signal generation unit 38 extracts the signal component of the second position from the distance signal acquired from the distance signal acquisition unit 32 based on the information identifying the second position acquired from the position identification unit 34, and generates the second detection signal based on the extracted signal component.

[0066] The differential signal generating unit 40 receives the first detection signal from the first detection signal generating unit 36 ​​and receives the second detection signal from the second detection signal generating unit 38. The differential signal generating unit 40 generates a differential signal that represents the difference between the first detection signal and the second detection signal.

[0067] Here, the distance signal contains vibration noise of the millimeter-wave radar device 11. This vibration noise is contained as in-phase noise in both the first detection signal and the second detection signal. That is, the first detection signal and the second detection signal each contain the vibration of the millimeter-wave radar device 11 as noise. In contrast, the first detection signal contains a larger heartbeat component than the second detection signal. Therefore, in the differential signal representing the difference between the first detection signal and the second detection signal, the vibration noise of the millimeter-wave radar device 11 is canceled out and the heartbeat component remains. Therefore, the differential signal contains less noise generated in response to the vibration of the millimeter-wave radar device 11, making it easier to detect the heartbeat component.

[0068] The heartbeat signal generating unit 42 acquires the differential signal from the differential signal generating unit 40. Based on the differential signal, the heartbeat signal generating unit 42 generates a heartbeat signal representing the heartbeat component. The heartbeat signal generating unit 42 generates the heartbeat signal by detecting vertices, which are peak points or bottom points, from the differential signal, removing unnecessary high-frequency components higher than the frequency of the heartbeat, and removing DC components. An example of the processing flow in the heartbeat signal generating unit 42 will be described later with reference to FIG. 6.

[0069] The vertex data generation unit 44 acquires the heartbeat signal from the heartbeat signal generation unit 42. The vertex data generation unit 44 generates a plurality of vertex data from the heartbeat signal. Each of the plurality of vertex data represents the time and level of a vertex, which is a peak point or a bottom point in the heartbeat signal. In this embodiment, the vertex data generation unit 44 divides the heartbeat signal by unit time, for example, S time (S is a positive integer), and generates a plurality of vertex data for each unit time.

[0070] The cardiac waveform includes one waveform in which the level rises or falls sharply in a short period of time within one cycle. Therefore, the times included in each of the plurality of vertex data generated by the vertex data generating unit 44 are candidates for the times at which the waveform in which the level rises or falls sharply in the cardiac waveform occurs. An example of the processing flow in the vertex data generating unit 44 will be described later with reference to FIG. 8.

[0071] The valid data extraction unit 46 acquires multiple vertex data sets from the vertex data generation unit 44. The valid data extraction unit 46 extracts multiple sets of valid vertex data sets that satisfy the periodicity of a heartbeat from the multiple vertex data sets. In this embodiment, the valid data extraction unit 46 extracts multiple sets of valid vertex data sets for each unit time.

[0072] The plurality of vertex data may include vertex data that does not represent the time when a waveform with abrupt increases or decreases in level occurred in the heartbeat waveform due to noise or the like. A set of multiple valid vertex data is a data group obtained by removing vertex data caused by noise or the like from the plurality of vertex data. The range of a human heartbeat cycle and the range of heartbeat fluctuation are known in advance. Therefore, the valid data extraction unit 46 can extract, from the plurality of vertex data, a set of multiple valid vertex data that satisfies the periodicity of a heartbeat using a predetermined algorithm that takes into account the range of a human heartbeat cycle and the range of heartbeat fluctuation.

[0073] Furthermore, due to the influence of errors and the like, the plurality of vertex data may not include vertex data representing the time at which a waveform with a steep increase or decrease in level occurred in some cardiac cycles. In such cases, the set of valid vertex data may lack valid vertex data representing the time at which some cardiac cycles occurred. In other words, the plurality of cardiac cycles represented by the set of valid vertex data may include blank cardiac cycles that do not include valid vertex data.

[0074] Furthermore, in this embodiment, the valid data extraction unit 46 determines, for each unit time, whether the number of valid vertex data included in the extracted set of valid vertex data is equal to or greater than a predetermined number. If the number of valid vertex data included in the extracted set of valid vertex data is less than the predetermined number, the valid data extraction unit 46 outputs a notification indicating an error to the error output unit 54. Furthermore, if the number of valid vertex data included in the extracted set of valid vertex data is less than the predetermined number, the valid data extraction unit 46 terminates subsequent processing for that unit time.

[0075] An example of the processing flow in the valid data extraction unit 46 will be described later with reference to FIG.

[0076] The error correction unit 48 acquires multiple sets of valid vertex data from the valid data extraction unit 46. In this embodiment, the valid data extraction unit 46 acquires multiple sets of valid vertex data from the valid data extraction unit 46 if the number of valid vertex data included in the extracted sets of valid vertex data per unit time is equal to or greater than a predetermined number.

[0077] If some of the sets of valid vertex data are missing or erroneous, the error correction unit 48 generates a heartbeat data sequence by performing error correction on the missing or erroneous valid vertex data. For example, the error correction unit 48 determines whether the acquired sets of valid vertex data are missing valid vertex data representing some of the cardiac cycle times. If the acquired sets of valid vertex data are missing valid vertex data representing some of the cardiac cycle times, the error correction unit 48 generates new valid vertex data representing the missing valid vertex data by performing error correction on the sets of valid vertex data. Then, the error correction unit 48 generates a heartbeat data sequence by adding the new valid vertex data generated by error correction to the sets of valid vertex data.

[0078] In this embodiment, the error correction unit 48 generates such a heartbeat data sequence for each unit time. An example of the processing flow in the error correction unit 48 will be described later with reference to FIG.

[0079] The storage unit 50 acquires and stores the heartbeat data sequence from the error correction unit 48. In this embodiment, the storage unit 50 acquires and stores the heartbeat data sequence for each unit time.

[0080] The phase difference calculation unit 52 calculates, for each unit time, the phase difference between the heartbeat data sequence stored in the storage unit 50 and the heartbeat data sequence newly output from the error correction unit 48. For example, if the unit time of the heartbeat data sequence stored in the storage unit 50 is a first unit time and the unit time of the heartbeat data sequence newly output from the error correction unit 48 is a second unit time, the second unit time is a unit time that comes later than the first unit time. In this case, the phase difference calculation unit 52 calculates the phase difference between the heartbeat data sequence of the first unit time and the heartbeat data sequence of the second unit time.

[0081] The phase difference calculation unit 52 determines, for each unit time, whether the phase difference between the heartbeat data sequence for the first unit time stored in the storage unit 50 and the heartbeat data sequence for the first unit time newly output from the error correction unit 48 is equal to or greater than a predetermined value. The phase difference calculation unit 52 outputs the determination result to the error output unit 54 and the output unit 56 for each unit time.

[0082] The error output unit 54 determines, for each unit time, whether or not it has received an error notification from the valid data extraction unit 46, indicating that the number of valid vertex data included in the set of extracted valid vertex data is less than a predetermined number. When the error output unit 54 receives an error notification from the valid data extraction unit 46, it outputs error information to the display device 15, indicating that heart rate information could not be detected during that unit time.

[0083] Furthermore, the error output unit 54 obtains, for each unit time, from the phase difference calculation unit 52, a determination result as to whether or not the phase difference is equal to or greater than a predetermined value. If the phase difference is equal to or greater than the predetermined value, the error output unit 54 outputs error information indicating that heartbeat information could not be detected for the first unit time to the display device 15. That is, if the phase difference is equal to or greater than the predetermined value, the error output unit 54 outputs error information indicating that heartbeat information could not be detected for the heartbeat data sequence stored in the storage unit 50 to the display device 15.

[0084] The output unit 56 generates heartbeat information based on the heartbeat data sequence stored in the storage unit 50 and outputs the generated heartbeat information to the display device 15. The output unit 56 may calculate at least one of the heart rate and the heartbeat period based on the heartbeat data sequence stored in the storage unit 50 and output the calculated at least one of the heart rate and the heartbeat period as heartbeat information. Alternatively, the output unit 56 may output the heartbeat data sequence stored in the storage unit 50 as heartbeat information.

[0085] In this embodiment, the output unit 56 obtains, for each unit time, from the phase difference calculation unit 52, a determination result as to whether the phase difference is equal to or greater than a predetermined value. If, for each unit time, the phase difference is not equal to or greater than the predetermined value, that is, if the phase difference is smaller than the predetermined value, the output unit 56 outputs heartbeat information based on the heartbeat data sequence for the first unit time that is stored in the storage unit 50. Furthermore, if, for each unit time, the phase difference is equal to or greater than the predetermined value, the output unit 56 deletes the heartbeat data sequence for the first unit time that is stored in the storage unit 50, and does not output the heartbeat information.

[0086] An example of the processing flow in the phase difference calculation section 52 and the output section 56 will be described later with reference to FIG.

[0087] Fig. 6 is a flowchart showing an example of the processing flow of the heartbeat signal generating unit 42. Fig. 7 is a diagram for explaining the processing of joining vertices in the heartbeat signal generating unit 42. The heartbeat signal generating unit 42 executes the processing, for example, according to the flow shown in Fig. 6.

[0088] First, in S11, the heartbeat signal generator 42 generates a moving average signal by taking a moving average of the differential signal, thereby enabling the heartbeat signal generator 42 to generate a signal with reduced high-frequency noise and the like.

[0089] Next, in S12, the heart rate signal generating unit 42 detects vertices, which are peak points or bottom points, from the moving average signal, and generates a vertex data string that indicates the level and time of the vertex.

[0090] Next, in S13, the heartbeat signal generation unit 42 performs first-order differentiation on the vertex data sequence to generate a first-order differentiated data sequence. This allows the heartbeat signal generation unit 42 to generate a data sequence from which DC components have been removed. Note that the heartbeat signal generation unit 42 may perform other processing, such as second differentiation, instead of first-order differentiation to remove DC components.

[0091] Next, in S14, the heartbeat signal generating unit 42 detects peaks in the signal representing the first-order differentiated data string.

[0092] Next, in S15, the heartbeat signal generating unit 42 generates a signal by connecting the vertices detected in S14. For example, as shown in Fig. 7, the heartbeat signal generating unit 42 generates a signal by connecting the vertices with straight lines. Instead of using straight lines, the heartbeat signal generating unit 42 may generate a signal by smoothly connecting the vertices using a predetermined function.

[0093] Next, in S16, the heartbeat signal generating unit 42 performs band-pass filtering on the signal obtained by joining the peaks.

[0094] Next, in S17, the heartbeat signal generating unit 42 removes the DC component from the band-pass filtered signal by transversal filtering.

[0095] By performing the above-described processes from S11 to S17, the heartbeat signal generating unit 42 can generate a heartbeat signal from which noise and the like have been removed from the differential signal.

[0096] 8 is a flowchart showing an example of the processing flow of the vertex data generation unit 44. The vertex data generation unit 44 executes processing according to the flow shown in FIG.

[0097] First, in S21, the vertex data generation unit 44 divides the heartbeat signal into unit times, for example, time S. Then, the vertex data generation unit 44 executes the loop process between S22 and S24 for each unit time.

[0098] In the loop processing for each unit time, in S23, the vertex data generation unit 44 detects vertices, which are peak points or bottom points, from the heartbeat signal. Then, the vertex data generation unit 44 generates a plurality of vertex data, each of which represents the time and level of the vertex. The vertex data generation unit 44 repeatedly executes the loop processing between S22 and S24 until measurement of the heartbeat of the subject is completed.

[0099] By executing the above-described processes from S21 to S24, the vertex data generating unit 44 can generate a plurality of vertex data from the heartbeat signal for each unit time.

[0100] Fig. 9 is a flowchart showing an example of the processing flow of valid data extraction unit 46. Fig. 10 is a diagram showing a first example for explaining the processing for extracting a plurality of sets of valid vertex data. Fig. 11 is a diagram showing a second example for explaining the processing for extracting a plurality of sets of valid vertex data.

[0101] The valid data extraction unit 46 executes the process according to the flow shown in FIG. 9, for example.

[0102] First, the valid data extraction unit 46 repeatedly executes the loop process between S31 and S39 for each unit time.

[0103] In the loop processing between S31 and S39, first, in S32, the valid data extraction unit 46 determines whether or not there is a plurality of vertex data for the next unit time. If there is not a plurality of vertex data for the next unit time (No in S32), the valid data extraction unit 46 ends this flow. If there is a plurality of vertex data for the next unit time (Yes in S32), the valid data extraction unit 46 advances the processing to S33.

[0104] In S33, the valid data extraction unit 46 selects the N vertex data items with the highest amplitude from the plurality of vertex data items for the corresponding unit time generated by the vertex data generation unit 44. If the vertex is a peak point, the valid data extraction unit 46 selects the N vertex data items with the highest level. If the vertex is a bottom point, the valid data extraction unit 46 selects the N vertex data items with the lowest level.

[0105] It should be noted that N is a predetermined integer equal to or greater than 2. As an example, N is a value that allows extraction of the maximum amount of valid vertex data that may include a heartbeat during the unit time S.

[0106] Subsequently, in S34, the valid data extraction unit 46 extracts a set of valid vertex data that satisfies the periodicity of a heartbeat from the selected N pieces of vertex data.

[0107] For example, as shown in FIG. 10, the valid data extraction unit 46 calculates the time difference (T x Then, the valid data extraction unit 46 calculates the average period (T xsn ) (n is an integer equal to or greater than 1) plus or minus a predetermined margin. In this case, the valid data extraction unit 46 extracts a plurality of sets of valid vertex data so that the range of the average period (T xsn ) is within the range of the human heartbeat cycle. This allows the valid data extraction unit 46 to extract, from the selected N pieces of vertex data, a set of valid vertex data that satisfies the periodicity of the heartbeat.

[0108] 11, the valid data extraction unit 46 may extract a plurality of valid vertex data sets that are missing valid vertex data that represent the times of some cardiac cycles. That is, the valid data extraction unit 46 may extract a plurality of valid vertex data sets that include blank cardiac cycles that do not include valid vertex data.

[0109] Subsequently, in S35, the valid data extraction unit 46 determines whether the number of valid vertex data included in the set of multiple valid vertex data is M or more.

[0110] Note that M is a predetermined integer equal to or greater than 2 and smaller than N. In this embodiment, M is N / 2. As an example, M is a value obtained by subtracting a predetermined margin from the minimum value at which a heartbeat may be included in a unit time, S.

[0111] If the number of valid vertex data included in the set of multiple valid vertex data is not equal to or greater than M (=N / 2) (No in S35), the valid data extraction unit 46 advances the process to S36.

[0112] In S36, the valid data extraction unit 46 discards the data for the current unit time. That is, the valid data extraction unit 46 deletes the set of multiple valid vertex data for the current unit time, and ends the subsequent processing for that unit time.

[0113] Next, in S37, the valid data extraction unit 46 outputs an error notification indicating that the number of valid vertex data included in the set of multiple valid vertex data is less than M (=N / 2) to the error output unit 54. This allows the error output unit 54 to output error information indicating that heartbeat information could not be detected in the current unit time. After completing S37, the valid data extraction unit 46 repeats the process from S32 for the next unit time.

[0114] If the number of valid vertex data included in the set of valid vertex data is M (=N / 2) or more (Yes in S35), the valid data extraction unit 46 advances the process to S38.

[0115] In S38, the valid data extraction unit 46 calculates the average period of the heartbeats represented by the sets of valid vertex data. For example, the valid data extraction unit 46 may acquire the average period used to extract the sets of valid vertex data, or may calculate the average period from the sets of valid vertex data after extraction. After completing S38, the valid data extraction unit 46 repeats the process from S32 for the next unit time.

[0116] By performing the above processes from S31 to S39, the valid data extraction unit 46 can extract, from the plurality of vertex data, a set of multiple valid vertex data that satisfies the periodicity of a heartbeat. Furthermore, if the number of valid vertex data included in the extracted set of multiple valid vertex data is smaller than a predetermined number, the valid data extraction unit 46 can output error information indicating that heartbeat information could not be detected in the current unit time to the error output unit 54. Furthermore, if the number of valid vertex data included in the extracted set of multiple valid vertex data is smaller than a predetermined number, the valid data extraction unit 46 can terminate subsequent processing for the current unit time.

[0117] FIG. 12 is a flowchart showing an example of the flow of processing by the error correction unit 48.

[0118] The error correction unit 48 executes the process according to the flow shown in FIG. 12, for example.

[0119] First, the error correction unit 48 repeatedly executes the loop process between S41 and S49 for each unit time.

[0120] In the loop processing between S41 and S49, first, in S42, the error correction unit 48 determines whether or not there are multiple sets of valid vertex data for the current unit time. That is, the error correction unit 48 determines whether or not multiple sets of valid vertex data for the current unit time have been output from the valid data extraction unit 46. If there are not multiple sets of valid vertex data for the current unit time (No in S42), the error correction unit 48 repeats the processing from S42 for the next unit time. If there are multiple sets of valid vertex data for the current unit time (Yes in S42), the error correction unit 48 proceeds with the processing to the loop processing between S43 and S47.

[0121] In the loop processing between S43 and S47, the error correction unit 48 repeatedly executes the processing for each pair of valid vertex data that are adjacent in time and are included in a set of multiple valid vertex data. Two pairs of valid vertex data that are adjacent in time are pairs of valid vertex data that have no other valid vertex data between them.

[0122] In the loop process between S43 and S47, first, in S44, the error correction unit 48 calculates the time difference between two pieces of valid vertex data that are adjacent in time to be processed.

[0123] Next, in S45, the error correction unit 48 determines whether the time difference is within a first time range. The first time range ranges from a lower limit obtained by reducing a predetermined first margin amount (α1) by twice the average period to an upper limit obtained by increasing a predetermined second margin amount (α2) by twice the average period. More specifically, the first time range is a time range that is equal to or greater than (average period × 2 - α1) and equal to or less than (average period × 2 + α2). Note that the first margin amount (α1) and the second margin amount (α2) are values ​​equal to or greater than the maximum value of human heartbeat fluctuation. The first margin amount (α1) and the second margin amount (α2) may be the same. By executing the process of S45, the error correction unit 48 can detect the position of the missing valid vertex data.

[0124] If the time difference is not within the first time range (No in S45), the error correction unit 48 returns the process to S44 and proceeds with the process on the two valid vertex data whose times are adjacent to each other and which will be the next processing targets. If the time difference is within the first time range (Yes in S45), the error correction unit 48 proceeds with the process to S46.

[0125] In S46, the error correction unit 48 generates new valid vertex data that represents the midpoint time of the two valid vertex data that are adjacent in time to be processed. By executing the process of S46, the error correction unit 48 can generate new valid vertex data that represents the missing valid vertex data. After completing the process of S46, the error correction unit 48 returns the process to S44 and proceeds with the process on the next two valid vertex data that are adjacent in time to be processed.

[0126] When the error correction unit 48 has completed the processes from S44 to S46 for all two pieces of valid vertex data that are adjacent in time, the process proceeds to S48.

[0127] In S48, the error correction unit 48 generates a heartbeat data sequence by adding all of the generated new valid vertex data to a plurality of sets of valid vertex data, and then outputs the generated heartbeat data sequence.

[0128] By executing the above processes from S41 to S49, when valid vertex data representing some cardiac cycle times is missing from a set of multiple valid vertex data, the error correction unit 48 can generate new valid vertex data representing the missing valid vertex data by performing error correction from the set of multiple valid vertex data. Then, the error correction unit 48 can generate a cardiac data string in which the new valid vertex data generated by error correction is added to the set of multiple valid vertex data.

[0129] FIG. 13 is a flowchart showing an example of the processing flow of the phase difference calculation section 52 and the output section 56.

[0130] The phase difference calculation section 52 and the output section 56 execute the process according to the flow shown in FIG. 13, for example.

[0131] First, the phase difference calculation unit 52 and the output unit 56 repeatedly execute the loop process between S51 and S60 for each unit time.

[0132] In the loop process between S51 and S60, first, in S52, the phase difference calculation unit 52 determines whether or not a heartbeat data sequence for the current unit time exists. That is, the phase difference calculation unit 52 determines whether or not a heartbeat data sequence for the current unit time has been output from the error correction unit 48. If a heartbeat data sequence for the current unit time does not exist (No in S52), the phase difference calculation unit 52 repeats the process from S52 for the next unit time. If a heartbeat data sequence for the current unit time exists (Yes in S52), the phase difference calculation unit 52 proceeds to S53.

[0133] In S53, the phase difference calculation unit 52 determines whether or not there is a heartbeat data sequence for a past unit time stored in the storage unit 50. If there is no heartbeat data sequence for a past unit time stored in the storage unit 50 (No in S53), the phase difference calculation unit 52 proceeds to S59. If there is a heartbeat data sequence for a past unit time stored in the storage unit 50 (Yes in S53), the phase difference calculation unit 52 proceeds to S54.

[0134] In S54, the phase difference calculation unit 52 calculates, for each unit time, the phase difference between the heartbeat data sequence for the past unit time stored in the memory unit 50 and the heartbeat data sequence for the current unit time output from the error correction unit 48.

[0135] For example, the phase difference calculation unit 52 calculates the time difference between the latest valid vertex data in the heartbeat data sequence for the past unit time stored in the storage unit 50, and the earliest valid vertex data in the heartbeat data sequence for the current unit time output from the error correction unit 48, that is, the time of the oldest valid vertex data. Next, the phase difference calculation unit 52 calculates the remainder of the calculated time difference and the average period of the heartbeat data sequence for the current unit time output from the error correction unit 48. Then, the phase difference calculation unit 52 calculates the phase difference based on the calculated remainder and the average period of the heartbeat data sequence. For example, the phase difference calculation unit 52 calculates the phase difference based on the ratio of the remainder to the average period.

[0136] Subsequently, in S55, the phase difference calculation unit 52 determines whether the calculated phase difference is equal to or greater than a predetermined value (β). If the phase difference is equal to or greater than the predetermined value (β) (Yes in S55), the phase difference calculation unit 52 proceeds to S56.

[0137] In S56, the output unit 56 deletes the heartbeat data sequence for the past unit time stored in the storage unit 50 from the storage unit 50.

[0138] Subsequently, in S57, the phase difference calculation unit 52 outputs an error notification indicating that the phase difference of the heartbeat data sequence for the past unit time stored in the storage unit 50 is equal to or greater than a predetermined value (β) to the error output unit 54. This enables the error output unit 54 to output error information indicating that heartbeat information could not be detected for the past unit time stored in the storage unit 50. After completing S57, the phase difference calculation unit 52 advances the process to S59.

[0139] If the phase difference is not equal to or greater than the predetermined value (β) (No in S55), the phase difference calculation unit 52 advances the process to S58.

[0140] In S58, the output unit 56 calculates and outputs heartbeat information based on the heartbeat data sequence for a past unit time stored in the storage unit 50. For example, the output unit 56 calculates the heart rate based on the heartbeat data sequence and outputs it as heartbeat information. Alternatively, for example, the output unit 56 calculates the heartbeat period based on the heartbeat data sequence and outputs it as heartbeat information. Alternatively, for example, the output unit 56 may output the heartbeat data sequence as is as heartbeat information. After completing the process of S58, the output unit 56 proceeds to S59.

[0141] In S59, the output unit 56 stores the heartbeat data string for the current unit time output from the error correction unit 48 in the storage unit 50 as the next processing target.

[0142] After completing the process of S59, the phase difference calculation section 52 and the output section 56 repeat the process for the next unit time from S52.

[0143] By executing the above processes from S51 to S60, the phase difference calculation unit 52 and the output unit 56 can output heartbeat information for each unit time. Furthermore, if the phase difference between the heartbeat data sequence for the past unit time stored in the storage unit 50 and the heartbeat data sequence for the current unit time output from the error correction unit 48 is equal to or greater than a predetermined value, the phase difference calculation unit 52 and the output unit 56 delete the heartbeat data sequence for the past unit time stored in the storage unit 50 and cause the error output unit 54 to output error information.

[0144] As described above, the heartbeat detection system 10 according to the fourth embodiment generates a first detection signal representing the distance at a first position where vibrations including the heartbeat component of the subject occur, and a second detection signal representing the distance at a second position where vibrations include fewer heartbeat components than the first detection signal, based on a distance signal detected by the millimeter-wave radar device 11. The heartbeat detection system 10 then generates a heartbeat signal based on a differential signal representing the difference between the first detection signal and the second detection signal.

[0145] As a result, the heartbeat detection system 10 according to the fourth embodiment can remove vibration noise of the millimeter wave radar device 11 from the distance signal and detect heartbeat information with high accuracy.

[0146] Furthermore, the heartbeat detection system 10 according to the fourth embodiment extracts, from the heartbeat signal, a plurality of sets of valid vertex data that satisfy the periodicity of the heartbeat. If a set of valid vertex data is missing valid vertex data that represents a part of the time of the heartbeat cycle, the heartbeat detection system 10 generates new valid vertex data representing the missing valid vertex data by performing error correction from the set of valid vertex data, and generates a heartbeat data sequence in which the new valid vertex data is added to the set of valid vertex data.

[0147] As a result, the heartbeat detection system 10 according to the fourth embodiment can detect heartbeat information even if the heartbeat signal component contained in the differential signal is weak.

[0148] In addition, the heartbeat detection system 10 of the fourth embodiment outputs error information when it is unable to extract a set of multiple valid vertex data that satisfies the periodicity of a heartbeat from the heartbeat signal, and when the phase difference between the heartbeat data sequence of the first unit time and the heartbeat data sequence of the second unit time prior to the first unit time is greater than or equal to a predetermined value.

[0149] As a result, when the heartbeat information cannot be detected, the heartbeat detection system 10 according to the fourth embodiment can make the person taking the measurement or the like aware that the heartbeat information cannot be detected.

[0150] (Fifth embodiment) Next, a heartbeat detection system 10 according to a fifth embodiment will be described. The heartbeat detection system 10 according to the fifth embodiment has substantially the same functions and configuration as the heartbeat detection system 10 according to the fourth embodiment. Therefore, in the heartbeat detection system 10 according to the fifth embodiment, components having substantially the same functions and configurations as those in the fourth embodiment are denoted by the same reference numerals, and detailed description thereof will be omitted except for the differences.

[0151] FIG. 14 is a diagram showing the configuration of a heartbeat detecting device 24 according to the fifth embodiment.

[0152] The heartbeat detecting device 24 according to the fifth embodiment includes P second detection signal generating sections 38-1 to 38-P instead of the single second detection signal generating section 38. P is an integer of 2 or greater.

[0153] In the fifth embodiment, the position identifying unit 34 identifies one first position and P second positions. Each of the P second positions is different from the first position, contains fewer heartbeat components than the first position, and is a position where it is more difficult to detect the heartbeat component. Furthermore, each of the P second positions is different from each other.

[0154] Each of the P second detection signal generators 38-1 to 38-P acquires a distance signal from the distance signal acquirer 32.

[0155] Furthermore, each of the P second detection signal generation units 38-1 to 38-P acquires one piece of information identifying the P second positions from the position identification unit 34. Each of the P second detection signal generation units 38-1 to 38-P acquires information identifying a different second position from the others.

[0156] Each of the P second detection signal generators 38-1 to 38-P generates a second detection signal representing the distance at the corresponding second position based on the distance signal. For example, based on information identifying the second position acquired from the position identifying unit 34, the second detection signal generator 38 extracts a signal component of the corresponding second position from the distance signal acquired from the distance signal acquirer 32, and generates a second detection signal based on the extracted signal component.

[0157] The heartbeat detection device 24 according to the fifth embodiment may include a vibration sensor device, as in the fifth embodiment. In this case, the heartbeat detection device 24 includes a vibration sensor signal acquisition unit, as in the fifth embodiment. In this case, any one second detection signal generation unit 38 among the P second detection signal generation units 38-1 to 38-P acquires a sensor signal from the vibration sensor signal acquisition unit instead of a distance signal, and generates a second detection signal including vibration of the millimeter-wave radar device 11 based on the acquired sensor signal.

[0158] The differential signal generator 40 according to the fifth embodiment generates differential signals corresponding to the P second detection signals, each of which represents a difference between the first detection signal and the corresponding second detection signal. Thus, the differential signal generator 40 generates P differential signals.

[0159] The heartbeat signal generating section 42 according to the fifth embodiment generates a heartbeat signal for each of the P differential signals. Therefore, the heartbeat signal generating section 42 generates P heartbeat signals.

[0160] The vertex data generating section 44 according to the fifth embodiment generates a plurality of vertex data for each of the P heartbeat signals for each unit time.

[0161] The valid data extracting section 46 according to the fifth embodiment extracts a plurality of sets of valid vertex data based on all of the plurality of vertex data of the P heartbeat signals for each unit time.

[0162] Fig. 15 is a flowchart showing an example of the flow of processing by the valid data extraction unit 46 according to the fifth embodiment. Fig. 16 is a diagram for explaining processing for extracting a plurality of sets of valid vertex data.

[0163] In the fifth embodiment, the valid data extraction unit 46 executes processing according to the flow shown in FIG. 15, for example.

[0164] First, the valid data extraction unit 46 repeatedly executes the loop process between S71 and S83 for each unit time.

[0165] In the loop processing between S71 and S83, first, in S72, the valid data extraction unit 46 determines whether or not multiple vertex data for the next unit time exists. If multiple vertex data for the next unit time does not exist (No in S72), the valid data extraction unit 46 ends this flow. If multiple vertex data for the next unit time exists (Yes in S72), the valid data extraction unit 46 advances the processing to the loop processing between S73 and S76.

[0166] The valid data extraction unit 46 executes the loop process between S73 and S76 for each of the P heartbeat signals.

[0167] In the loop process between S73 and S76, first, in S74, the valid data extraction unit 46 selects the top N vertex data in descending order of amplitude from the plurality of vertex data detected from the target heartbeat signal.

[0168] Subsequently, in S75, the valid data extraction unit 46 extracts a plurality of sets of intermediate vertex data that satisfy the periodicity of a heartbeat from the selected N pieces of vertex data.

[0169] When the loop process between S73 and S76 is completed for all P heartbeat signals, the valid data extraction unit 46 advances the process to S77.

[0170] In S77, the valid data extraction unit 46 selects a plurality of sets of Q or more intermediate peak data whose time difference is within a predetermined value from the plurality of intermediate peak data for each of the P heartbeat signals. In this embodiment, Q is an integer equal to or greater than P / 2.

[0171] For example, as shown in Fig. 16, suppose that the heartbeat signal generator 42 generates five types of heartbeat signals based on five types of difference signals. In such a case, the valid data extractor 46 selects a set of three or more intermediate vertex data whose time difference is within a predetermined value. The predetermined value represents the error range within which the heartbeat timings can be considered to be synchronized.

[0172] Next, in S78, for each of the selected sets, the valid data extraction unit 46 generates one valid vertex data item from among the valid vertex data items based on the Q or more pieces of intermediate vertex data items. For example, as shown in FIG. 16 , the valid data extraction unit 46 generates valid vertex data for each set by averaging the Q or more pieces of valid vertex data items included in that set.

[0173] Subsequently, in S79, the valid data extraction unit 46 determines whether the number of valid vertex data included in the set of multiple valid vertex data is M or more.

[0174] If the number of valid vertex data included in the set of valid vertex data is not equal to or greater than M (=N / 2) (No in S79), the valid data extraction unit 46 advances the process to S80.

[0175] In S80, the valid data extraction unit 46 deletes the set of valid vertex data for the current unit time. Subsequently, in S81, the valid data extraction unit 46 outputs an error notification indicating that the number of valid vertex data included in the set of valid vertex data is less than M (=N / 2) to the error output unit 54. This allows the error output unit 54 to output error information indicating that heartbeat information could not be detected in the current unit time. After completing S81, the valid data extraction unit 46 repeats the process from S72 for the next unit time.

[0176] If the number of valid vertex data included in the set of valid vertex data is M (=N / 2) or more (Yes in S79), the valid data extraction unit 46 advances the process to S82.

[0177] In S82, the valid data extraction unit 46 calculates the average period of the heartbeat represented by a set of multiple valid vertex data. After completing S82, the valid data extraction unit 46 repeats the process from S72 for the next unit time.

[0178] By performing the above processes from S71 to S83, the valid data extraction unit 46 can extract, from the plurality of vertex data, a set of multiple valid vertex data that satisfies the periodicity of a heartbeat. Furthermore, if the number of valid vertex data included in the extracted set of multiple valid vertex data is smaller than a predetermined number, the valid data extraction unit 46 can output error information indicating that heartbeat information could not be detected in the current unit time to the error output unit 54. Furthermore, if the number of valid vertex data included in the extracted set of multiple valid vertex data is smaller than a predetermined number, the valid data extraction unit 46 can terminate subsequent processing for the current unit time.

[0179] As described above, the heartbeat detection system 10 according to the fifth embodiment, like the fourth embodiment, can remove vibration noise of the millimeter-wave radar device 11 from the distance signal and detect heartbeat information with high accuracy. Also, like the fourth embodiment, the heartbeat detection system 10 according to the fifth embodiment can detect heartbeat information even if the heartbeat signal component included in the differential signal is weak. Also, like the fourth embodiment, the heartbeat detection system 10 according to the fifth embodiment can inform the person taking the measurement that heartbeat information cannot be detected when heartbeat information cannot be detected. Furthermore, the heartbeat detection system 10 according to the fifth embodiment uses P differential signals, and therefore can detect heartbeat information with higher accuracy.

[0180] (Sixth embodiment) Next, a heartbeat detection system 10 according to a sixth embodiment will be described. The heartbeat detection system 10 according to the sixth embodiment has substantially the same functions and configuration as the heartbeat detection system 10 according to the fifth embodiment. In the heartbeat detection system 10 according to the sixth embodiment, components having substantially the same functions and configurations as those in the first to fifth embodiments are denoted by the same reference numerals, and detailed description thereof will be omitted except for differences.

[0181] 17 is a diagram showing the functional configuration of the heartbeat detecting device 24 according to the sixth embodiment. The heartbeat detecting device 24 according to the fourth embodiment has the functional configuration shown in FIG.

[0182] The heartbeat detection device 24 includes an acquisition unit 82, an area identification unit 84, a detection signal generation unit 88, a difference signal generation unit 40, a heartbeat signal generation unit 42, a vertex data generation unit 44, a valid data extraction unit 46, an error correction unit 48, a memory unit 50, a phase difference calculation unit 52, an error output unit 54, and an output unit 56.

[0183] The acquisition unit 82 acquires a moving image signal of the subject captured by the heart rate detection camera 18. The acquisition unit 82 performs image processing on the moving image signal to adjust sharpness, brightness, etc., so that subsequent processing can be performed appropriately. The acquisition unit 82 may also perform frame synthesis, composite processing, contrast correction, etc. on the moving image signal.

[0184] The region identifying unit 84 acquires the moving image signal from the acquiring unit 82. The region identifying unit 84 performs image analysis on the acquired moving image signal, and identifies a first partial region and a second partial region for each frame in the moving image signal.

[0185] The first partial region is a region in the exposed skin region where the subject's skin is exposed, where the heartbeat component is easily detected.

[0186] The second partial region is a region in the exposed skin region where the subject's skin is exposed, which is different from the first partial region, and is a region in which it is more difficult to detect a heartbeat component than the first partial region.

[0187] For example, the region identification unit 84 may detect the contours of the subject's face and features such as the eyes, nose, and mouth based on the video signal to identify a predetermined portion of the subject's body where the skin is exposed and identify the identified portion as a first partial region. Alternatively, the region identification unit 84 may identify a predetermined portion of the subject's body where the skin is exposed, different from the first partial region, and identify the identified portion as a second partial region. Alternatively, the region identification unit 84 may detect blood flow rates at each position in the exposed skin region of the subject based on the video signal using a predetermined algorithm, and identify a portion with a high blood flow rate as a first partial region and a portion with a blood flow rate that is a predetermined amount or a predetermined percentage lower than that of the first partial region as a second partial region.

[0188] The detection signal generation unit 88 acquires the moving image signal from the acquisition unit 82. The detection signal generation unit 88 acquires, from the region identification unit 84, information identifying the first partial region and information identifying the second partial region.

[0189] The detection signal generating unit 88 generates a first detection signal including the subject's heartbeat component and a second detection signal including less of the heartbeat component than the first detection signal, based on the signal component of the exposed skin region of the moving image signal. The first detection signal and the second detection signal are signals whose levels include the heartbeat component. Note that the second detection signal does not necessarily need to include any heartbeat component.

[0190] For example, the detection signal generation unit 88 extracts the signal component of the first partial region from the moving image signal acquired from the acquisition unit 82 based on information identifying the first partial region acquired from the region identification unit 84, and generates a first detection signal and a second detection signal based on the extracted signal component.

[0191] Alternatively, the detection signal generation unit 88 cuts out the signal component of the first partial region in the video signal acquired from the acquisition unit 82 based on the information identifying the first partial region acquired from the region identification unit 84, and generates a first detection signal based on the cut-out signal component. In this case, the detection signal generation unit 88 further cuts out the signal component of the second partial region in the video signal acquired from the acquisition unit 82 based on the information identifying the second partial region acquired from the region identification unit 84, and generates a second detection signal based on the cut-out signal component.

[0192] The differential signal generating section 40 acquires the first detection signal and the second detection signal from the detection signal generating section 88. The differential signal generating section 40 generates a differential signal that represents the difference between the first detection signal and the second detection signal.

[0193] Here, optical noise contained in the moving image signal capturing the exposed skin area is included as in-phase noise in both the first detection signal and the second detection signal. In contrast, the first detection signal contains more heartbeat components than the second detection signal. Therefore, the differential signal representing the difference between the first and second detection signals contains the optical noise contained in the signal components of the exposed skin area canceled out, leaving the heartbeat component. Therefore, the differential signal, captured by the heartbeat detection camera 18, contains less noise and is a signal from which the heartbeat component can be easily detected.

[0194] The heartbeat detection system 10 according to the sixth embodiment generates a first detection signal including a heartbeat component of the subject and a second detection signal including a smaller heartbeat component than the first detection signal, based on signal components of an exposed skin region of the subject in a moving image signal obtained by capturing an image of the subject.The heartbeat detection system 10 then generates a heartbeat signal based on a differential signal representing the difference between the first detection signal and the second detection signal.

[0195] As a result, the heartbeat detection system 10 according to the sixth embodiment can remove noise from the moving image signal obtained by capturing an image of the subject, and can detect heartbeat information with high accuracy.

[0196] Furthermore, the heartbeat detection system 10 according to the sixth embodiment extracts, from the heartbeat signal, a plurality of sets of valid vertex data that satisfy the periodicity of the heartbeat. If a set of valid vertex data is missing valid vertex data that represents a part of the time of the heartbeat cycle, the heartbeat detection system 10 generates new valid vertex data representing the missing valid vertex data by performing error correction from the set of valid vertex data, and generates a heartbeat data sequence in which the new valid vertex data is added to the set of valid vertex data.

[0197] As a result, the heartbeat detection system 10 according to the sixth embodiment can detect heartbeat information even if the heartbeat signal component contained in the differential signal is weak.

[0198] In addition, the heartbeat detection system 10 of the sixth embodiment outputs error information when it is not possible to extract a set of multiple valid peak data that satisfies the periodicity of a heartbeat from the heartbeat signal, and when the phase difference between the heartbeat data series of the first unit time and the heartbeat data series of the second unit time before the first unit time is equal to or greater than a predetermined value.

[0199] As a result, when the heartbeat information cannot be detected, the heartbeat detection system 10 according to the sixth embodiment can make the person taking the measurement or the like aware that the heartbeat information cannot be detected.

[0200] Seventh embodiment Next, a heartbeat detection system 10 according to a seventh embodiment will be described. The heartbeat detection system 10 according to the seventh embodiment has substantially the same functions and configuration as the heartbeat detection system 10 according to the sixth embodiment. Therefore, in the heartbeat detection system 10 according to the seventh embodiment, components having substantially the same functions and configurations as those in the first to sixth embodiments are denoted by the same reference numerals, and detailed description thereof will be omitted except for the differences.

[0201] FIG. 18 is a diagram showing the configuration of a heartbeat detecting device 24 according to the seventh embodiment.

[0202] The detection signal generating section 88 according to the seventh embodiment generates P types of pairs of first and second detection signals, where P is an integer of 2 or greater.

[0203] Each of the P sets of first and second detection signals has at least one of the first and second detection signals different from other sets. In this case, each of the P sets of first and second detection signals is calculated using a configuration different from other sets.

[0204] The differential signal generating section 40 according to the seventh embodiment generates a differential signal for each of P pairs of first and second detection signals. Therefore, the differential signal generating section 40 generates P differential signals. Each of the P differential signals differs from the other differential signals in at least one of the first and second detection signals.

[0205] The heartbeat signal generating section 42 according to the seventh embodiment generates a heartbeat signal for each of the P differential signals. Therefore, the heartbeat signal generating section 42 generates P heartbeat signals.

[0206] The vertex data generating section 44 according to the seventh embodiment generates a plurality of vertex data for each of the P heartbeat signals for each unit time.

[0207] The valid data extraction unit 46 according to the seventh embodiment extracts a plurality of sets of valid vertex data based on all of the plurality of vertex data of the P heartbeat signals for each unit time. The valid data extraction unit 46 according to the seventh embodiment executes the same processing as that of the fifth embodiment described with reference to FIGS. 15 and 16.

[0208] The heartbeat detection system 10 according to the seventh embodiment described above, like the sixth embodiment, can remove noise from a moving image signal capturing an image of the subject and detect heartbeat information with high accuracy. Also, like the sixth embodiment, the heartbeat detection system 10 according to the seventh embodiment can detect heartbeat information even if the heartbeat signal component included in the differential signal is weak. Also, like the sixth embodiment, the heartbeat detection system 10 according to the seventh embodiment can inform the subject or the like that heartbeat information cannot be detected when heartbeat information cannot be detected. Furthermore, the heartbeat detection system 10 according to the seventh embodiment uses P differential signals, and therefore can detect heartbeat information with higher accuracy.

[0209] Fig. 19 is a diagram showing the hardware configuration of information processing device 14. As an example, information processing device 14 is realized by a device having a hardware configuration similar to that of a general computer as shown in Fig. 19. Information processing device 14 includes a CPU (Central Processing Unit) 301, an operation device 302, a display device 303, a main memory device 305, an auxiliary memory device 306, a communication device 307, and a bus 309. Each unit is connected via bus 309.

[0210] The CPU 301 executes various processes in cooperation with various programs stored in advance in the auxiliary storage device 306, etc., using a predetermined area of ​​the main storage device 305 as a work area, and comprehensively controls the operations of the components constituting the control device 22 and the heartbeat detection device 24. The CPU 301 also operates the operation device 302, the display device 303, the communication device 307, etc. in cooperation with the programs.

[0211] The operation device 302 is an input device such as a touch panel, a mouse, or a keyboard, and receives information input by a user as an instruction signal, and outputs the instruction signal to the CPU 301. The display device 303 displays various information based on a display signal from the CPU 301.

[0212] The main storage device 305 is a volatile storage medium such as a Synchronous Dynamic Random Access Memory (SDRAM), etc. The main storage device 305 functions as a work area for the CPU 301.

[0213] The auxiliary storage device 306 is a rewritable storage device such as a semiconductor storage medium such as a flash memory, or a magnetically or optically recordable storage medium. The auxiliary storage device 306 stores programs used for control. The communication device 307 transmits and receives data to and from other devices.

[0214] The programs executed by the control device 22 and the heartbeat detection device 24 are stored on a computer connected to a network such as the Internet and are provided by being downloaded via the network. Alternatively, the programs executed by the control device 22 and the heartbeat detection device 24 may be provided by being pre-installed in a portable storage medium or the like.

[0215] The program for causing the information processing device 14 to function as the control device 22 has a modular configuration including a subject position identification module and a detection direction control module. The program may further include a cutout module. The CPU 301 reads such a program from a storage medium or the like and loads the above-mentioned modules into the main memory device 305. By executing such a program, the CPU 301 functions as the subject position identification unit 25 and the detection direction control unit 26. The CPU 301 may also function as the cutout unit 28. By executing such a program, the CPU 301 causes the main memory device 305 or the auxiliary memory device 306 to function as the storage unit 50. Note that the subject position identification unit 25, the detection direction control unit 26, and the cutout unit 28 may be partially or entirely configured by hardware.

[0216] Although the embodiments of the present invention have been described above, these embodiments are presented as examples and are not intended to limit the scope of the invention. Various modifications can be made to the embodiments. [Explanation of symbols]

[0217] 10 heartbeat detection system, 11 millimeter wave radar device, 12 movement control mechanism, 13 subject detection camera, 14 information processing device, 15 display device, 17 housing, 18 heartbeat detection camera, 22 control device, 24 heartbeat detection device, 28 cutting section

Claims

1. a sensor device that detects a sensor signal including a heartbeat component of a subject without contacting the subject; a subject detection camera that captures an image of the subject and generates an image signal; a subject position specifying unit that specifies a subject position, which is the position of the subject, from the image signal; a detection direction control unit that controls the detection direction of the sensor device in accordance with the position of the subject; a heartbeat detection device that generates a heartbeat signal representing the heartbeat component based on the sensor signal and outputs heartbeat information based on the heartbeat signal; A heart rate detection system comprising:

2. the sensor device is a millimeter wave radar device that measures the distance to the subject for each position and outputs a distance signal as the sensor signal, The detection direction control unit controls the direction of emission of electromagnetic waves from the millimeter wave radar device as the detection direction. The heartbeat detection system of claim 1 .

3. a movement control mechanism for moving the emission direction of the millimeter wave radar device; The detection direction control unit controls the movement control mechanism to control the emission direction of electromagnetic waves in the millimeter wave radar device. The heartbeat detection system of claim 2 .

4. the sensor device is a heartbeat detection camera that captures an image of the subject and outputs a moving image signal as the sensor signal, The detection direction control unit controls the direction of the angle of view of the moving image signal acquired from the heartbeat detection camera as the detection direction. The heartbeat detection system of claim 1 .

5. The heart rate detection camera has a narrower angle of view than the subject detection camera. The heartbeat detection system according to claim 4 .

6. The apparatus further includes a movement control mechanism for moving the direction of the angle of view of the heartbeat detection camera, The detection direction control unit controls the movement control mechanism to control the direction of the angle of view of the heartbeat detection camera. The heartbeat detection system according to claim 5 .

7. a cutting unit that cuts out a signal of a part of the angle of view from the moving image signal; The cutting unit changes the angle of view at which the signal is cut out depending on the position of the subject and the detection direction. The heartbeat detection system according to claim 5 .

8. The heartbeat detection device a heartbeat signal generating unit that generates the heartbeat signal based on the sensor signal; a vertex data generating unit that generates a plurality of vertex data representing times of vertices that are peak points or bottom points in the heartbeat signal; a valid data extraction unit that extracts a set of valid vertex data that satisfies the periodicity of a heartbeat from the plurality of vertex data; an error correction unit that, when a part of the sets of the plurality of valid vertex data is missing or erroneous, corrects the error in the missing or erroneous valid vertex data to generate the plurality of valid vertex data as a heartbeat data sequence; an output unit that outputs the heartbeat information based on the heartbeat data sequence; 8. The heartbeat detection system according to claim 1, comprising:

9. 1. A heartbeat detection method performed in a heartbeat detection system, comprising: The heartbeat detection system includes: a sensor device that detects a sensor signal including a heartbeat component of a subject without contacting the subject; a subject detection camera that captures an image of the subject and generates an image signal; an information processing device; Equipped with The information processing device, Identifying a subject position, which is the position of the subject, from the image signal; controlling the detection direction of the sensor device in accordance with the position of the subject; A heartbeat signal representing the heartbeat component is generated based on the sensor signal, and heartbeat information is output based on the heartbeat signal. Heart rate detection method.

10. A program for causing a computer to function as an information processing device for generating heartbeat information representing a heartbeat of a subject, The information processing device includes: acquiring a sensor signal from a sensor device that detects a sensor signal including a heartbeat component of the subject without contacting the subject; acquiring an image signal from a subject detection camera that captures an image of the subject and generates the image signal; The computer a subject position specifying unit that specifies a subject position, which is the position of the subject, from the image signal; a detection direction control unit that controls the detection direction of the sensor device in accordance with the position of the subject; a heartbeat detection device that generates a heartbeat signal representing the heartbeat component based on the sensor signal and outputs the heartbeat information based on the heartbeat signal; A program that makes it work.

Images