Heart rate estimation system and heart rate estimation method

Abstract

A heart rate estimation system and a heart rate estimation method are provided that can improve the accuracy of estimating the heart rate of a living body using a drone. [Solution] The heart rate estimation system 1 comprises a drone 6, a first camera 10 that acquires visible light images, a first signal estimation unit 62 that estimates a first heart rate signal from the images or videos captured by the first camera using the rPPG method, a second camera 12 that acquires infrared light images, a second signal estimation unit 64 that estimates a second heart rate signal from the images or videos captured by the second camera using the rPPG method, a third camera 14 that acquires spectroscopic images, and a third signal estimation unit 66 that estimates a third heart rate signal from the images or videos captured by the third camera using the rPPG method.

Description

[Technical Field]

[0001] The present invention relates to a heart rate estimation system and a heart rate estimation method. [Background technology]

[0002] Conventionally, there is known a biological information measuring device that extracts the luminance value of green light as pulse wave information and estimates blood pressure fluctuations, as shown in Patent Document 1. In such a biological information measuring device, blood pressure information of a subject is measured by acquiring an image of the subject and analyzing the received image signal. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-190022 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the technology described in Patent Document 1 assumes that images of the subject are captured and analyzed using a camera-equipped mobile phone, smartphone, tablet device, etc., and images are primarily captured from a relatively stable, handheld device. If technology such as that shown in Patent Document 1 is to be installed on a drone that can move more freely in space, the stability of the images and videos may be reduced due to the influence of the drone's shaking and air currents, or disturbances in the images and videos acquired, which could result in a problem of reduced accuracy in estimating the heart rate. Also, because the drone's position moves relatively quickly as it flies through the air, its position is likely to change relative to sunlight and lighting, and images may be affected by light from lighting, etc.

[0005] The present invention has been made to solve such problems, and aims to provide a heart rate estimation system and a heart rate estimation method that can improve the accuracy of estimating the heart rate of a living body using a drone. [Means for solving the problem]

[0006] In order to achieve the above-mentioned object, according to one embodiment of the present invention, a heart rate estimation system for estimating the heart rate of a living body comprises a drone, a first camera that acquires visible light images, a first signal estimation unit that estimates a first heart rate signal using the rPPG method from the images or videos captured by the first camera, a second camera that acquires infrared light images, a second signal estimation unit that estimates a second heart rate signal using the rPPG method from the images or videos captured by the second camera, a third camera that acquires spectroscopic images, and a third signal estimation unit that estimates a third heart rate signal using the rPPG method from the images or videos captured by the third camera. According to the embodiment of the present invention configured as described above, the heart rate of a living body can be estimated using a first camera, a second camera, a third camera, etc., mounted on a drone. The heart rate of a living body can be estimated non-contactly and easily repositioned using a drone. Furthermore, the estimated heart rate signals from different images captured by the first, second, and third cameras can be combined. Because the drone can be easily repositioned, drone-specific effects such as positional deviation, vibrations, and backlighting, which are inherent to drones, can be reduced. This allows the heart rate of a living body to be estimated non-contactly using the rPPG technique from images or video. This improves the accuracy of estimating the heart rate of a living body using a drone. For example, a heart rate estimation system for estimating the heart rate of a living body can be provided for practical applications in disaster relief, security, remote medical support, and other on-site applications.

[0007] According to one embodiment of the present invention, preferably, the method for estimating the heart rate of a living body includes a preparation step of mounting a first camera for acquiring visible light images, a second camera for acquiring infrared light images, and a third camera for acquiring spectroscopic images on a drone; a first signal estimation step of estimating a first heart rate signal using the rPPG method from an image or video captured by the first camera; a second signal estimation step of estimating a second heart rate signal using the rPPG method from an image or video captured by the second camera; and a third signal estimation step of estimating a third heart rate signal using the rPPG method from an image or video captured by the third camera. According to the embodiment of the present invention configured as described above, the heart rate of a living body can be estimated using a first camera, a second camera, a third camera, etc., mounted on a drone. The heart rate of a living body can be estimated non-contactly and easily repositioned using a drone. Furthermore, the estimated heart rate signals from different images captured by the first, second, and third cameras can be combined. Because the drone can be easily repositioned, drone-specific effects such as positional deviation, vibrations, and backlighting, which are inherent to drones, can be reduced. This allows the heart rate of a living body to be estimated non-contactly using the rPPG technique from images or video. This improves the accuracy of estimating the heart rate of a living body using a drone. For example, a heart rate estimation system for estimating the heart rate of a living body can be provided for practical applications in disaster relief, security, remote medical support, and other on-site applications. [Effects of the Invention]

[0008] According to the heart rate estimation system and heart rate estimation method of the present invention, the accuracy of estimating the heart rate of a living body using a drone can be improved. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a schematic diagram illustrating an overview of a heart rate estimation system according to an embodiment of the present invention; [Figure 2] FIG. 1 is a block diagram illustrating the connection between a drone device and a control unit of a heart rate estimation system according to an embodiment of the present invention. [Figure 3] FIG. 1 is a block diagram showing the configuration of a drone device in a heart rate estimation system according to an embodiment of the present invention. [Figure 4] 2 is a block diagram showing the configuration of a control unit of the heart rate estimation system according to the embodiment of the present invention. FIG. [Figure 5] 10 is a diagram illustrating how an integrated heartbeat signal 42 is estimated based on a first heartbeat signal, a second heartbeat signal, and a third heartbeat signal in a heart rate estimation system according to an embodiment of the present invention. FIG. [Figure 6] FIG. 2 is a flowchart illustrating a heart rate estimation method for a heart rate estimation system according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0010] A heart rate estimation system 1 according to an embodiment of the present invention will be described below with reference to the accompanying drawings. The embodiments of the present disclosure have been described as examples, and it will be apparent to those skilled in the art that many variations, modifications, and substitutions can be made within the spirit and scope of the present invention. Therefore, the present invention is not limited to the disclosed embodiments, and various variations, modifications, etc. can be made in form and details without departing from the scope of the claims. Furthermore, the components disclosed in the specification can be freely combined.

[0011] As shown in FIG. 1, a heart rate estimation system 1 according to one embodiment of the present invention has a function of estimating the heart rate of a living organism. For example, the heart rate estimation system 1 has a function of estimating the heart rate of biological information from a position separated from a living organism A, such as a human. The heart rate estimation system 1 provides a multimodal heart rate estimation system. The heart rate estimation system 1 includes a drone device 2 and a control unit 60. The heart rate estimation system 1 also functions as a biological information display system that displays biological information. The heart rate estimation method also functions as a biological information display method. In the following description of one embodiment of the present invention, the sky side of the drone device 2 of the heart rate estimation system 1 is referred to as the upper side, and the ground side of the drone device 2 is referred to as the lower side, as shown in Figure 1.

[0012] As shown in Figures 1 and 3, the drone device 2 includes a drone 6, a gimbal 9, a first camera 10, a second camera 12, a third camera 14, a first drone side altitude measuring device 20, a first drone side GPS device 22, a first drone side communication unit 24, and a first drone side control unit 16.

[0013] The drone 6 is an unmanned aerial vehicle, such as a multicopter drone, but may be another type of unmanned aerial vehicle. The drone 6 includes a main body 6a and six arms extending from the main body 6a. Each arm has a rotor 6b and blades (rotating wings) 6c for rotating the blades. By controlling the rotation speed of each blade 6c, the drone 6 can move forward and backward, left and right, and up and down. The drone 6 is configured to generate lift sufficient to fly while carrying a first camera 10, a second camera 12, and a third camera 14. In this embodiment, the drone 6 includes six arms and one blade attached to each arm (a total of six blades), but other numbers of arms and blades attached to each arm may be used. The drone 6 can fly to a predetermined location, altitude, and course, and can also take off and land automatically according to a predetermined program, controlled by a control unit 60 (described later). For example, after recognizing (detecting) living organism A, the drone 6 can fly so as to automatically maintain a predetermined distance from living organism A and maintain a predetermined angle position diagonally above living organism A. The drone device 2 may be provided with a manual operation unit 70, and all or part of the control may be manually operated by the manual operation unit. The drone 6 may also be changed to another type of flying object that can fly at any position, for example, an unmanned aerial vehicle (UAV) such as a helicopter.

[0014] The gimbal 9 is a device that stably holds the first camera 10, the second camera 12, and the third camera 14. The gimbal 9 can easily maintain a predetermined orientation by canceling out shaking, vibration, and the like. The gimbal 9 is formed so that it can rotate independently about multiple axes of rotation, for example, three axes (yaw axis, pitch axis, and roll axis). The gimbal 9 constitutes, for example, a three-axis stabilized gimbal. The gimbal 9 is provided below the air volume of the drone 6. The first camera 10, the second camera 12, and the third camera 14 are attached to the gimbal 9. It is preferable that the first camera 10, the second camera 12, and the third camera 14 are attached to the gimbal 9, but the gimbal 9 may be omitted and the first camera 10, the second camera 12, and the third camera 14 may be provided on the drone 6.

[0015] As shown in FIG. 1, the first camera 10 is mounted on the drone 6's main body 6a, allowing the drone 6 to capture and view the surroundings. The first camera 10 can acquire visible light images. The first camera 10 is, for example, an RGB camera that captures images using three color components: red, green, and blue. The first camera 10 has the ability to capture video and photographs. The first camera 10 can be used to check the surface condition of the face Aa and body Ab of the target living body A, as well as to capture and record the surface condition. The first camera 10 may also capture and record the surroundings for use in flying the drone 6. In FIG. 1, the dashed line 10a illustrates an example in which the first camera 10 is oriented so as to capture the living body A as a common target. The first camera 10 is mounted with an adjustable angle so that it can capture images from the drone 6 in the horizontal direction and diagonally downward from the horizontal direction to check the condition of the subject's face or body surface. When the first camera 10 is mounted on the same drone 6 as the second camera 12 and the third camera, the first camera 10 and the second camera 12 share the same camera body, making it easier to compare images and integrate the results of each image. The first camera 10 has, for example, a 4K (3840 x 2160) resolution and a frame rate of 60 fps. As a modified example, a third camera such as a multispectral camera may function as the first camera. As a modified example, the first camera 10, the second camera 12, and the third camera may each be mounted on the body of a different drone.

[0016] The second camera 12 is mounted on the drone 6's main body 6a and allows the drone 6 to photograph and view the surroundings. The second camera 12 can capture infrared light images. For example, the second camera 12 is an infrared camera that captures images at near-infrared (NIR) wavelengths. The second camera 12 is equipped with, for example, an 850 nm bandpass filter and captures infrared images at the near-infrared wavelength of 850 nm. The second camera 12 has the capability to capture video and photographic images. The second camera 12 converts infrared light into electrical signals using a CCD (charge-coupled device) or CMOS (complementary metal-oxide semiconductor) and reads out the electrical signals to construct an image. The second camera 12 can primarily display the thermal radiation emitted by an object at each position. The second camera 12 can be used to check the surface condition of the object's face or body using infrared light, and can also photograph and record the surface condition. The second camera 12 may also be used to photograph and record the surroundings during drone 6 flight. 1, the broken line 12a illustrates an example in which the second camera 12 is oriented so as to capture an image of the living body A as a common object. The second camera 12 has, for example, a resolution of 2048×2048 and a frame rate of 60 fps. The second camera 12 is angle-adjustable so that it can capture images from the drone 6 in the horizontal direction and diagonally downward from the horizontal to check the surface condition of the face Aa or body Ab of the target living body A. The third camera, a multispectral camera, may also function as the second camera. The second camera 12 may alternatively be a SWIR camera (InGaAs) capable of capturing short-wave infrared (SWIR) light, or a MWIR / LWIR camera (thermal infrared camera) capable of capturing mid-wave infrared (MWIR) and long-wave infrared (LWIR) light. Even under conditions such as nighttime or cloudy weather where it is difficult to capture images using the first camera 10, the second camera 12 can capture images using infrared light, which has properties different from visible light images.

[0017] The third camera 14 is provided on the drone 6's main body 6a and can photograph and view the surrounding environment from the drone 6. The third camera 14 can acquire spectral images. The third camera 14 has a function of acquiring images for each of multiple wavelengths (spectra) as spectral images. The third camera 14 is, for example, a multispectral camera that can acquire spectral images. The third camera 14 has, for example, 16 wavelength channels in the range of 450 nm to 950 nm. The third camera 14 has a function of being capable of capturing video and photographic images. The third camera 14 can acquire images of specific wavelengths using a multispectral camera. The third camera 14 has a filter for specific wavelengths and can acquire images of wavelengths with a number of bands ranging from, for example, 3 bands to 10 bands. Depending on the filter for the specific wavelength, the third camera 14 can also capture light with wavelengths such as blue, green, and red, as well as near-infrared light. The third camera 14 is configured as a line scan type or a snapshot type, but may be configured as other types. The third camera 14 is formed at a frame rate of, for example, 60 fps. The third camera 14 can mainly indicate differences in reflection and absorption characteristics at each position on the object. The third camera 14 can confirm the surface condition of the face Aa or body Ab of the target living organism A using infrared light and can also photograph and record the surface condition. The third camera 14 can also be used to photograph and record the surrounding environment for the drone 6. In FIG. 1, the dashed line 14a illustrates an example of the third camera 14 being oriented so that it can photograph the living organism A as a common target. The third camera 14 is angle-adjustable so that it can photograph the horizontal direction and a diagonal downward direction from the horizontal direction from the drone 6 to confirm the surface condition of the face Aa or body Ab of the target living organism A. Note that a multispectral camera or the like of the third camera may also function as the first camera 10 or the second camera 12. Alternatively, the third camera 14 may be a hyperspectral camera. Using a hyperspectral camera enables analysis using relatively high wavelength resolution. Similarly, third camera 14 can also acquire images etc. using spectral images that have properties different from visible light images, even under conditions such as nighttime or cloudy weather when it is difficult for first camera 10 to capture images.

[0018] The first camera 10, the second camera 12, and the third camera 14 are positioned in such a direction that they can simultaneously capture images of a common object, such as living body A (the capture timing is synchronized). Because the common object, such as living body A, can be captured simultaneously in this manner, even if some images are poor due to lighting conditions such as weather, the heart rate of living body A can be estimated with relatively high accuracy. The first camera 10, the second camera 12, and the third camera 14 are positioned on a common horizontal plane and are offset horizontally by a predetermined angle, for example, approximately 15 degrees. When the first camera 10 is at a position of, for example, 0 degrees, the second camera 12 is positioned approximately 15 degrees to the left of the first camera 10, and the third camera 14 is positioned approximately 15 degrees to the right of the first camera 10. The first camera 10, the second camera 12, and the third camera 14 are positioned such that the distance between their optical axes is, for example, within a range of 20 cm to 25 cm. For example, such an arrangement makes it possible to simultaneously capture images of the same subject from different angles, thereby improving resistance to disturbances caused by the parallax effect. The first camera 10, the second camera 12, and the third camera 14 are positioned so that relatively clear images and videos can be obtained relative to a common virtual plane. Therefore, it is possible to simultaneously capture images of a single object using the three cameras and use the captured images. The first camera 10, the second camera 12, and the third camera 14 constitute a multimodal video measurement module. The first camera 10, the second camera 12, and the third camera 14 may be fixed to a common rigid frame. By being fixed to a common rigid frame, shaking, vibration, and the like are more likely to be shared, making it easier to perform common operations.

[0019] The first drone altitude measurement device 20 is provided on the airframe main body 6a and can measure the altitude H (distance) of the drone 6 relative to the ground G on which the target living body A is standing. The first drone altitude measurement device 20 uses, for example, an ultrasonic altimeter that can measure the height to the ground G. The first drone altitude measurement device 20 may be configured with any one of a barometric pressure measurement sensor that can measure flight altitude by measuring air pressure, an ultrasonic sonar that can measure the distance from the drone 6 to the ground G, a laser measurement sensor that can measure the distance from the drone 6 to the ground G, a LIDAR sensor that can measure the distance from the drone 6 to the ground G, or any combination thereof. This allows the first drone altitude measurement device 20 to measure the altitude H (distance) from the drone 6 to the ground G. For example, the first drone altitude measurement device 20 can measure the altitude (distance) H from the drone 6 to the ground G within a predetermined distance range of 1 m to 10 m, more preferably within a range of 1 m to 3 m, and more preferably within a range of 1 m to 2 m, allowing the drone 6 to fly at the predetermined altitude H. The altitude range in which the drone 6 is flown can be, for example, an altitude range of 1 m to 10 m, which is a predetermined distance from the drone 6 to the ground G, more preferably an altitude range of 1 m to 3 m, and more preferably an altitude range of 1 m to 2 m.

[0020] The first drone GPS device 22 is capable of identifying the current position of the drone 6 using satellites. The first drone GPS device 22 can acquire position information (e.g., information such as latitude and longitude) of the locations where the first camera 10, the second camera 12, and the third camera 14 captured images of the living body A. Furthermore, the first drone GPS device 22 can recognize the position of the drone 6 and provide the position information necessary for predetermined flight control of the drone 6. The first drone GPS device 22 is equipped with, for example, a GPS navigation module.

[0021] The first drone communication unit 24 can wirelessly communicate data from the drone device 2 with the control unit 60. For example, the first drone communication unit 24 can transmit information such as the position (coordinates, altitude) of the drone 6, and the positions where the first camera 10, the second camera 12, and the third camera 14 captured images of the living body A to the control unit 60. In addition, the first drone communication unit 24 can mutually share control information with the control unit 60.

[0022] The drone device 2 may be equipped with a manual operation unit 70, a monitor 72 for the operation unit 70, etc. as necessary.

[0023] The first drone control unit 16 has a built-in CPU 17 and a storage device 19 such as memory, and controls connected devices to execute predetermined controls based on predetermined control programs stored in the memory, etc. The first drone control unit 16 is electrically connected to the drone 6, first camera 10, second camera 12, third camera 14, first drone camera 11, first drone altitude measuring device 20, first drone GPS device 22, first drone communication unit 24, control unit 60, etc. These electrical connections may be made via wireless communication, etc.

[0024] The first drone control unit 16 can execute flight control of the drone 6. The first drone control unit 16 is configured to perform predetermined functions in cooperation with the control unit 60. The first drone control unit 16, together with the control unit, controls the drone device 2 and the flight of the drone 6. More specifically, the first drone control unit 16 can control the positions (coordinates, altitude) at which the first camera 10, the second camera 12, and the third camera 14 capture images, attitude control, yawing rotation suppression control, movement between spraying points, and the like. In this way, the first drone control unit 16 can control the flight altitude, flight route, rotation speed of each blade, attitude (including left and right roll and yawing in the rotational direction), and operation control of the first camera 10, the second camera 12, and the third camera 14 as necessary. The first drone control unit 16 can control the drone 6 to reach a predetermined altitude above the target point (search point) and to capture images of the living organism A using the first camera 10, the second camera 12, and the third camera 14. The first drone-side control unit 16 may be provided integrally with the control unit 60. For example, all or part of the functions of the first drone-side control unit 16 may be provided on the control unit 60 side. All or part of the functions of the first drone-side control unit 16 may be provided in an information terminal device or the like on the operation unit 70 side.

[0025] As shown in Figures 1 to 4, the control unit 60 is configured to perform control such as estimating the heart rate of living body A using a first camera 10, a second camera 12, and a third camera 14 mounted on a drone 6. The control unit 60 includes a face detection unit 61, a first signal estimation unit 62, a second signal estimation unit 64, a third signal estimation unit 66, an evaluation unit 68, an integration unit 69, a heart rate estimation unit 75, and a correction function unit 76. The control unit 60 also controls, for example, the drone device 2. The control unit 60 is provided in, for example, a computer located away from the drone device 2.

[0026] As shown in FIG. 2, the control unit 60 is electrically connected to the drone device 2 via the Internet 3. The control unit 60 may be provided in an electronic device that functions as a computer, such as a smartphone or tablet. The control unit 60 has a built-in CPU 63 (see FIG. 1) and a storage device 65 such as a memory, and controls connected devices based on a predetermined control program recorded in the memory. Thus, the control unit 60 functions as a computer. The electrical connection between the control unit 60 and other devices may be entirely or partially wirelessly connected via infrared communication or other methods. The control unit 60 may also include an on-board AI processor for processing video signals in real time. The control unit 60 has a predetermined program for executing a predetermined control function. The control unit 60 may also be composed of multiple devices. The storage device 65 of the control unit 60 stores a predetermined program, but it is not necessarily required to store all of the program. Some or all of the program may be stored separately in multiple devices or on a server via the Internet. For example, a first drone-side control unit 16 mounted on the drone device 2 may be configured to execute some or all of the control functions. The control unit 60 includes an output device 71 such as a monitor and an input device 67 that can be operated to input data, and is capable of setting various modes and the like.

[0027] The control unit 60 includes a face detection unit 61. The control unit 60 can operate the computer as the face detection unit 61, which detects a living body, such as a human face, from an image or video captured by the first camera 10, using various programs stored in the storage device 65. The face detection unit 61 can detect the location of a face in an image, as well as its contours and structure. For example, the face detection unit 61 may be configured with a program that executes a convolutional neural network (CNN) model. For example, the face detection unit 61 may be configured by executing a face detection program such as MediaPipe. The face detection unit 61 has the function of, for example, detecting a face region in real time, calculating the average pixel value of the face region, and generating a time-series signal. Alternatively, the face detection unit 61 may be configured with a program that executes a Haar feature classifier or other technology such as MTCNN (Multi-task Cascaded CNN). The face detection unit 61 may be provided by a program that executes a face recognition function.

[0028] The control unit 60 may cause the computer to function as a synchronization control unit that synchronizes the shooting timing of the first camera 10, the second camera 12, and the third camera 14 to shoot at the same time, using the respective programs stored in the storage device 65. The synchronization control unit can realize, for example, a synchronization trigger control module.

[0029] The control unit 60 includes a first signal estimation unit 62 , a second signal estimation unit 64 , and a third signal estimation unit 66 . The control unit 60, using the programs stored in the storage device 65, can cause the computer to function as a first signal estimation unit 62 that estimates a first heartbeat signal using the rPPG method (rPPG means) from images or videos captured by the first camera 10. The control unit 60 stores a program for realizing this function in the storage device. The rPPG (remote photoplethysmography) method is a technical technique for measuring a person's biometric information, such as heartbeat, without contact. More specifically, the first signal estimation unit 62 extracts subtle changes in RGB components from RGB images or videos captured by the first camera and estimates the heartbeat based on these subtle changes. To extract biometric information from the RGB camera images or videos, the CHROM (Chrominance-based remote Photoplethysmography) method is used. The CHROM method is composed of an algorithm (program) that extracts a heartbeat signal from images or videos based on changes in skin color (chrominance). For example, since the skin color (e.g., red, green, and blue components) changes as the blood volume changes due to the heartbeat, the CHROM method of rPPG has a mechanism for estimating the heartbeat based on such changes. A program for executing the CHROM method is also stored in the storage device of the control unit 60. Therefore, the first signal estimation unit 62 can estimate the first heartbeat signal by rPPG from the image or video captured by the first camera 10.

[0030] The control unit 60, using the respective programs stored in the storage device 65, can cause the computer to function as a second signal estimation unit 64 that estimates a second heartbeat signal by rPPG from images or video captured by the second camera 12. The control unit 60 stores a program for realizing this function in the storage device. Mounting the first camera 10, second camera 12, third camera 14, etc. on the drone 6 significantly improves non-contact and freedom of image capture position. On the other hand, mounting the first camera 10, second camera 12, third camera 14, etc. on the drone 6 may result in a decrease in the quality of the images or video due to misalignment of the drone 6, vibrations such as shaking, or the effects of the drone being positioned in a backlit or shaded area, which may temporarily reduce the accuracy of heartbeat estimation based on the images. Therefore, to reduce the impact of a decrease in the quality of the RGB images captured by the first camera 10, the heartbeat signal by rPPG is also estimated from images captured by the second camera 12, third camera 14, etc.

[0031] The second signal estimation unit 64 estimates a second heartbeat signal by rPPG from an image or video captured by the second camera 12. The image or video captured by the second camera 12 is, for example, an infrared image or video. rPPG (remote photoplethysmography) is a technology for non-contact measurement of a person's biometric information, such as heartbeat. More specifically, the second signal estimation unit 64 extracts subtle changes in the shading (light and dark) of the infrared image from the infrared image or video captured by the second camera 12 and estimates the heartbeat based on these subtle changes. The shading (light and dark) of the infrared image indicates the intensity of infrared radiation (e.g., the amount of thermal radiation) emitted by the object for each pixel. This shading (light and dark) can also be said to indicate differences in temperature (radiant energy) at each location. In this case, the Chrominance-based remote Photoplethysmography (CHROM) method is used to extract biometric information from the infrared camera image or video. The CHROM method is configured with an algorithm (program) that extracts a heartbeat signal from an image or video based on changes in skin (chrominance) color, for example, changes in the shading (light and dark) of an infrared image. For example, since changes in blood volume due to heartbeat cause changes in skin color (for example, the shading (light and dark) component of an infrared image), the CHROM method of rPPG has a mechanism for estimating the heartbeat based on such changes. The program that executes the CHROM method is also stored in the storage device of the control unit 60. Therefore, the second signal estimation unit 64 can estimate a second heartbeat signal by rPPG from the infrared image or video captured by the second camera 12.

[0032] The third signal estimation unit 66 estimates the third heartbeat signal using the rPPG technique from the image or video captured by the third camera 14. The image or video captured by the third camera 14 is, for example, a spectroscopic image or video. The rPPG (remote photoplethysmography) technique is a technical technique for non-contact measurement of a person's biometric information, such as heartbeat. More specifically, the third signal estimation unit extracts subtle changes in the shading (light and dark) of the spectroscopic image captured by the third camera 14 and estimates the heartbeat based on these subtle changes. The spectroscopic image or video is, for example, a spectral image or a hyperspectral image, and is, for example, a single-band image extracted from a specific wavelength. The shading (light and dark) of the spectroscopic image indicates the level of reflection intensity at a specific wavelength. For example, a relatively bright area indicates a relatively high reflectance, while a relatively dark area indicates a low reflectance. The spectroscopic image or video indicates differences in the reflection and absorption characteristics of an object at each position in a certain wavelength band. The spectral image or video may be displayed as a pseudocolor image in which multiple bands are assigned to RGB. In this case, the Chrominance-based Remote Photoplethysmography (CHROM) method is used to extract biological information from the spectral image or video captured by a hyperspectral camera or the like. The CHROM method is implemented using an algorithm (program) that extracts a heartbeat signal from the image or video based on changes in skin (chrominance) color, e.g., the shading (light / dark) of the spectral image. For example, changes in blood volume due to heartbeat change the reflectance at each location on the skin (e.g., the shading (light / dark) component of the spectral image). The rPPG CHROM method has a mechanism for estimating the heartbeat based on such changes. The program for executing the CHROM method is also stored in the storage device of the control unit 60. Therefore, the third signal estimation unit 66 can estimate the third heartbeat signal using the rPPG technique from the spectral image or video captured by the third camera 14. Alternatively, multiple time-series signals obtained from each channel of the third camera 14 may be added together, and signals from a total of 20 channels may be processed using the ICA method to extract statistically independent components (independent components: ICs). Of the multiple independent components obtained in this way, the component with the maximum SNR in the heart rate band (0.75 Hz to 3 Hz) can be selected as the heart rate signal.

[0033] The control unit 60, using the respective programs stored in the storage device 65, can cause the computer to function as an evaluation unit 68 that evaluates the first heartbeat signal 30, the second heartbeat signal 32, and the third heartbeat signal 34 estimated using the rPPG technique. The first heartbeat signal 30, the second heartbeat signal 32, and the third heartbeat signal 34 are acquired as time-series signals that are a collection of signals that change over time. The control unit 60 then performs weighted evaluation of the first heartbeat signal 30, the second heartbeat signal 32, and the third heartbeat signal 34 based on the evaluation score output by the evaluation unit 68. The evaluation unit 68 assigns an evaluation score to each signal based on the frequency intensity (SNR: signal-to-noise ratio) of the first heartbeat signal 30, the second heartbeat signal 32, and the third heartbeat signal 34. For example, the evaluation unit 68 assigns a first evaluation score 35 to the first heartbeat signal 30, a second evaluation score 36 to the second heartbeat signal 32, and a third evaluation score 37 to the third heartbeat signal 34. For example, a larger ratio of signal to noise power (intensity) results in a larger score value, and a smaller ratio results in a smaller score value. The evaluation unit 68 may also assign an evaluation score to each of the first heartbeat signal 30, the second heartbeat signal 32, and the third heartbeat signal 34 based on their signal periodicity. Signal periodicity indicates whether or not a signal repeats the same pattern at regular time intervals, and the degree of periodicity. For example, if the value indicating the periodicity is large, a large score value is assigned to the first evaluation score, etc., and if the value indicating the periodicity is small, a small score value is assigned to the first evaluation score, etc. Furthermore, the evaluation unit 68 may assign an evaluation score to each signal based on the stability of the face in the image when the first heartbeat signal 30, the second heartbeat signal 32, and the third heartbeat signal 34 were acquired. If the face was shaking when the image was acquired, the evaluation accuracy may be reduced, so for example, if the stability of the face is low, a small score value is assigned to the first evaluation score, etc., and if the stability is high, a large score value is assigned to the first evaluation score, etc. In addition, under circumstances where it is difficult to obtain an image by the first camera 10 or the like, such as at night, the evaluation unit 68 can lower the evaluation of the first heartbeat signal 30 and perform processing to estimate the heart rate of the living body using other signals. In this way, it is possible to provide a heart rate estimation system that estimates the heart rate of the living body even under circumstances where it is difficult to obtain some heartbeat signals or the like. The evaluation unit 68 may assign an evaluation score to each signal based on other indicators such as a coefficient indicating the communication quality of the communication signal when the image was acquired.

[0034] 5, the control unit 60 calculates a first evaluation heartbeat signal 38 by, for example, taking into account a first evaluation score 35 as a weighting evaluation from a first heartbeat signal 30 estimated using the rPPG technique. For example, the control unit 60 multiplies the first heartbeat signal 30 by the first evaluation score 35 to calculate the first evaluation heartbeat signal 38. For example, the control unit 60 calculates a second evaluation heartbeat signal 39 by, for example, taking into account a second evaluation score 36 as a weighting from a second heartbeat signal 32 estimated using the rPPG technique. For example, the control unit 60 multiplies the second heartbeat signal by the second evaluation score 36 to calculate the second evaluation heartbeat signal 39. For example, the control unit 60 calculates a third evaluation heartbeat signal 40 by, for example, taking into account a third evaluation score 37 as a weighting from a third heartbeat signal 34 estimated using the rPPG technique. For example, the control unit 60 multiplies the third heartbeat signal by the third evaluation score 37 to calculate the third evaluated heartbeat signal 40. In this way, the control unit 60 can perform a weighted evaluation of the first heartbeat signal 30, the second heartbeat signal 32, and the third heartbeat signal 34.

[0035] The control unit 60 can cause the computer to function as an integrating unit 69 that estimates an integrated heartbeat signal based on the first heartbeat signal 30, the second heartbeat signal 32, and the third heartbeat signal 34, using the respective programs stored in the storage device 65. The integrating unit 69 estimates the integrated heartbeat signal 42 based on the first evaluated heartbeat signal 38, the second evaluated heartbeat signal 39, and the third evaluated heartbeat signal 40 that are based on the first heartbeat signal 30, the second heartbeat signal 32, and the third heartbeat signal 34. However, the evaluation by the evaluator 68 may be omitted, and the integrated heartbeat signal 42 may be estimated based on the first heartbeat signal 30, the second heartbeat signal 32, and the third heartbeat signal 34.

[0036] The integration unit 69 estimates an integrated heartbeat signal 42 by weighting and combining the first heartbeat signal 30, the second heartbeat signal 32, and the third heartbeat signal 34. The integration unit 69 estimates an integrated heartbeat signal by summing and averaging the first evaluation heartbeat signal 38, which takes the first evaluation score 35 into consideration as a weighting factor, the second evaluation heartbeat signal 39, which takes the second evaluation score 36 into consideration as a weighting factor, and the third evaluation heartbeat signal 40, which takes the third evaluation score 37 into consideration as a weighting factor. The integration unit 69 aligns the time axes of the first evaluation heartbeat signal 38, the second evaluation heartbeat signal 39, and the third evaluation heartbeat signal 40, for example, by aligning the time axes so that their start times are the same, and performs integration processing using independent component analysis (ICA). The integration unit 69 extracts (calculates) an integrated heartbeat signal as an independent component from the first evaluation heartbeat signal 38, the second evaluation heartbeat signal 39, and the third evaluation heartbeat signal 40, for example. Therefore, the integrating unit 69 can obtain a single integrated heartbeat signal with improved accuracy. Alternatively, the integrating unit 69 may align the time axes of the first evaluation heartbeat signal 38, the second evaluation heartbeat signal 39, and the third evaluation heartbeat signal 40, and then calculate the integrated heartbeat signal relatively simply by averaging the sum of the three signals. By combining signals from three different images, the accuracy of estimating the heart rate of a living body can be improved. In addition, even if some signals are defective or missing, the accuracy of the heart rate estimation can be prevented from decreasing. Furthermore, by integrating signals based on images with three different characteristics, the accuracy of estimating the heart rate of a living body can be further improved.In addition, even if some signals based on a specific characteristic are defective or missing, the accuracy of estimating the heart rate can be prevented from decreasing. Furthermore, by noting that the waveform of a living body's heartbeat is a periodic sequence of relatively similar waveforms, and by integrating signals based on three different images, the signal corresponding to the heartbeat can be effectively complemented, making effective use of the complementary relationship.

[0037] The control unit 60 can execute various programs stored in the storage device 65 to cause the computer to function as a heart rate estimation unit 75 that estimates the heart rate by frequency analyzing the integrated heart rate signal 42. The heart rate estimation unit 75 of the control unit 60 estimates the heart rate by frequency analyzing the integrated heart rate signal, which is obtained as a time-series signal, and extracting frequency components (periodic components). For example, frequency analysis can be performed using mathematical techniques such as Fourier transform (FT), fast Fourier transform (FFT), and continuous wavelet transform (CWT). By frequency analyzing the integrated heart rate signal, components that are periodically added in accordance with the heart rate, such as frequency components, are analyzed and the heart rate value is estimated. By frequency analyzing the integrated heart rate signal 42, which is obtained by integrating each signal, the heart rate can be estimated with relatively high accuracy. Note that the heart rate estimation unit 75 may also estimate the heart rate using techniques other than frequency analysis, such as an AI program directly estimating the heart rate from changes in a captured image.

[0038] The control unit 60 can cause the computer to function as a correction function unit 76 that corrects the integrated heart rate signal by using the programs stored in the storage device 65. The AI ​​program realizes a function of predicting and correcting the integrated heartbeat signal based on its temporal dependency when it determines that a portion of the integrated heartbeat signal is missing, defective, or irregular. The control unit 60, for example, further includes a program implementing a long short-term memory (LSTM) function in addition to the AI ​​program, which can predict the temporal dependency of the integrated heartbeat signal and recognize deviations of the actual signal from the prediction. The LSTM function in the AI ​​function allows the control unit 60 to distinguish between input and output values ​​and values ​​to be included or discarded for the prediction of the integrated heartbeat signal 42 or the acquired signal of the integrated heartbeat signal 42. Through this mechanism, the correction function unit 76 of the control unit 60 can realize a function of correcting defective or other values ​​in the integrated heartbeat signal 42 to the predicted value. The LSTM function in the AI ​​function of the control unit 60 may also function as a correction function unit that corrects partial signal defects, etc., to predicted values ​​for the first heartbeat signal 30, the second heartbeat signal 32, the third heartbeat signal 34, etc., the first evaluation heartbeat signal 38, the second evaluation heartbeat signal 39, the third evaluation heartbeat signal 40, etc. Because the correction function unit 76 can correct individual signals, corrections can be made more easily when defects, etc., occur in individual signals, resulting in improved accuracy in estimating the overall heart rate.

[0039] The correction function unit of the control unit 60 may, for example, include a program that realizes the AI ​​function and also includes a program that realizes the function of a Transformer model. The Transformer model function has a function of learning and correcting the time-series relationship of time-series signal data. For example, the Transformer model learns the time-series relationship of the integrated heartbeat signal and corrects noise due to lighting changes, momentary signal loss, distortion due to movement or shaking, etc. based on the learned time-series relationship of the integrated heartbeat signal. In this way, partial loss or defects of the signal are corrected based on the time-series relationship of the integrated heartbeat signal. With this mechanism, the correction function unit of the control unit 60 can realize a function of correcting partial defects of the integrated heartbeat signal to predicted values. Note that the Transformer model function in the AI ​​function of the control unit 60 may also realize a function of correcting partial defects of the signal to predicted values ​​for the first heartbeat signal, the second heartbeat signal, the third heartbeat signal, the first evaluation heartbeat signal, the second evaluation heartbeat signal, the third evaluation heartbeat signal, etc. As a modified example, the correction function unit 76 may be omitted to simplify and speed up the overall processing.

[0040] In addition, the control unit 60 may function as a tracking function unit that, after recognizing (detecting) living organism A, automatically flies the drone 6 to maintain a predetermined distance from living organism A and / or maintain a predetermined angle position diagonally above living organism A, using each program stored in the memory device 65.

[0041] Next, a series of operations of a heart rate estimation method for estimating the heart rate of a living body will be described as shown in FIG. As shown in Fig. 6, in preparation step S1 of the heart rate estimation system 1, the drone device 2, first camera 10, second camera 12, third camera 14, control unit 60, and the like of the heart rate estimation system 1 are prepared. The first camera 10, second camera 12, and third camera 14 are prepared so that they can perform a series of functions while mounted on the drone 6. Furthermore, the control unit 60 prepares or acquires, if necessary, flight data for the drone device 2, such as coordinates and flight route, and flight altitude data (detected altitude data) relative to the ground G. When step S1 is completed, the control unit 60 proceeds to S2.

[0042] In step S2, the control unit 60 executes a detection step of detecting the face area of ​​the subject. The face Aa of the subject, living body A, is recognized by the first camera 10, and the location of the face area in the image is detected. When detecting the face area, images from the second camera 12, the third camera 14, etc. may be used. When step S2 is completed, the control unit 60 proceeds to S3.

[0043] In step S3, the control unit 60 executes an image acquisition step in which the first camera 10, second camera 12, and third camera 14 mounted on the drone 6 acquire an image or video of the facial region of the subject, living organism A. The first camera 10, second camera 12, and third camera 14 operate approximately simultaneously to acquire an image of the face Aa of living organism A. For example, the first camera 10, second camera 12, and third camera 14 start capturing an image or video of the face Aa at synchronized timing. Thus, the control unit 60 can simultaneously acquire a visible light image captured by the first camera 10, an infrared light image captured by the second camera 12, and a spectral image captured by the third camera 14, all of which are captured at synchronized timing. The control unit 60 has a function to synchronize the first camera 10, second camera 12, and third camera 14 to capture images at the same time by, for example, executing a program that realizes a synchronization trigger control module. The start of capture does not need to be completely simultaneous as long as the images are captured at the same time. Furthermore, since the first camera 10, the second camera 12, and the third camera 14 capture images of different characteristics, the heart rate of the living body can be estimated even if the images from some of the cameras are poor. Simultaneous imaging of the same subject, living body A, from different angles is possible, and the parallax effect makes it difficult for the accuracy of the heart rate estimation to decrease even if the subject is shaking. After step S3 is completed, the control unit 60 proceeds to S4.

[0044] In step S4, the control unit 60 executes a first signal estimation step in which the first signal estimation unit 62 estimates the first heartbeat signal 30 by the rPPG technique from the image or video captured by the first camera 10. The control unit 60 detects minute image changes, such as color changes, of the skin surface of the face, etc., from the image captured by the first camera 10, and estimates changes in blood flow and the first heartbeat signal 30 corresponding to the changes in blood flow. When step S4 ends, the control unit 60 proceeds to S5.

[0045] In step S5, the control unit 60 executes a second signal estimation step in which the second signal estimation unit 64 estimates the second heartbeat signal 32 by the rPPG technique from the image or video captured by the second camera 12. The control unit 60 detects minute changes in the image of the skin surface of the face, etc., from the image captured by the second camera 12, and estimates changes in blood flow and the second heartbeat signal 32 corresponding to the changes in blood flow. When step S5 ends, the control unit 60 proceeds to S6.

[0046] In step S6, the control unit 60 executes a third signal estimation step in which the third signal estimation unit 66 estimates the third heartbeat signal 34 by the rPPG technique from the image or video captured by the third camera 14. The control unit 60 detects minute image changes, such as changes in brightness, on the skin surface of the face, etc., from the image captured by the third camera 14, and estimates changes in blood flow and the third heartbeat signal 34 corresponding to the changes in blood flow. When step S6 ends, the control unit 60 proceeds to S7.

[0047] In step S7, the control unit 60 executes an evaluation step in which the evaluation unit 68 evaluates the first heartbeat signal 30, the second heartbeat signal 32, and the third heartbeat signal 34 estimated using the rPPG technique. In the evaluation step, the control unit 60 calculates an evaluation score for each of the first heartbeat signal 30, the second heartbeat signal 32, and the third heartbeat signal 34 based on the frequency intensity and the like of the signals, and assigns the score to each signal as shown in FIG. 5. The evaluation unit 68 can weight the importance of the first heartbeat signal 30, the second heartbeat signal 32, and the third heartbeat signal 34 in consideration of the influence of the environment when the first heartbeat signal 30, the second heartbeat signal 32, and the third heartbeat signal 34 were acquired, as well as the signal intensity, stability, and the like, and reflect the weighted importance in the evaluation and the signals. By executing the evaluation step S7, the control unit 60 can acquire the first evaluation heartbeat signal 38, the second evaluation heartbeat signal 39, and the third evaluation heartbeat signal 40. The control unit 60 may omit the processing by the evaluation unit 68 and integrate the first heartbeat signal 30, the second heartbeat signal 32, and the third heartbeat signal 34 in an integration step S8, which will be described later. After step S7 is completed, the control unit 60 proceeds to S8.

[0048] In step S8, as shown in FIG. 5 , the control unit 60 executes an integration step in which the integration unit 69 estimates an integrated heartbeat signal 42 based on the first heartbeat signal 30, the second heartbeat signal 32, and the third heartbeat signal 34. In the integration step, the control unit 60 executes an integration step in which the integration unit 69 estimates an integrated heartbeat signal from the first evaluated heartbeat signal 38, the second evaluated heartbeat signal 39, and the third evaluated heartbeat signal 40. In the integration step, the integration unit 69 temporally synchronizes the first evaluated heartbeat signal 38, the second evaluated heartbeat signal 39, and the third evaluated heartbeat signal 40, and estimates the integrated heartbeat signal 42 based on the processing of the integration unit 69. Note that the processing of the integration unit 69 can be changed to a method that can estimate the integrated heartbeat signal 42, which is assumed to represent an actual heartbeat, from the first evaluated heartbeat signal 38, the second evaluated heartbeat signal 39, and the third evaluated heartbeat signal 40 in any reasonable manner, in addition to a method such as averaging. After step S8 is completed, the control unit 60 proceeds to S9.

[0049] In step S9, the control unit 60 executes a heart rate estimation step in which the heart rate estimation unit 75 estimates the heart rate by frequency analyzing the integrated heart rate signal 42. The heart rate estimation unit 75 of the control unit 60 estimates the heart rate by frequency analyzing the integrated heart rate signal 42, which is obtained as a time-series signal, and extracting frequency components (periodic components). This allows the heart rate to be estimated based on the integrated heart rate signal 42 acquired from multiple images, thereby improving the accuracy of the heart rate estimation. The control unit 60 can display the acquired heart rate on the monitor 72 of the operation unit 70 or transmit it as data to another application. When step S9 is completed, the control unit 60 proceeds to END. If the drone 6 is not to return due to the operation of the operation unit 70, the control unit 60 automatically controls the drone 6 to move to the return position and ends the series of processes.

[0050] Examples of an embodiment of the present invention may be provided in each aspect as described below.

[0051] (1) A heart rate estimation system for estimating the heart rate of a living body, comprising: a drone; a first camera that acquires visible light images; a first signal estimation unit that estimates a first heart rate signal from the images or videos captured by the first camera using the rPPG method; a second camera that acquires infrared light images; a second signal estimation unit that estimates a second heart rate signal from the images or videos captured by the second camera using the rPPG method; a third camera that acquires spectroscopic images; and a third signal estimation unit that estimates a third heart rate signal from the images or videos captured by the third camera using the rPPG method.

[0052] (2) A heart rate estimation system as described in (1), wherein the first camera, the second camera, and the third camera are oriented so as to photograph a common object at the same time.

[0053] (3) The heart rate estimation system described in (1) includes an evaluation unit that evaluates the first heart rate signal, the second heart rate signal, and the third heart rate signal, and performs a weighted evaluation of the first heart rate signal, the second heart rate signal, and the third heart rate signal based on the evaluation score output by the evaluation unit.

[0054] (4) The heart rate estimation system according to (1), further comprising an integration unit that estimates an integrated heart rate signal based on the first heart rate signal, the second heart rate signal, and the third heart rate signal.

[0055] (5) The heart rate estimation system according to (4), further comprising a heart rate estimation unit that estimates the heart rate by performing frequency analysis on the integrated heart rate signal.

[0056] (6) The heart rate estimation system according to (5), further comprising a correction function unit that corrects the first heart rate signal, the second heart rate signal, the third heart rate signal, or the integrated heart rate signal.

[0057] (7) The heart rate estimation system described in (1) includes a gimbal provided on the drone, and the first camera, the second camera, and the third camera are attached to the gimbal.

[0058] (8) A heart rate estimation system as described in (1), wherein the first camera, the second camera 12, and the third camera are positioned on a common horizontal plane and are arranged at angles offset by a predetermined angle in the horizontal direction.

[0059] (9) A heart rate estimation method for estimating the heart rate of a living body, comprising: a preparation step of mounting a first camera that acquires visible light images, a second camera that acquires infrared light images, and a third camera that acquires spectroscopic images on a drone; a first signal estimation step of estimating a first heart rate signal using the rPPG method from an image or video captured by the first camera; a second signal estimation step of estimating a second heart rate signal using the rPPG method from an image or video captured by the second camera; and a third signal estimation step of estimating a third heart rate signal using the rPPG method from an image or video captured by the third camera.

[0060] The embodiments for carrying out the present invention are not limited to the above, and other modifications may be applied. Various alternative embodiments and examples will be apparent to those skilled in the art based on the disclosed technology. In this embodiment, the control unit 60 estimates the heart rate in real time from the images captured (image acquisition) by the first camera 10, the second camera 12, and the third camera 14. In contrast, as a modified example, an image or video may be captured for a predetermined time, for example, within a range of 5 to 30 seconds, and the image captured for the predetermined time may be analyzed to estimate the heart rate. Even when there are restrictions on the flight of the drone 6, the heart rate can be estimated within a limited time. The heart rate of living organism A can be measured by measuring for any time.

[0061] In this embodiment, the drone is equipped with a first camera that acquires visible light images, a second camera that acquires infrared light images, and a third camera that acquires spectral images. However, as a modified example, the configuration of the first camera, second camera, and third camera can be changed as desired. For example, the camera that acquires spectral images may be the second camera, and a third camera that acquires spectral images may also be mounted. Furthermore, for example, the drone may be equipped with a first camera that acquires visible light images, a second camera that acquires visible light images, and a third camera that acquires infrared light images. Furthermore, as a modified example, the number of cameras is not limited to three and may be reduced. For example, a multispectral camera or a hyperspectral camera may be used as the first camera, and visible light images, infrared light images, and spectral images may be acquired by only the first camera. As a modified example, the number of cameras is not limited to three and may be increased. For example, the drone may be equipped with a first camera that captures visible light images, a second camera that captures infrared light images, a third camera that captures spectral images, and a fourth camera that captures visible light images and is positioned differently from the first camera.

[0062] As another modified example, the present invention can also function as a biological information estimation system or biological information estimation method that estimates not only the heart rate of a living body but also other biological information, such as respiratory rate and heart rate variability, based on rPPG technology means. [Explanation of symbols]

[0063] 1: Heart rate estimation system 6: Drone 9: Gimbal 10: First camera 12: Second camera 14: Third camera 30: First heartbeat signal 32: Second heartbeat signal 34: Third heartbeat signal 42: Integrated heart rate signal 62: First signal estimation unit 64: Second signal estimation unit 66: Third signal estimation unit 68: Evaluation section 69: Integration Department A: Living organism

Claims

1. A heart rate estimation system for estimating a heart rate of a living body, comprising: Drones and a first camera for capturing a visible light image; a first signal estimation unit that estimates a first heartbeat signal from an image or video captured by the first camera using an rPPG technique; a second camera for capturing an infrared image; a second signal estimation unit that estimates a second heartbeat signal from the image or video captured by the second camera using an rPPG technique; a third camera for acquiring a spectral image; A heart rate estimation system comprising: a third signal estimation unit that estimates a third heart rate signal using an rPPG technique from an image or video captured by the third camera.

2. The heart rate estimation system according to claim 1 , wherein the first camera, the second camera, and the third camera are arranged in such orientations that they can simultaneously capture an image of a common object.

3. 2. The heart rate estimation system according to claim 1, further comprising an evaluation unit that evaluates the first heart rate signal, the second heart rate signal, and the third heart rate signal, and performs weighted evaluation of the first heart rate signal, the second heart rate signal, and the third heart rate signal based on evaluation scores output by the evaluation unit.

4. The heart rate estimation system according to claim 1 , further comprising an integration unit that estimates an integrated heart rate signal based on the first heart rate signal, the second heart rate signal, and the third heart rate signal.

5. The heart rate estimation system according to claim 4 , further comprising a heart rate estimation unit that estimates the heart rate by performing frequency analysis on the integrated heart rate signal.

6. The heart rate estimation system according to claim 5 , further comprising a correction function unit that corrects the first heart rate signal, the second heart rate signal, the third heart rate signal, or the integrated heart rate signal.

7. The heart rate estimation system according to claim 1 , further comprising a gimbal provided on the drone, the first camera, the second camera, and the third camera being attached to the gimbal.

8. The heart rate estimation system according to claim 1 , wherein the first camera, the second camera, and the third camera are positioned on a common horizontal plane and are arranged at angles offset from each other by a predetermined angle in the horizontal direction.

9. A heart rate estimation method for estimating a heart rate of a living body, comprising: a preparation step of mounting a first camera for acquiring a visible light image, a second camera for acquiring an infrared light image, and a third camera for acquiring a spectral image on a drone; a first signal estimation step of estimating a first heartbeat signal by an rPPG technique from an image or video captured by the first camera; a second signal estimation step of estimating a second heartbeat signal from the image or video captured by the second camera using an rPPG technique; A heart rate estimation method comprising a third signal estimation step of estimating a third heart rate signal using an rPPG technique from an image or video captured by the third camera.

Images