Signal processing method, signal processing device, and program

Abstract

This signal processing method executed by a signal processing device includes: an acquisition step for acquiring a first signal (32) corresponding to a user state that changes over time in a first period T1; a first generation step for generating a plurality of first processing signals (34) by executing, on the first signal (32), a plurality of first filter processes having mutually different frequencies to be output; a determination step for determining a first output signal (36) from among the plurality of first processing signals (34) on the basis of the signal values of each of the plurality of first processing signals (34); and a second generation step for generating biological information on the basis of the first output signal (36).

Description

Signal processing method, signal processing device, and program

[0001] The present disclosure relates to a signal processing method, a signal processing device, and a program.

[0002] Conventionally, techniques for measuring the pulse rate, heart rate, etc. of a living body using an optical sensor, an imaging device, etc. have been known. For example, a technique has been disclosed in which a pulse wave component is extracted from a video signal using a type of filter that selectively passes a fixed frequency component related to the pulse wave. For example, a technique has been disclosed in which noise is removed from an electrical signal captured by a lead wire attached to a subject by using a signal that emphasizes interference at a fixed frequency, thereby obtaining an electrical signal of the heart.

[0003] International Publication No. 2016 / 158624 Japanese Patent Application Laid-Open No. 2018-187381

[0004] However, the signal-to-noise ratio (SNR) of input signals such as captured images and electrical signals used to generate biometric information is small, making it difficult to generate highly accurate biometric information using conventional techniques.

[0005] The problem to be solved by the present disclosure is to provide a signal processing method, a signal processing device, and a program that are capable of generating highly accurate biometric information.

[0006] The signal processing method according to the present disclosure includes an acquisition step of acquiring a first signal corresponding to a user's state that changes over time during a first period; a first generation step of generating a plurality of first processed signals by performing a plurality of first filter processes on the first signal, the first filter processes outputting frequencies that are different from each other; a determination step of determining a first output signal from among a plurality of first processed signals based on the signal values ​​of each of the plurality of first processed signals; and a second generation step of generating biometric information based on the first output signal.

[0007] According to the signal processing method, signal processing device, and program of the present disclosure, highly accurate biological information can be generated.

[0008] FIG. 1 is a schematic diagram showing an example of a signal processing system according to an embodiment. FIG. 2 is a hardware configuration diagram of an example of a signal processing device. FIG. 3 is a schematic diagram showing an example of a functional configuration of a signal processing system. FIG. 4 is an explanatory diagram of an example of a processing flow by a processing unit according to an embodiment. FIG. 5 is an explanatory diagram of an example of a comb filter. FIG. 6 is a flowchart showing an example of an information processing flow executed by a signal processing device. FIG. 7 is a schematic diagram showing an example of an experimental result using the signal processing device according to an embodiment.

[0009] Hereinafter, embodiments of a signal processing method, a signal processing device, and a program according to the present disclosure will be described with reference to the accompanying drawings.

[0010] In the present disclosure, "at least one selected from the group consisting of A1, A2, ..., and An" may be interpreted as "A1, A2, ..., An, or any combination of A1, A2, ..., An," where n is an integer of 2 or greater.

[0011] For example, at least one of A1 and A2 may be interpreted as "A1", "A2", or "A1 and A2".

[0012] For example, at least one of A1, A2, and A3 may be interpreted as "A1," "A2," "A3," "A1 and A2," "A1 and A3," "A2 and A3," or "A1, A2, and A3."

[0013] In this disclosure, "A and / or B" may be interpreted as "A," "B," or "A and B."

[0014] FIG. 1 is a schematic diagram showing an example of a signal processing system 1 according to this embodiment.

[0015] The signal processing system 1 is a system that generates biometric information of a user P.

[0016] Biometric information is information that changes over time depending on the state of user P. In this embodiment, a description will be given assuming that the biometric information is information related to the pulse of user P. The information related to the pulse may include information related to the heartbeat. Specifically, the information related to the pulse is, for example, the pulse rate, pulse wave interval, heart rate, heartbeat interval, etc. Details of the biometric information will be described later.

[0017] The signal processing system 1 includes a signal processing device 10, an imaging device 11, an illumination device 12, and a display device 13. The imaging device 11, the illumination device 12, and the display device 13 are communicatively connected to the signal processing device 10.

[0018] The image capturing device 11 captures and obtains captured video data of the user P. In the following description, the captured video data will be simply referred to as a captured video.

[0019] The image capturing device 11 has, for example, a CMOS (Complementary Metal-Oxide Semiconductor) image sensor with three-channel color filters of R (Red), G (Green), and B (Blue), and captures RGB captured images of the user P. By providing the image capturing device 11 with a CMOS with three-channel RGB color filters, it is possible to separate the body movement and vibration components of the user P and extract the pulse component by comparing pixel signals in the G wavelength range (wavelength range around 550 nm) where hemoglobin has a high absorption rate with pixel signals in the R wavelength range where hemoglobin has a low absorption rate (described in detail below).

[0020] The image capturing device 11 does not necessarily have to have three RGB channels, but may have two or more channels with different spectral sensitivities. In this case, it may have a channel with spectral sensitivity in the G wavelength range, where hemoglobin has a high absorption rate, and a channel with spectral sensitivity in a wavelength range other than the G wavelength range, where hemoglobin has a low absorption rate. The image capturing element of the image capturing device 11 may be a CCD (Charge Coupled Device) image sensor.

[0021] The photographing device 11 may be configured to include a monochrome image sensor without a color filter, and to capture monochrome photographed images of the user P. When a monochrome image sensor is used, light absorption by a color filter does not occur, and therefore photographed images of the user P with high detection sensitivity can be obtained. When the photographing device 11 is configured to include a monochrome image sensor, an interference-type bandpass filter with better wavelength separation characteristics than a color filter may be sandwiched between the lens and image sensor included in the photographing device 11.

[0022] In this embodiment, an example will be described in which the photographing device 11 includes a CMOS image sensor having RGB color filters, and an RGB photographed image of the user P is acquired by photographing.

[0023] There is no limitation on the type of lens provided in the imaging device 11. Examples of the lens provided in the imaging device 11 include a narrow-angle lens and a wide-angle lens.

[0024] By using a narrow-angle lens for the lens of the imaging device 11, it is possible to acquire a captured image with a high signal-to-noise ratio of a specific region where changes in the state of user P are easily observed, such as the face of user P. The specific region is sometimes called a region of interest. By using a narrow-angle lens, it is possible to acquire a captured image of the specific region with a larger number of pixels than when a wide-angle lens is used.

[0025] By using a wide-angle lens for the lens of the imaging device 11, it becomes possible to obtain, for example, a captured image including multiple users P, and the signal processing device 10 described later becomes able to generate biometric information of each of the multiple users P from the captured image.

[0026] The angle of view of the image capturing device 11 is not limited. For example, assume that the specific area is the face of user P. In this case, assuming that the distance between the image capturing device 11 and user P at the time of image capturing is 70 cm and the vertical width of user P's face is 25 cm, the total vertical angle of view of the image capturing device 11 may be approximately 20 degrees. Assuming that the distance between the image capturing device 11 and user P at the time of image capturing is 50 cm and the vertical width of user P's face is 60 cm, including the predicted vertical movement of user P, the total vertical angle of view of the image capturing device 11 may be approximately 60 degrees.

[0027] The angle of view of the imaging device 11 may be adjusted by adjusting the angle of view of the lens used in the imaging device 11 and the size of the image sensor. The angle of view may also be adjusted by extracting a region of interest (ROI) from the captured image obtained by the imaging device 11. Even if the resolution of the image sensor is high, extracting the necessary ROI and outputting it to the signal processing device 10 as the captured image captured by the imaging device 11 makes it possible to improve communication efficiency and output captured image at a high frame rate to the signal processing device 10. This makes it possible for the signal processing device 10, described later, to improve the accuracy of generating biometric information and the signal-to-noise ratio of the first output signal, described later, used to generate the biometric information. Improving the signal-to-noise ratio may be interpreted as increasing the signal-to-noise ratio.

[0028] The lighting device 12 is a device that irradiates the user P with light. When the lighting device 12 irradiates the user P with light, the imaging device 11 captures an image of the user P illuminated by the light, and obtains a captured image.

[0029] The lighting device 12 emits, for example, visible light or near-infrared light. By using near-infrared light as the illumination to illuminate the user P, the image capturing device 11 can capture video of the user P even in a dark environment where the user P is photographed, such as at night or while sleeping, where the illuminance is below a predetermined level. When capturing a photo of the user P illuminated with near-infrared light by the lighting device 12, the CMOS image sensor of the image capturing device 11 must detect the near-infrared light. Therefore, an IR (InfraRed) cut filter may not be installed in the image capturing device 11. The image capturing of the user P by the image capturing device 11 may be performed in an environment where the user P is not illuminated by the lighting device 12. For example, the image capturing of the user P by the image capturing device 11 may be performed under ambient light, such as sunlight or indoor electric lights. The ambient light includes light in the wavelength range of visible light.

[0030] The lighting device 12 is controlled to switch between on and off under the control of the signal processing device 10, which will be described later. The lighting device 12 is controlled to switch between the type of light it emits, visible light and near-infrared light, under the control of the signal processing device 10, which will be described later. Known lighting equipment that emits visible light or near-infrared light may be used as the lighting device 12. When photographing a user P in a vehicle, an interior light or an ambient light provided in the vehicle may be used as the lighting device 12.

[0031] The display device 13 is a display that displays various information, such as biometric information of the user P generated by the signal processing device 10.

[0032] The signal processing device 10 is an information processing device that generates biometric information of the user P based on an input signal corresponding to the state of the user P that changes over time.

[0033] FIG. 2 is a diagram illustrating an example of a hardware configuration of the signal processing device 10. As shown in FIG.

[0034] The signal processing device 10 has a hardware configuration that utilizes a typical computer, with a CPU (Central Processing Unit) 10A, a ROM (Read Only Memory) 10B, a RAM (Random Access Memory) 10C, and an I / F (Interface) 10D, etc., all interconnected by a bus 10E.

[0035] The CPU 10A is a computing device that controls the signal processing device 10 of this embodiment. The ROM 10B stores programs and the like that realize various processes by the CPU 10A. The RAM 10C stores data necessary for various processes by the CPU 10A. The I / F 10D is an interface for transmitting and receiving data.

[0036] A program for executing information processing executed by the signal processing device 10 of this embodiment is provided by being pre-installed in the ROM 10B, etc. Note that the program executed by the signal processing device 10 of this embodiment may be provided by being recorded on a computer-readable recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, or a digital versatile disk (DVD) in a format that can be installed or executed by the signal processing device 10.

[0037] Next, the functional configuration of the signal processing system 1 including the signal processing device 10 will be described.

[0038] FIG. 3 is a schematic diagram showing an example of the functional configuration of the signal processing system 1. As shown in FIG.

[0039] The signal processing system 1 includes an imaging device 11, an illumination device 12, a display device 13, a communication unit 14, a storage unit 15, and a signal processing device 10. The imaging device 11, the illumination device 12, the display device 13, the communication unit 14, the storage unit 15, and the signal processing device 10 are communicatively connected via a bus 16 or the like. Note that at least one of the imaging device 11, the illumination device 12, the display device 13, and the storage unit 15 may be communicatively connected to the signal processing device 10 via a network or the like.

[0040] The communication unit 14 communicates with an external device such as an external information processing device via a network, etc. The storage unit 15 stores various data. At least a part of the data stored in the storage unit 15 may be stored in a server device or the like that is provided outside the signal processing system 1 and communicably connected to the signal processing device 10 via a network, etc.

[0041] The signal processing device 10 includes a processing unit 20. The processing unit 20 executes various types of information processing in the signal processing device 10. For example, the CPU 10A reads a program from the ROM 10B onto the RAM 10C and executes the program, thereby realizing each of the later-described functional units of the processing unit 20 on the computer.

[0042] The processing unit 20 includes an acquisition unit 20A, a band-pass filter (BPF) 20B, a noise removal unit 20C, a first generation unit 20D, a determination unit 20E, and a second generation unit 20F. Some or all of the acquisition unit 20A, the BPF 20B, the noise removal unit 20C, the first generation unit 20D, the determination unit 20E, and the second generation unit 20F may be implemented, for example, by causing a processing device such as a CPU 10A to execute a program, i.e., by software, or by hardware such as an integrated circuit (IC), or by a combination of software and hardware. At least one of the acquisition unit 20A, the BPF 20B, the noise removal unit 20C, the first generation unit 20D, the determination unit 20E, and the second generation unit 20F may be mounted on an external information processing device communicatively connected to the signal processing device 10 via a network or the like.

[0043] FIG. 4 is an explanatory diagram of an example of the flow of processing by the processing unit 20 of this embodiment.

[0044] First, an outline of the processing flow by the processing unit 20 will be described.

[0045] The acquisition unit 20A acquires a first signal 32 corresponding to the state of the user P that changes over time during a first period T1 of the input signal 30, based on the input signal 30 corresponding to the captured video of the user P. The first generation unit 20D generates a plurality of first processed signals 34 by performing a plurality of first filtering processes with different output frequencies on the first signal 32 acquired by the acquisition unit 20A and filtered by the BPF 20B. The determination unit 20E determines a first output signal 36 from the plurality of first processed signals 34 based on the signal values ​​of each of the plurality of first processed signals 34. The second generation unit 20F generates biometric information based on the first output signal 36.

[0046] Next, each functional unit of the processing unit 20 will be described in detail.

[0047] First, the acquisition unit 20 A will be described. The acquisition unit 20 A acquires an input signal 30 corresponding to the state of the user P, which changes over time.

[0048] The input signal 30 is a signal of the user P that the processing unit 20 uses to generate biometric information. The input signal 30 may be any signal that represents the state of the user P that changes over time. The state of the user P that changes over time may be, for example, but is not limited to, the pulse rate, heart rate, etc. of the user P. In this embodiment, a form in which the input signal 30 is generated based on a video of the user P captured by the imaging device 11 will be described as an example.

[0049] For example, the acquisition unit 20A controls the start and end of shooting of the user P by the image capturing device 11. The acquisition unit 20A may further control the number of pixels and the pixel area of ​​the captured image output from the image capturing device 11. The acquisition unit 20A controls the lighting device 12 to illuminate when the image capturing device 11 captures the user P. For example, when the acquisition unit 20A detects the user P by analyzing the captured image, the acquisition unit 20A controls the lighting device 12 to turn on the illumination and controls the image capturing device 11 to start shooting the user P. The acquisition unit 20A may control the image capturing device 11 to start shooting the user P by accepting an operation instruction from the user P, etc. Then, the acquisition unit 20A acquires the captured image of the user P illuminated by the lighting device 12 from the image capturing device 11.

[0050] The acquisition unit 20A may control the lighting device 12 to turn off the light when the photographing device 11 finishes photographing the user P. That is, the acquisition unit 20A may synchronously control the photographing of the user P by the photographing device 11 and the turning on of the light by the lighting device 12 so that the light is irradiated on the user P during photographing. The acquisition unit 20A may control the photographing device 11 and the lighting device 12 asynchronously or in a non-coordinated manner. The photographing device 11 and the lighting device 12 may each be independently controlled by a separate control circuit or the like.

[0051] The acquisition unit 20A may control the lighting device 12 to turn on near-infrared light when it determines that the shooting environment of the user P is a dark environment with illuminance below a predetermined level based on the detection result of the illuminance sensor or analysis of the captured video. This is because, even when the user P is sleeping, the user P can be photographed using near-infrared light without disturbing the sleep of the user P. Note that, although the captured video captured under near-infrared light irradiation may have reduced accuracy in detecting the state of the user P that changes over time, such as the pulse rate, the signal processing device 10 of this embodiment can generate biometric information with high accuracy by performing the process described below.

[0052] The acquisition unit 20A may control the lighting device 12 to switch between the type of light to be emitted (near-infrared light, visible light) and non-emission of light, depending on the determination result of the user P's sleep state.

[0053] In this case, the acquisition unit 20A determines whether the user P is asleep. The acquisition unit 20A may determine the user P's sleep state using a known sleep determination technique. The acquisition unit 20A may determine whether the user P is asleep based on the latest biometric information of the user P generated by the second generation unit 20F described below. In detail, the acquisition unit 20A determines that the user P is asleep if the pulse rate included in the biometric information is below a threshold, and determines that the user P is not asleep if the pulse rate is above the threshold. Note that the pulse used to determine the user P's sleep state may be generated based on a signal acquired by the image capture device 11 or a signal acquired by a device other than the image capture device 11, such as a wearable device.

[0054] The acquisition unit 20A may determine whether the user P is asleep based on the amount of body movement of the user P. For example, the acquisition unit 20A determines that the user P is asleep if the amount of variation in the user P's body movement over a predetermined period is less than a threshold, and determines that the user P is not asleep if the amount of variation is equal to or greater than the threshold. The amount of variation may be a standard deviation or the like. The body movement of the user P may be derived by a known method based on the captured video captured by the image capture device 11. In this case, for example, the acquisition unit 20A may identify the user P included in the captured video by analyzing the captured video using a known image analysis process or the like, and derive the amount of variation in the identified user P's movement as the amount of variation in the user P's body movement. The acquisition unit 20A may derive the amount of variation in the user P's body movement based on the detection results of an acceleration sensor of a wearable device or the like.

[0055] When the acquisition unit 20A determines that the sleep state of the user P is sleeping, the acquisition unit 20A controls the lighting device 12 to turn on near-infrared light. In this case, the acquisition unit 20A may generate the first signal 32 by detecting light from the user P caused by irradiation with near-infrared light.

[0056] On the other hand, when the acquisition unit 20A determines that the sleep state of the user P is not sleeping, it controls the lighting device 12 to turn on visible light. When the acquisition unit 20A determines that the sleep state of the user P is not sleeping, it may control the lighting device 12 not to emit light. In this case, the image capture device 11 may generate the first signal 32 by detecting light in the wavelength range of visible light emitted from the lighting device 12 or light from the user P caused by ambient light.

[0057] The acquisition unit 20A controls the lighting device 12 to switch between near-infrared light, visible light, and no light irradiation depending on the result of the judgment of the user P's sleep state, so that the signal processing device 10 of this embodiment can generate biometric information such as pulse rate with high accuracy.

[0058] The acquisition unit 20A acquires, as the first signal 32, the number of frames of the captured video in the first period T1 in the captured video, which is the acquired input signal 30. The input signal 30 may be an average value of an ROI, which is a specific region to be calculated, as described later.

[0059] In detail, the acquisition unit 20A shifts the time window W of the first period T1 of the captured video, which is the input signal 30, by a predetermined time along the time series in a direction toward a more current capture timing. The predetermined time may be shorter than the first period T1. In this embodiment, the description will be made assuming that the predetermined time corresponds to one frame of the captured video. That is, in this embodiment, the acquisition unit 20A acquires the captured video of the first period T1 of the time window W in the input signal 30 as the first signal 32 each time the time window W is shifted by one frame. In other words, in this embodiment, the acquisition unit 20A shifts the time window W so that it includes the most recent frame each time a new frame of video is captured by the imaging device 11, thereby shifting the time window W in a direction toward a more current capture timing.

[0060] The first period T1 is a period that serves as a processing unit for the first generating unit 20D and the determining unit 20E. If the biometric information to be generated includes a pulse, the first period T1 may be a period that includes at least two to three pulse cycles. In this case, the first period T1 may be specifically 2 to 3 seconds.

[0061] By setting the time window W of the first period T1 to such a short time, the signal processing device 10 does not need to wait until it acquires the first signal 32, which is the captured image for the first period T1, and can output biometric information at high speed, thereby improving the real-time nature of the output biometric information.

[0062] By setting the time window W of the first period T1 to such a short time, it becomes possible to obtain biometric information such as PPI (Peak to Peak Interval) at each moment of the user P, which changes over time, with high accuracy, and to generate biometric information that accurately represents the pulse rate fluctuation status. For example, when the user P is in a tense state, the PPI is constant, and when the user P is in a relaxed state, the PPI is likely to fluctuate. Therefore, by setting the time window W of the first period T1 to such a short time, the signal processing device 10 can generate biometric information of the user P with high accuracy.

[0063] When the first period T1 is expressed by the number of frames of the captured video, the first period T1 may be, for example, a period of at least 10 fps (frames per second) or more, or may be a period of 30 fps or more, in the captured video acquired from the imaging device 11. From the viewpoint of improving the accuracy of generating biometric information, the first period may be a period of 60 fps or more and 120 fps or less.

[0064] The acquisition unit 20A may use the captured image of the first period T1 in the input signal 30 as the first signal 32. However, the acquisition unit 20A may extract pixel data within a specific ROI included in each frame of the captured image of the first period T1, and use intensity value time series data consisting of a group of pixel data within the extracted ROI for the first period T1 along the time series, as the first signal 32. The extracted pixel data within the ROI is, for example, an average value of the pixel values ​​of the pixels included in the ROI.

[0065] Specifically, the acquisition unit 20A sets an ROI, which is a specific region to be calculated, included in the captured video of the first period T1. For example, the acquisition unit 20A performs face detection or known landmark detection based on facial feature points, and sets the relative position and size of the ROI included in the captured video based on the detected coordinate values.

[0066] Specifically, a region on the body surface of the user P where blood vessels pass close to the skin surface is a region where changes in light intensity corresponding to changes in the amount of hemoglobin in the blood due to pulse are easily observed. Therefore, the acquisition unit 20A may set a predetermined region on the face of the user P, such as the forehead, cheeks, or chin, where blood vessels are easily observed, as the ROI. The acquisition unit 20A may also set another region on the face of the user P, such as the fingertips, where blood vessels are easily observed, as the ROI. In this embodiment, an example will be described in which the acquisition unit 20A sets a region on the face of the user P as the ROI.

[0067] Note that the ROI does not need to include the eyes because blinking can cause noise components. The mouth moves during conversation, etc., so the ROI does not need to include the mouth. Therefore, the acquisition unit 20A may set, for example, a rectangular region that surrounds the area of ​​the user P's face below the eyes and above the mouth as the ROI. Alternatively, to prevent the influence of uneven illumination caused by the nose shadow, the ROI may be set so as not to include the nose. For example, the ROI may be set to include at least one of the left cheek and the right cheek.

[0068] The shape of the ROI is not limited to a rectangle. For example, the acquisition unit 20A may set, as the ROI, a region obtained by detecting landmarks and enclosing the skin part of the face of the user P, excluding the eyes and mouth, with a free curve. The acquisition unit 20A may set, as the ROI, a skin region detected by extracting a hue corresponding to the skin color from a color space such as HSV (H: Hue, S: Saturation Chroma, V: Value Brightness).

[0069] The larger the number of pixels included in the ROI, the larger the S / N ratio of the average pixel value of the ROI can be. Therefore, for example, when the acquisition unit 20A determines that the mouth of the user P is not moving by analyzing the captured video, the acquisition unit 20A may set the ROI to an area including the mouth of the user P so that a wider area of ​​the face is set.

[0070] The acquisition unit 20A may set an ROI for each of a plurality of frames included in the captured video of the first period T1, or may set a fixed ROI for the plurality of frames.

[0071] The acquisition unit 20A may reset the ROI when a certain level of movement of the user P is observed for multiple frames included in the captured video of the first period T1. For example, when a change of a threshold or more is detected in the average pixel value of a frame, the acquisition unit 20A may determine that a luminance change due to facial movement has occurred and reset the ROI at the frame at the detection timing. Specifically, since the change in absorbance of a pulse is approximately 1.5%, when the average pixel value within the set ROI changes by 1.5% or more from the average pixel value at the time of ROI setting, it may be determined that the face has moved and the ROI may be reset at the frame at that determination timing. The acquisition unit 20A may update the ROI at a predetermined timing, such as once every 3 seconds, for multiple frames included in the captured video of the first period T1.

[0072] In this embodiment, the acquisition unit 20A calculates, for each frame, pixel data that is the average value of pixel values ​​of the ROI for each of a plurality of frames included in the captured video during the first period T1. The acquisition unit 20A then generates intensity value time series data in which the pixel data that is the average value of pixel values ​​of the ROI calculated for each frame is arranged in a time sequence that follows the frame arrangement included in the captured video during the first period T1. The acquisition unit 20A then acquires this intensity value time series data as the first signal 32 for the time window W of the first period T1 in the input signal 30. Note that the acquisition unit 20A may also acquire, as the first signal 32, intensity value time series data that further converts the average pixel values ​​into values ​​relative to an initial value.

[0073] Next, the BPF 20B will be described. The BPF 20B is a filter that removes noise components contained in the intensity value time-series data. The BPF 20B is a band-pass filter that removes signal components other than the pulse component by using, for example, a Butterworth filter of order 6 with a cutoff frequency of 0.5 Hz. The BPF 20B may also be a low-cut filter or a high-pass filter that removes low-frequency components contained in the intensity value time-series data.

[0074] The BPF 20B may apply a filter to the intensity time series data using a time window W2. The time window W2 is, for example, the most recent seven frames (for example, approximately 0.23 seconds), but is not limited to this value.

[0075] The noise removal unit 20C may apply Auto Gain Control to the intensity value time series data to remove steep signal noise due to body movement of the user P from the intensity value time series data. For example, the noise removal unit 20C removes noise using a tanh function. By passing the intensity value time series data through the tanh function, it is possible to suppress divergence of the intensity values ​​of the intensity value time series data due to the inclusion of large intensity values ​​due to noise, and it is possible to improve the accuracy of the first filtering process by the first generation unit 20D using the intensity value time series data.

[0076] The acquisition unit 20A updates the intensity value time series data for the time window W of the latest first period T1 in the intensity value time series data from which noise has been removed by the noise removal unit 20C as the latest first signal 32. That is, each time one frame is captured, one sample of data that has passed through the BPF 20B is added to the first signal 32, and the oldest sample of data is removed one sample at a time, thereby updating the first signal 32 as the latest first signal 32. Through these processes, the acquisition unit 20A acquires the captured video of the first period T1 in the time window W as the first signal 32, each time the acquisition unit 20A moves the time window W of the first period T1 by one frame along the time series with respect to the captured video, which is the input signal 30.

[0077] Next, the first generation unit 20D will be described.

[0078] The first generation unit 20D generates a plurality of first processed signals 34 by performing a plurality of first filtering processes on the first signal 32. In the present embodiment, the first generation unit 20D generates a plurality of first processed signals 34 by performing a plurality of first filtering processes on the first signal 32 in the time window W of the first period T1 that has been acquired by the acquisition unit 20A and from which noise has been removed by the BPF 20B and the noise removal unit 20C.

[0079] The plurality of first filtering processes are filtering processes that output frequencies different from each other.

[0080] In particular, the first generating unit 20D includes a plurality of filters that output different frequencies, and executes a plurality of first filtering processes by inputting the first signal 32 to each of these plurality of filters.

[0081] In this embodiment, the first generation unit 20D includes multiple comb filters CF with different output frequencies. That is, in this embodiment, the first generation unit 20D performs multiple first filter processes by inputting the first signal 32 to each of the multiple comb filters CF with different output frequencies. The first generation unit 20D inputs the first signal 32 to each of the multiple comb filters CF at the same timing. Therefore, the first generation unit 20D performs multiple first filter processes in parallel on the first signal 32 during the first period T1 using each of the multiple comb filters CF. However, in actual calculations performed in the signal processing device 10, parallel processing is not necessarily required. The processing unit 20 of the signal processing device 10 may also perform calculations sequentially.

[0082] 4 shows a configuration in which first generation unit 20D includes n comb filters CF (n is an integer equal to or greater than 2), ie, comb filters CF1 to CFn. In FIG. 4, m is an integer less than n.

[0083] The comb filter CF is a comb filter whose sampling characteristics on the time axis are comb-shaped. That is, the comb filter CF of this embodiment is a filter that emphasizes and outputs a signal of a specific time period contained in the first signal 32. The multiple comb filters CF have different emphasized frequencies, i.e., different emphasized time periods.

[0084] 5 is an explanatory diagram of an example of a comb filter CF. Fig. 5 shows the waveform of the first signal 32 in the first period T1 and a schematic diagram representing the comb filter CF. The horizontal axis of the waveform of the first signal 32 represents time, and the vertical axis represents the signal value. The horizontal axis of the schematic diagram representing the comb filter CF represents time, and the vertical axis represents the weight value of the comb C.

[0085] The comb filter CF includes multiple combs C. FIG. 5 shows an example in which the comb filter CF includes three combs C (combs C1 to C3). The combs C represent sampling timings for sampling the first signal 32 at time intervals SI corresponding to the period of the frequency output by the comb filter CF within a time window W of the first period T1. For each comb C, which is a sampling timing of the first signal 32 at the time interval SI corresponding to the frequency output by the comb filter CF, the comb filter CF samples the signal value of the first signal 32 at the sampling timing of the comb C, multiplies the sampled signal value of each comb C by the weight value of the comb C, and outputs the sum of the weighted signal values ​​as the first processed signal 34 for the first signal 32 within the time window W.

[0086] In the example shown in Figure 5, the comb filter CF outputs a total signal value obtained by adding up the signal values ​​of the sampling timings of combs C1, C2, and C3 in the first signal 32 in the first period T1 as the first processed signal 34 in the first signal 32 in the time window W of the first period T1.

[0087] Therefore, when the first generation unit 20D inputs the first signal 32 to the comb filter CF, the signal values ​​of the first signal 32 are sampled at the sampling timing of each comb C of the comb filter CF in the first signal 32, and a single total signal value, which is the sum of the sampled signal values, is output as the first processed signal 34 for the first signal 32 in the time window W (see also FIG. 4 ). The comb filter CF outputs the total signal value of the signal values ​​for each time period of the time interval SI included in the first signal 32, thereby outputting the first processed signal 34 in which the signal values ​​of the first signal 32 are emphasized at a specific frequency, i.e., at a specific time period.

[0088] Returning to FIG. 4, the description will continue.

[0089] In this embodiment, the first generator 20D includes a plurality of comb filters CF that output different frequencies, i.e., the first generator 20D includes a plurality of comb filters CF that have different time intervals SI of combs C that sample signals at time periods corresponding to the output frequencies and that emphasize different frequencies.

[0090] Therefore, the first generation unit 20D executes a plurality of first filter processes in which the first signal 32 is input to each of the plurality of comb filters CF, and a plurality of first processed signals 34 with different emphasized frequencies are output from the plurality of comb filters CF. Here, different frequencies correspond to different beats per minute (bpm), which is a unit of measurement for pulse or heart rate, expressed in terms of biological information. Therefore, the plurality of comb filters CF with different output frequencies provided by the first generation unit 20D can be said to be comb filters CF with different emphasized bpm values ​​included in the first signal 32.

[0091] The number of comb filters CF included in the first generator 20D is not limited as long as it is plural. For example, a pulse rate range of 50 bpm to 90 bpm corresponding to the frequency to be emphasized may be divided into 130 stages. The first generator 20D may then be configured to include 130 comb filters CF that emphasize different frequencies (pulses) in each stage. In this case, the time interval SI of the combs C of the multiple comb filters CF included in the first generator 20D ranges from 667 ms to 1200 ms in terms of the PPI (Pulse-Pulse Interval) value, which is the pulse interval. Therefore, in this case, the first generator 20D is configured to include multiple comb filters CF in which the time interval SI of the combs C is gradually changed by approximately 4 ms. In this case, it is possible to achieve an error of 1% or less in calculating the PPI, which is the pulse wave interval included in the biometric information described below.

[0092] The first generator 20D may be configured to include 130 or more comb filters CF that output different frequencies from the viewpoint of improving the accuracy of the biological information generated in the signal processing device 10. The first generator 20D may be configured to include fewer than 130 comb filters CF that output different frequencies from the viewpoint of reducing the calculation time.

[0093] The number of combs C included in each of the plurality of comb filters CF included in the first generation unit 20D may be more than one and is not limited to three. For example, the comb filter CF may be configured to include four or more combs C. However, the number of combs C included in each of the comb filters CF included in the first generation unit 20D may be the same. The more combs C included in a comb filter CF, the greater the noise reduction effect. However, from the viewpoint of being able to generate an instantaneous pulse value as biological information, the number of combs C included in a comb filter CF may be two or more and five or less.

[0094] The weight values ​​of the combs C included in the time window W of each comb filter CF included in the first generation unit 20D may be the same or different. The weight values ​​correspond to filter coefficients. That is, the weight values ​​are weight values ​​by which the signal values ​​sampled at the sampling timing of each comb C are multiplied.

[0095] When the weight values ​​of multiple combs C included in the comb filter CF are the same, a first processed signal 34 having a value obtained by averaging the signal values ​​at the sampling timing of each comb C can be output from the comb filter CF, thereby improving the noise reduction effect.

[0096] If the weight value of at least one of the multiple combs C included in the comb filter CF is set to a value different from the weight values ​​of the other combs C, the second generation unit 20F described later will generate generated information based on the first output signal 36 determined from the first processed signal 34 output from the comb filter CF, thereby making it possible to improve the detection accuracy of pulse fluctuations included in the biometric information.

[0097] When the weight values ​​of the combs C included in the comb filter CF are different, the weight value of the central comb C in the time axis direction among the multiple combs C included in the time window W may be greater than the weight values ​​of the other combs C. For example, if the comb filter CF includes three combs C, combs C1 to C3, the weight values ​​of combs C1 and C3 may be 0.7, and the weight value of the central comb C2 may be 1.0. Because the combs C represent sampling timings in the time window W, by making the weight value of the central comb C in the time axis direction greater than the weight values ​​of the other combs C, the comb filter CF can output a first processed signal 34 that more accurately represents the state of the user P at the current moment in the time window W. The second generator 20F, described later, generates generated information based on a first output signal 36 determined from the first processed signal 34, thereby improving the accuracy of detecting pulse fluctuations included in biological information.

[0098] The first generating unit 20D may perform the first filtering process using a comb filter CF that outputs frequencies within a specific frequency range. For example, the first generating unit 20D may perform the first filtering process using, among the multiple comb filters CF included in the first generating unit 20D, multiple comb filters CF that output frequencies within a frequency range according to a predetermined condition related to the user P. The predetermined condition related to the user P may be, for example, information indicating whether the user P is awake or asleep, information indicating the health state of the user P, information indicating the characteristics of the user P, etc.

[0099] For example, it is known that the heart rate and pulse rate of a sleeping user P are lower than when awake. Therefore, for example, when the first generation unit 20D determines that the user P is sleeping by image analysis of the captured video using a known method, it may perform first filtering using multiple comb filters CF that output frequencies in a frequency range of 40 bpm to 80 bpm, which is the typical pulse rate during sleep. Note that the first generation unit 20D may set the upper limit of the frequency range when it is determined that the user P is sleeping to a value lower than the upper limit of the frequency range when it is determined that the user P is not sleeping. The first generation unit 20D may set the lower limit of the frequency range when it is determined that the user P is sleeping to a value lower than the lower limit of the frequency range when it is determined that the user P is not sleeping.

[0100] The first generating unit 20D stores information representing the health state of the user P and information representing the characteristics of the user P in advance in the storage unit 15, and stores information representing the frequency range to be used in the first filtering process in association with the information. For example, biometric information generated by the second generating unit 20F described later may be used as the information representing the health state of the user P and the information representing the characteristics of the user P. The first generating unit 20D may then change the frequency range to be used in the first filtering process in accordance with the health state and characteristics of the user P, and perform the first filtering process using a comb filter CF that outputs frequencies in the changed frequency range.

[0101] At the start of measurement of the user P's biometric information, the first generator 20D performs a first filtering process using multiple comb filters CF that output frequencies in a wider frequency range. Then, when the pulse rate represented by the biometric information generated by the second generator 20F (described later) stabilizes, the first generator 20D may perform the first filtering process using multiple comb filters CF that output frequencies in a narrower frequency range than at the start of measurement, corresponding to a bpm range that includes the stable pulse rate. For example, the frequency range 15 seconds, 30 seconds, or 1 minute after measurement begins may be narrower than the frequency range immediately after measurement begins. This allows measurement even when it is unclear which frequency the user P's pulse rate is near at the start of measurement, and after a period of time, the estimated frequency range can be narrowed to improve pulse rate detection accuracy.

[0102] The first generation unit 20D may perform a first filter process using multiple comb filters CF that output frequencies over a wide frequency range when the user P's pulse rate fluctuates significantly within a short period of time, and may narrow the frequency range when the pulse rate fluctuations become smaller. In other words, the frequency range when the pulse is stable may be narrower than the frequency range when the pulse is unstable. For example, the first generation unit 20D may determine that the pulse is stable when the standard deviation of the pulse rate per unit time is within a reference value, and determine that the pulse is unstable when the standard deviation exceeds the reference value. For example, the first generation unit 20D may determine that the pulse is stable when the standard deviation of the pulse rate within 5 seconds or 5 beats is within 10 bpm, and that the pulse is unstable when it exceeds that value. In this manner, the signal processing device 10 of this embodiment can stably acquire the user P's pulse rate. Instead of varying the frequency ranges output by the multiple comb filters CF, the frequency ranges output by the multiple comb filters CF may be kept constant, and the frequency range selected by the determination unit 20E described below may be widened or narrowed.

[0103] In this way, the first generation unit 20D changes the frequency range used in the first filter processing in accordance with predetermined conditions related to the user P, and performs the first filter processing to output frequencies within the changed frequency range, thereby making it possible to shorten the time required for the first filter processing by the first generation unit 20D, reduce the processing load, and improve the accuracy of calculation of biometric information such as pulse rate.

[0104] The comb filter CF may be configured with an FIR (Finite Impulse Response) filter or an IIR (Infinite Impulse Response) filter. An IIR filter holds past data components infinitely. Therefore, in this embodiment, the comb filter CF may be configured with an FIR filter that does not hold past data. By configuring the comb filter CF with an FIR filter, the second generation unit 20F, which will be described later, can generate biometric information with higher accuracy.

[0105] The first generating unit 20D may further perform peak hold processing on the first processed signal 34 output from each of the plurality of comb filters CF.

[0106] In detail, for each of the multiple first processed signals 34 output from each of the multiple comb filters CF by executing multiple first filter processes, the first generation unit 20D may generate, for each of the multiple comb filters CF, the weighted average result of the first processed signal 34 generated by the previous execution of the first filter process and the first processed signal 34 generated by the current execution of the first filter process as the first processed signal 34 generated by the current execution of the first filter process.

[0107] The first processed signal 34 generated by the previous execution of the first filtering process may be any of the first processed signal 34 generated by the previous execution of the first filtering process by the comb filter CF, the first processed signal 34 generated by the previous or earlier execution of the first filtering process by the comb filter CF, or the average value of multiple first processed signals 34 generated by multiple executions of the first filtering process by the comb filter CF. Specifically, if the comb filter CF is an FIR filter, the first processed signal 34 generated by the previous execution of the first filtering process represents the first processed signal 34 previously output from the comb filter CF. If the comb filter CF is an IIR filter, the first processed signal 34 generated by the previous execution of the first filtering process represents the first processed signal 34 after the first processed signal 34 previously output from the comb filter CF has been further passed through the IIR filter.

[0108] The first generating unit 20D may perform the peak hold process using these weighted averages by using, for example, a smoothing filter. This smoothing filter may be configured as either an FIR filter or an IIR filter.

[0109] By performing these peak hold processes, the first generation unit 20D can output the first processed signal 34 with the influence of noise suppressed as the first processed signal 34 generated by the current first filter process, even if the first processed signal 34 output from the comb filter CF becomes a value that temporarily fluctuates due to the influence of noise.

[0110] The first generating unit 20D may vary the weighting value used in this peak hold process.

[0111] For example, for each of the plurality of comb filters CF by execution of the plurality of first filtering processes, the first generation unit 20D executes the following process if the first processed signal 34 generated by the current execution of the first filtering process is greater than the first processed signal 34 generated by the previous execution of the first filtering process. In this case, the first generation unit 20D generates, as the first processed signal 34 generated by the current execution of the first filtering process, a weighted average result in which the weight value of the first processed signal 34 generated by the current execution of the first filtering process is made higher than the weight value of the first processed signal 34 generated by the previous execution of the first filtering process.

[0112] The first generation unit 20D performs the following process for each of the multiple first processed signals 34 output from the multiple comb filters CF by the execution of the multiple first filtering processes, if the first processed signal 34 generated by the current execution of the first filtering process is smaller than the first processed signal 34 generated by the previous execution of the first filtering process. In this case, the first generation unit 20D generates a weighted average value as the first processed signal 34 generated by the current execution of the first filtering process, by lowering the weighting value of the first processed signal 34 generated by the current execution of the first filtering process compared to the weighting value of the first processed signal 34 generated by the previous execution of the first filtering process. The process of increasing or decreasing the weighting value can be adjusted by increasing or decreasing the normal weighting value.

[0113] By changing the weighting values ​​used in the peak hold process by the first generation unit 20D, during a period in which the pulse rate, which is an example of the state of the user P that changes over time, is constant and stable, the first processed signal 34 after the peak hold process output from the comb filter CF that emphasizes the frequency that coincides with the pulse among the multiple comb filters CF always exhibits a higher signal value both when the amplitude of the pulse rises and falls, compared to the first processed signal 34 output from the comb filter CF that emphasizes other frequencies. This allows the determination unit 20E, described later, to determine a more accurate first processed signal 34 as the first output signal 36.

[0114] During a period in which the pulse rate fluctuates and is unstable, which is an example of the state of the user P, interference from the comb filter CF decreases, and the value of the first processed signal 34 after peak hold processing also decreases. Therefore, in this case, the determining unit 20E (described later) can determine the first processed signal 34 with increased flexibility, and can sequentially select and determine more accurate first processed signals 34 as the first output signal 36.

[0115] Next, the determining unit 20E will be described.

[0116] The determination unit 20E determines a first output signal 36 from among the multiple first processed signals based on the signal values ​​of each of the multiple first processed signals 34 generated by the multiple first filter processes output from the first generation unit 20D.

[0117] Specifically, the determining unit 20E determines the first processed signal 34 with the largest signal value from among the plurality of first processed signals 34 as the first output signal 36.

[0118] It can be said that the frequency output by the comb filter CF that outputs the first processed signal 34 with the largest signal value among the multiple first processed signals 34 matches the pulse bpm of the user P contained in the first signal 32 used to output the first processed signal 34. In other words, the first processed signal 34 with the largest signal value among the multiple first processed signals 34 is the first processed signal 34 output from the comb filter CF at the time interval SI that matches the pulse interval of the user P, represented by the first signal 32 used to output the first processed signal 34. This allows the determination unit 20E to sequentially determine, over time, first output signals 36 that match with high accuracy each moment-to-moment state of the user P, which changes over time.

[0119] As described above, the acquisition unit 20A moves the time window W of the first period T1 by one frame along the time series in the direction toward the current shooting timing for the time-series input signal 30, which is the captured video acquired from the imaging device 11. Then, every time the acquisition unit 20A moves the time window W by one frame, the acquisition unit 20A outputs the first signal 32, which is intensity value time-series data for the first period T1, generated from the captured video of the first period T1 of the time window W, to the first generation unit 20D via the BPF 20B, etc.

[0120] The first generation unit 20D then inputs the first signal 32 for the first period T1 to each of a plurality of comb filters CF that output signals at different frequencies. Therefore, a single signal value, which is the sum of signal values ​​sampled at the sampling timings of the time intervals SI of the combs C for the first signal 32 for the first period T1 from each of the plurality of comb filters CF included in the first generation unit 20D, is used as a single first processed signal 34 (first processed signal 34 １ ~ First processed signal 34 ｎ The acquisition unit 20A moves the time window W by one frame to acquire the input signal 30 for a new time window W, and each time a first signal 32 corresponding to the input signal 30 is input to the plurality of comb filters CF, the plurality of comb filters CF output first processed signals 34 in parallel.

[0121] Then, each time the acquisition unit 20A moves the time window W, the determination unit 20E determines, as the first output signal 36, the first processed signal 34 with the largest signal value from among the multiple first processed signals 34 output from each of the multiple comb filters CF in accordance with the first signal 32 of the time window W. Therefore, the determination unit 20E can determine, for each moment during one movement period of the time window W, for example, one frame, as the first output signal 36 for that moment.

[0122] Therefore, the determination unit 20E can sequentially determine, over time, the first output signal 36 that matches with high accuracy each moment-to-moment state of the user P, which changes over time.

[0123] Alternatively, the frequency range α of the comb filter CF used by the first generation unit 20D may be constant, and the determination unit 20E may determine the first output signal 36 from the first processed signal 34 output from the comb filter CF that outputs frequencies in a specific frequency range β. If the minimum frequency in the frequency range α is αmin, the maximum frequency in the frequency range α is αmax, the minimum frequency in the specific frequency range β is βmin, and the maximum frequency in the specific frequency range β is βmax, then αmin < βmin < βmax < αmax may be satisfied.

[0124] For example, if the bpm (Beat Per Minute) of the first output signal 36 determined by the determination unit 20E for the input signal 30 corresponding to the (t-1)th time is 65, and the minimum bpm of the multiple first processed signals received by the determination unit 20E from the first generation unit 20D for the input signal 30 corresponding to the tth time is 50 and the maximum bpm is 90, the determination unit 20E may select one from one or multiple first processed signals having a bpm of 65±5 from the multiple first processed signals, and output it as the first output signal 36 corresponding to the time t.

[0125] For example, the determiner 20E may determine the first output signal 36 from the first processed signals 34 of frequencies within a frequency range according to at least one of a predetermined condition related to the user P and a reliability determination result of the biometric information generated based on the first output signal 36, among the multiple first processed signals 34 generated by the multiple first filtering processes performed by the first generator 20D. The predetermined condition related to the user P is the same as described above.

[0126] The reliability determination result is a determination result of the reliability of the biological information generated by the first generator 20D, which will be described later, based on the first output signal 36. For example, the determiner 20E identifies the first output signal 36 by receiving, from the determiner 20E, a reliability determination result of a pulse rate, which is an example of biological information recently generated by the first generator 20D. Then, the determiner 20E may determine the first output signal 36 from a plurality of first processed signals 34 output from a comb filter CF that outputs a pulse rate whose identified reliability is equal to or greater than a threshold value or a frequency in a frequency range near the pulse rate.

[0127] For example, the determination unit 20E may specify a frequency range of ±10 bpm relative to the average or median value of the most recent six pulse rates included in the biological information generated by the second generation unit 20F. The second generation unit 20F may then determine the first output signal 36 from the first processed signal 34 output from the comb filter CF, which outputs frequencies within the specified frequency range. The determination unit 20E may specify a frequency range of ±5 bpm relative to the average or median value.

[0128] The determination unit 20E may determine whether the reliability of the biometric information is above a threshold based on, for example, whether the face of the user P can be detected from the captured image, whether the rate of change between adjacent pulse rates represented by the generated biometric information is constant (e.g., 5 bpm) or less, whether the standard deviation of the pulse rate is constant (e.g., 10 bpm with a standard deviation of 6 beats) or less, whether the peak value of the output signal waveform 38 represented by the first output signal 36 that continues in the sequentially determined time series is above a certain value, etc.

[0129] By having the determination unit 20E determine the first output signal 36 from the first processed signal 34 having a frequency within a specific frequency range in this manner, the accuracy of biometric information such as PPI can be improved even if the video of user P is captured in a noisy environment.

[0130] The first generator 20D may also continue to hold the first processed signal 34 output from the comb filter CF that outputs frequencies outside the target specific frequency range determined by the determiner 20E. By performing this process, the first generator 20D can refer to previously output first processed signals 34, enabling arithmetic processing such as time-series smoothing, for example, when the specific frequency range determined by the determiner 20E is changed due to a large fluctuation in pulse rate. Therefore, this process can improve the S / N ratio of the first output signal 36. Improving the S / N ratio may be interpreted as increasing the S / N ratio.

[0131] As described above, the acquisition unit 20A sequentially shifts the time window W of the first period T1 by one frame along the time series in the direction toward the current shooting timing for the input signal 30, which is a time-series image captured from the image capturing device 11. Each time the acquisition unit 20A shifts the time window W by one frame, the acquisition unit 20A acquires a first signal 32, which is intensity value time-series data for the first period T1 generated from the captured image of the time window W for the first period T1, and outputs the first signal 32 to the first generation unit 20D via the BPF 20B or the like. Each time the time window W is shifted by one frame, the determination unit 20E determines the first processed signal 34 with the largest signal value among the multiple first processed signals 34 output from the multiple comb filters CF as the first output signal 36. Note that the acquisition of the first signal 32, which is intensity value time-series data for the first period T1, may be performed after the input signal 30 has passed through the BPF 20B.

[0132] That is, each time the time window W is moved by one frame in a chronological order, the determination unit 20E sequentially determines the first processed signal 34 with the highest signal value as the first output signal 36 from among the multiple first processed signals 34 generated by multiple first filter processes with different frequencies on the first signal 32 of the time window W.

[0133] For this reason, even if the actual pulse rate of user P at a certain point in time falls outside the specific frequency range determined by the determination unit 20E, a frequency closest to the frequency of the actual pulse rate is selected from the upper and lower end frequencies of the specific frequency range for the output first output signal 36. Therefore, by changing the frequency range in accordance with the selected frequency, the frequency range of the first processed signal 34 to be determined by the determination unit 20E is gradually adjusted to a frequency range that includes the frequency of the actual pulse rate of user P, thereby improving the accuracy of the determined first output signal 36.

[0134] Next, the second generation unit 20F will be described.

[0135] The second generator 20F generates biometric information based on the first output signal 36 .

[0136] As described above, biometric information is information that changes over time depending on the state of user P. In this embodiment, the biometric information is information related to the pulse of user P. The definition of the information related to the pulse is the same as above. Specifically, the biometric information is, for example, pulse rate, heart rate, pulse wave interval, heartbeat interval, an index representing the state of user P estimated from these pulse rates, etc. The index is, for example, alertness, calmness, pleasure / unpleasantness, comfort, concentration, slowness, relaxation, degree of sympathetic dominance, degree of parasympathetic dominance, depth of sleep, health condition, level of attention, psychological state, etc.

[0137] As described above, each time the time window W is shifted by one frame continuously in time series, the first processed signal 34 having the largest signal value among the plurality of first processed signals 34 generated by the plurality of first filter processes on the first signal 32 in the time window W is sequentially determined as the first output signal 36. Therefore, the second generating unit 20F generates biological information based on an output signal waveform 38 consisting of a group of first output signals 36 that are sequentially determined each time the time window W is shifted.

[0138] For example, the second generating unit 20F detects peaks in the output signal waveform 38 and calculates the number of peaks per minute as the pulse rate or heart rate (bpm).The second generating unit 20F calculates the interval between the peaks in the output signal waveform 38 as the pulse interval (PPI).

[0139] The second generation unit 20F estimates the above-mentioned index representing the state of user P from the time series changes in at least one of the calculated pulse rate, the calculated heart rate, and the calculated pulse wave interval using a known estimation method, etc.

[0140] The second generation unit 20F may generate biometric information after removing high-frequency components included in the output signal waveform 38 that are generated when the first processed signal 34 determined by the determination unit 20E is switched between multiple comb filters CF. A high-cut filter, a low-pass filter, a band-pass filter, or the like may be used to remove the high-frequency components. The high-cut filter and the low-pass filter may be, for example, a filter that applies a Hanning window to 0.5 seconds of the output signal waveform 38. The Hanning window is sometimes called a Hann window.

[0141] The second generating unit 20F may output the generated biometric information. For example, the second generating unit 20F outputs the generated biometric information to at least one of the display device 13, the storage unit 15, and an external information processing device connected to the signal processing device 10 via the communication unit 14.

[0142] Specifically, for example, the second generating unit 20F displays the biometric information on the display device 13. In this case, the second generating unit 20F may display the generated biometric information on the display device 13. Specifically, for example, the second generating unit 20F displays on the display device 13 information representing the pulse rate represented by the biometric information, time-series changes in periodic signals due to the pulse, and the like.

[0143] The second generating unit 20F may store in the storage unit 15 at least one of the biometric information, the input signal 30 used to calculate the biometric information, the first signal 32, the first processed signal 34, and the first output signal 36.

[0144] The second generator 20F may output the biometric information to various driving devices that are driven in response to the biometric information. Examples of driving devices include air conditioners, lighting devices, audio devices, and mobile terminals that are driven in response to instruction signals. For example, if the comfort level included in the biometric information is below a predetermined value, the second generator 20F may control an air conditioner and a lighting device to create an environment that improves the comfort level. For example, the second generator 20F may control illuminance, temperature and humidity, audio devices, and the like to create an environment that allows for more comfortable sleep, depending on the sleep depth level included in the biometric information. For example, if the health condition included in the biometric information indicates a deterioration, the second generator 20F may control an audio device to output an alert sound or voice.

[0145] Next, an example of the flow of information processing executed by the signal processing device 10 of this embodiment will be described.

[0146] FIG. 6 is a flowchart showing an example of the flow of information processing executed by the signal processing device 10 of this embodiment.

[0147] The acquisition unit 20A acquires one frame of the video captured by the image capturing device 11 (step S100).

[0148] The acquisition unit 20A then sets an ROI for the frame acquired in step S100 (step S102). The acquisition unit 20A then extracts pixel data within the ROI set in step S102 that is included in the frame acquired in step S100, and adds one sample of the pixel data to the beginning (latest location) of the intensity time-series data that is made up of a group of pixel data generated by the processing in step S102 from the previous time onward (step S104).

[0149] Next, the BPF 20B applies a bandpass filter to the intensity value time series data generated in step S104, thereby removing signal components other than the pulse component contained in the intensity value time series data (step S106).

[0150] Next, the noise removal unit 20C applies Auto Gain Control to the intensity time-series data to which the bandpass filter has been applied, thereby removing steep signal noise due to the body movement of the user P from the intensity time-series data (step S108).

[0151] The acquisition unit 20A updates the intensity value time series data of the time window W for the latest first period T1 in the intensity value time series data from which noise was removed in step S108 as the latest first signal 32 (step S110). That is, each time one frame is captured, one sample of data that has passed through the noise removal unit 20C is added to the first signal 32, and the oldest sample of data is removed, thereby updating the first signal 32 as the latest first signal 32. By the process of step S110, the acquisition unit 20A acquires the captured video of the first period T1 in the time window W as the first signal 32, each time the time window W for the first period T1 is moved by one frame along the time series relative to the captured video, which is the input signal 30.

[0152] The first generating unit 20D generates a plurality of first processed signals 34 by performing a plurality of first filtering processes with different output frequencies on the first signal 32 in the time window W of the first period T1, which has been acquired by the acquiring unit 20A and subjected to noise removal by the BPF 20B and the noise removing unit 20C (steps S112 and S114). As described above, the first generating unit 20D may further perform peak hold processing to generate the plurality of first processed signals 34.

[0153] The determination unit 20E determines the first processed signal 34 with the largest signal value from among the multiple first processed signals 34 generated by the processes of steps S112 and S114 as the first output signal 36 (step S116).

[0154] Next, the second generation unit 20F adds the first output signal 36 newly determined in step S116 to the beginning (latest point) along the time series of the output signal waveform 38 consisting of a group of first output signals 36 determined sequentially along the time series by repeating the above process (step S118).

[0155] Then, the determination unit 20E removes high-frequency components that occur when switching the first processed signal 34 to be determined between multiple comb filters CF from the output signal waveform 38 of the new first output signal 36 added in step S118 (step S120).

[0156] The second generating unit 20F generates biometric information of the user P using the output signal waveform 38 from which the high-frequency components have been removed in step S120 (step S122). Then, the second generating unit 20F outputs the generated biometric information (step S124).

[0157] Next, the processing unit 20 determines whether or not to end the process (step S126). For example, the processing unit 20 makes the determination in step S126 by determining whether or not the number of times the processes of steps S100 to S124 have been repeated is equal to or greater than the minimum number of times required to calculate the biometric information of the user P. If the determination in step S126 is negative (step S126: No), the processing unit 20 returns to step S100. On the other hand, if the determination in step S126 is positive (step S126: Yes), the processing unit 20 ends this routine.

[0158] As described above, the signal processing device 10 of this embodiment includes an acquisition unit 20A, a first generation unit 20D, a determination unit 20E, and a second generation unit 20F. The acquisition unit 20A acquires a first signal 32 corresponding to the state of the user P that changes over time during a first period T1. The first generation unit 20D generates a plurality of first processed signals 34 by performing a plurality of first filter processes that output different frequencies on the first signal 32. The determination unit 20E determines a first output signal 36 from among the plurality of first processed signals 34 based on the signal values ​​of each of the plurality of first processed signals 34. The second generation unit 20F generates biometric information based on the first output signal 36.

[0159] Here, the prior art discloses a technique for extracting pulse wave components from a video signal using a single filter that selectively passes fixed frequency components related to the pulse wave. For example, a technique for obtaining cardiac electrical signals by removing noise by using a signal that emphasizes interference at a fixed frequency from an electrical signal captured by a lead wire attached to a subject has been disclosed. However, the signal-to-noise ratio of input signals such as captured video and electrical signals used to generate biometric information is low, making it difficult to generate highly accurate biometric information using the prior art.

[0160] On the other hand, the signal processing device 10 of this embodiment executes a plurality of first filter processes with different output frequencies on the first signal 32 according to the state of the user P that changes over time during the first period T1. Then, the signal processing device 10 of this embodiment determines a first output signal 36 from among the plurality of first processed signals 34 based on the signal values ​​of the plurality of first processed signals 34 obtained by the plurality of first filter processes with different output frequencies on the first signal 32.

[0161] In this way, in the signal processing device 10 of this embodiment, rather than performing a filter process that outputs one type of frequency on the first signal 32, multiple first filter processes that output different frequencies are performed in parallel on the first signal 32, and the first output signal 36 is determined from multiple first processed signals 34 obtained by the multiple first filter processes.

[0162] Therefore, the signal processing device 10 of the present embodiment can determine, as the first output signal 36, the first processed signal 34 generated by one of the multiple first filter processes that outputs a frequency that matches or is closer to the frequency represented by the biometric information of the user P. The signal processing device 10 determines the first output signal 36 from multiple first processed signals 34 obtained by multiple first filter processes that output frequencies that are different from each other, and can therefore determine the first processed signal 34 with reduced noise as the first output signal 36. The signal processing device 10 then generates biometric information using the determined first output signal 36.

[0163] Therefore, the signal processing device 10 of this embodiment can generate highly accurate biometric information.

[0164] In the signal processing device 10 of this embodiment, the acquisition unit 20A sequentially moves a time window W of a first period T1 toward the present along the time series with respect to the input signal 30, and acquires a first signal 32 in the first period T1 each time the time window W is moved. The first generation unit 20D performs multiple first filtering processes on the first signal 32 acquired each time the time window W is moved, thereby generating multiple first processed signals 34 each time the time window W is moved. The determination unit 20E determines a first output signal 36 from the multiple first processed signals 34 each time the time window W is moved, based on the signal values ​​of each of the multiple first processed signals 34.

[0165] Therefore, in the signal processing device 10 of this embodiment, the decision unit 20E can sequentially determine the first output signal 36 that more accurately represents the state of the user P, such as the pulse rate, for each moment in time that is a period equivalent to the movement width of the time window W for the input signal 30. In other words, the decision unit 20E can sequentially determine, for each moment, as the first output signal 36, the first processed signal 34 that is generated by the first filter processing that outputs a frequency that matches the frequency of, for example, the pulse rate of the user P at that moment. That is, the signal processing device 10 of this embodiment can sequentially determine, as the first output signal 36, the first processed signal 34 that is closest to the state of the user P at each moment that changes over time.

[0166] Therefore, the signal processing device 10 of this embodiment generates biometric information using the first output signal 36, thereby making it possible to generate highly accurate biometric information.

[0167] In the signal processing device 10 of this embodiment, the first output signal 36 is determined from a plurality of first processed signals 34 obtained by a plurality of first filter processes each having a different output frequency. Therefore, in the signal processing device 10 of this embodiment, even if the time window W is small, it is possible to determine the first processed signal 34 with reduced noise as the first output signal 36. Therefore, the signal processing device 10 of this embodiment can generate highly accurate biological information.

[0168] FIG. 7 is a schematic diagram showing an example of experimental results obtained using the signal processing device 10 of this embodiment.

[0169] FIG. 7 shows a first signal 32, an output signal waveform 38, and an electrocardiogram waveform 40. The horizontal axis represents time, and the vertical axis represents signal values, i.e., the signal intensity of each waveform. The first signal 32 shown in FIG. 7 is a waveform representing time-series intensity data of a captured image of a user P irradiated with 850 nm near-infrared light. The output signal waveform 38 is a waveform represented by a group of time-series first output signals 36 determined by the determination unit 20E from a plurality of first processed signals 34 generated by a plurality of first filter processes performed by the first generation unit 20D on the first signal 32. The electrocardiogram waveform 40 is a waveform representing an electrocardiogram obtained by measuring the heart rate of the user P with an electrocardiogram measuring device.

[0170] As shown in FIG. 7, the peaks represented by the output signal waveform 38 and the peaks represented by the electrocardiogram waveform 40 almost coincided, with the average error falling within 1.2%.

[0171] Compared to video of user P captured under near-infrared light, hemoglobin absorption changes are smaller in the video of user P captured under visible light. Therefore, the amplitude of the pulse component included in the first signal 32 is small and easily buried in noise. However, the signal processing device 10 of this embodiment executes multiple first filter processes that output different frequencies, and it has been confirmed that it is possible to effectively remove noise and obtain highly accurate pulse data even when the signal-to-noise ratio is low.

[0172] (Variation 1) The processing unit 20 may further perform upsampling on the intensity value time-series data from which noise has been removed by the noise removal unit 20C. The acquisition unit 20A then outputs the first signal 32 for the time window W of the first period T1 to the first generation unit 20D for the intensity value time-series data after the upsampling process. In this case, the processing unit 20 upsamples the number of samples (i.e., the number of frames) of the intensity value time-series data by interpolation. This upsampling process can increase the resolution of the spacing between the combs C of the comb filter CF used in the first generation unit 20D. This upsampling process can increase the number of first filter processes with different output frequencies used in the first generation unit 20D, enabling the generation of multiple first processed signals 34 for finer frequencies.

[0173] The degree of upsampling is not limited. For example, if the intensity value time series data for the time window W of the first period T1 corresponds to a 30 fps image, the processing unit 20 may upsample the intensity value time series data by eight times, i.e., to 240 fps. Then, the acquisition unit 20A outputs the first signal 32 for the time window W of the first period T1 to the first generation unit 20D for the intensity value time series data after the upsampling process. In this case, the first generation unit 20D and the determination unit 20E perform a series of processes eight times each time the time window W is moved, i.e., each time one frame of the image is acquired. In other words, compared to when the intensity value time series data before the upsampling process is used, when this eight-fold upsampled intensity value time series data is used, eight times as many first output signals 36 are output from the determination unit 20E each time one frame of the image is acquired. Therefore, by the processing unit 20 further performing upsampling processing, it is possible to adjust the accuracy of PPI, which is an example of biometric information obtained by the above processing using the intensity value time series data after upsampling, to an error of 1% or less.

[0174] (Modification 2) In the above embodiment, the imaging device 11 includes a CMOS image sensor having RGB color filters, and an RGB captured image of the user P is acquired by imaging.

[0175] In the above embodiment, the acquiring unit 20A calculates an average value of pixel values ​​of each ROI in a plurality of frames included in the captured video during the first period T1, thereby generating intensity value time series data in which the average values ​​of pixel values ​​of each ROI in a plurality of frames included in the captured video during the first period T1 are arranged in a chronological order. In the above embodiment, the acquiring unit 20A acquires this intensity value time series data as the first signal 32 of the time window W of the first period T1 in the input signal 30.

[0176] However, the acquisition unit 20A may acquire a first signal 32 corresponding to at least one of a signal in the G (green) wavelength region contained in the captured image and a signal in the B (blue) wavelength region contained in the captured image, as well as a signal in the R (red) wavelength region contained in the captured image. Light in the G (green) and B (blue) wavelengths is more highly absorbed by hemoglobin than light in the R (red) wavelength. Meanwhile, spurious signal intensity changes due to body movement or vibration are similar for G (green) and B (blue) and R (red). Therefore, the processing unit 20 may remove body movement or vibration components by utilizing the characteristics of different absorbances between wavelengths.

[0177] Specifically, the acquisition unit 20A calculates, for each frame, an average value within the ROI of the difference between the color value in the G or B wavelength region and the color value in the R wavelength region that constitutes the pixel value of each pixel for each pixel in each ROI in multiple frames included in the captured video for the first period T1. Then, the acquisition unit 20A may acquire, as the first signal 32, intensity value time series data in which the calculated average values ​​for each frame are arranged in chronological order.

[0178] The acquisition unit 20A generates first intensity value time series data by calculating, for each frame, an average value within the ROI of color values ​​in the R wavelength region that constitute the pixel value of each pixel for pixels within each ROI in multiple frames included in the captured video during the first period T1. The acquisition unit 20A generates second intensity value time series data by calculating, for each frame, an average value within the ROI of color values ​​in the G wavelength region that constitute the pixel value of each pixel for pixels within each ROI in multiple frames included in the captured video during the first period T1. The acquisition unit 20A may then extract a pulse component from waveforms represented by the first intensity value time series data and the second intensity value time series data through a known regression process using the first intensity value time series data and the second intensity value time series data, thereby generating intensity value time series data and acquiring it as the first signal 32.

[0179] By using a first signal 32 corresponding to a signal in the G wavelength region and at least one of a signal in the B wavelength region and a signal in the R wavelength region, the acquisition unit 20A can acquire corrected intensity value time series data of body movements such as the body movements of the user P as the first signal 32.

[0180] When the acquisition unit 20A acquires a captured image of the user P irradiated with visible light, it may acquire a first signal 32 corresponding to a signal in the G (green) wavelength region, where the absorbance of hemoglobin is high and the absorption by melanin is low.

[0181] In this case, the acquisition unit 20A calculates, for each frame, an average value within the ROI of the color values ​​in the G wavelength region that constitute the pixel value of each pixel for each pixel in each ROI of multiple frames included in the captured video for the first period T1. Then, the acquisition unit 20A acquires, as the first signal 32, intensity value time series data in which the calculated average values ​​for each frame are arranged in chronological order.

[0182] (Variation 3) In the above embodiment, the description is given assuming that the biological information is information related to the pulse of the user P. However, the biological information may be information related to at least one of the pulse of the user P and the respiration of the user P. In other words, the biological information may include information related to respiration.

[0183] Even when the biological information includes information related to respiration, the acquisition unit 20A may acquire the first signal 32 corresponding to the captured video, as in the above embodiment. This is because breathing causes body movement of the user P, and therefore the amount of reflected light from the user P detected in the captured video periodically changes with the user P's breathing.

[0184] When the second generating unit 20F generates information related to respiration as biometric information, the acquiring unit 20A may set an area where the face or feature points of the user P are tracked as the ROI to be set for the captured video of the first period T1. This is because the pixel values ​​of the tracked area are likely to change in response to the respiration of the user P.

[0185] When the second generating unit 20F generates information related to respiration as biological information, the type of light irradiated onto the user P during imaging is not limited.

[0186] When the second generation unit 20F generates information regarding breathing as biometric information, the acquisition unit 20A may set an area including a boundary such as the outline of user P in the captured image of user P as the ROI in order to more closely observe the body movements of user P.

[0187] When the second generating unit 20F generates information related to respiration as biological information, the BPF 20B may use a filter that outputs a respiration frequency component included in the first signal 32, and the first generating unit 20D may execute a plurality of first filter processes that output different frequencies within a frequency band that includes the respiration frequency. In this case, at least one of the BPF 20B, the first generating unit 20D, and the determining unit 20E may set their respective frequency bands so that the pulse component can be removed.

[0188] For example, the first generator 20D may be configured to include multiple comb filters CF whose output frequencies are gradually changed within a frequency range of 50 bpm to 90 bpm. It is known that the breathing period of a user P is generally approximately 3 to 5 seconds when at rest. Therefore, the first generator 20D may be configured to include multiple comb filters CF whose output frequencies are gradually changed within a frequency range of 3 to 5 seconds, i.e., 12 to 20 bpm. The first generator 20D may be configured to include multiple comb filters CF whose output frequencies are gradually changed within a frequency range of 1 to 10 seconds, i.e., 6 to 60 bpm. The first generator 20D may be configured to include multiple comb filters CF whose output frequencies are gradually changed within a frequency range of 6 to 50 bpm, so as to exclude pulse frequencies.

[0189] When the second generating unit 20F generates information related to respiration as biological information, the upsampling process performed in the above embodiment may not be necessary because the respiration cycle is longer than the pulse rate.

[0190] The first period T1 of the time window W used by the acquisition unit 20A may be a period of approximately two breathing cycles, for example, at least six seconds or more.

[0191] It is known that the breathing cycle of user P generally occurs at intervals of about 3 to 5 seconds when the user P is at rest. The breathing cycle of user P also changes depending on the indicators representing the state of user P described above, such as alertness, calmness, pleasantness / unpleasantness, comfort, concentration, slowness, relaxation, degree of sympathetic dominance, degree of parasympathetic dominance, depth of sleep, health condition, level of attention, psychological state, etc.

[0192] Therefore, the signal processing device 10 can generate biometric information including information related to breathing from the captured video of the user P in the same manner as in the above embodiment.

[0193] (Variation 4) In the above embodiment, an example has been described in which the first generation unit 20D executes a plurality of first filter processes by inputting the first signal 32 to each of a plurality of comb filters CF that output different frequencies.

[0194] However, the first generation unit 20D may perform a plurality of first filtering processes with different output frequencies, and the filters used are not limited to the comb filter CF. For example, the first generation unit 20D may perform a plurality of first filtering processes by inputting the first signal 32 to a plurality of band-pass filters that pass different frequency bands.

[0195] (Variation 5) In the above embodiment, an example was described in which the image capturing device 11 includes a CMOS image sensor with RGB color filters, captures RGB captured images of the user P by capturing an image, and uses the captured images as the input signal 30. However, the input signal 30 is not limited to captured images and may be any signal that corresponds to the state of the user P that changes over time. For example, the input signal 30 may be a signal detected by an optical sensor that detects changes in reflected light from a specific area of ​​the user P. The optical sensor includes at least one photodiode. The optical sensor may be a single-pixel sensor having a single photodiode.

[0196] The present technology may be configured as follows.

[0197] (1) A signal processing method executed by a signal processing device, comprising: an acquisition step of acquiring a first signal corresponding to a state of a user that changes over time in a first period; a first generation step of generating a plurality of first processed signals by performing a plurality of first filter processes with different output frequencies on the first signal; a determination step of determining a first output signal from among a plurality of first processed signals based on the signal values ​​of each of the plurality of first processed signals; and a second generation step of generating biometric information based on the first output signal.

[0198] (2) The signal processing method according to (1), wherein the acquiring step sequentially shifts a time window of the first period along a time series with respect to the input signal, and acquires the first signal in the first period of the time window each time the time window is shifted; the first generating step generates a plurality of the first processed signals each time the time window is shifted by performing a plurality of the first filter processes on the first signal acquired each time the time window is shifted; and the determining step determines the first output signal from the plurality of first processed signals each time the time window is shifted, based on the signal values ​​of each of the plurality of first processed signals.

[0199] (3) The signal processing method according to (1) or (2), wherein the second generating step generates the biological information based on output signal waveforms consisting of a time-series continuous group of the first output signals sequentially determined for each shift of the time window.

[0200] (4) The signal processing method according to any one of (1) to (3), wherein the first generating step executes a plurality of first filter processes in which the first signal is input to a plurality of comb filters each having a different emphasis time period, and generates a plurality of the first processed signals output from the plurality of comb filters.

[0201] (5) The signal processing method according to (4), wherein each of the plurality of comb filters includes a plurality of combs representing sampling timings for sampling a signal at periods corresponding to different frequencies to be output in a time window of the first period, and a total signal value obtained by sampling the first signal at each timing of the plurality of combs is output as the first processed signal.

[0202] (6) The signal processing method according to (4) or (5), wherein in each of the plurality of comb filters, a weight value of at least one of the plurality of combs included in the time window is different from a weight value of the other combs.

[0203] (7) The signal processing method according to (5) or (6), wherein, in each of the plurality of comb filters, a weight value of a central comb in the time axis direction among the plurality of combs included in the time window is greater than weight values ​​of the other combs other than the central comb.

[0204] (8) The signal processing method according to any one of (1) to (7), wherein the first generating step executes the first filtering process multiple times to output frequencies within a frequency range according to a predetermined condition related to the user.

[0205] (9) The signal processing method according to any one of (1) to (8), wherein the determining step determines a first output signal from among the plurality of first processed signals, the first processed signals having a frequency within a frequency range according to at least one of a predetermined condition related to the user and a reliability determination result of the biometric information generated based on the first output signal.

[0206] (10) The signal processing method according to any one of (1) to (9), wherein the determining step determines, as the first output signal, the first processed signal having the largest signal value among the plurality of first processed signals.

[0207] (11) The signal processing method according to any one of (1) to (10), wherein the first generating step generates, for each of the plurality of first processed signals generated by execution of the plurality of first filter processes, a weighted average result of the first processed signal generated by a previous execution of the first filter process and the first processed signal generated by a current execution of the first filter process as the first processed signal generated by the current execution of the first filter process.

[0208] (12) The signal processing method according to (11), wherein the first generation step, when the first processed signal generated by the current execution of the first filter processing is greater than the first processed signal generated by the previous execution of the first filter processing, generates a weighted average result as the first processed signal generated by the current execution of the first filter processing, in which the weighting value of the first processed signal generated by the current execution of the first filter processing is made higher than the weighting value of the first processed signal generated by the previous execution of the first filter processing.

[0209] (13) The signal processing method according to (11), wherein the first generating step generates, when the first processed signal generated by the current execution of the first filtering process is smaller than the first processed signal generated by the previous execution of the first filtering process, a weighted average value in which the weighting value of the first processed signal generated by the current execution of the first filtering process is lower than the weighting value of the first processed signal generated by the previous execution of the first filtering process, as the first processed signal generated by the current execution of the first filtering process.

[0210] (14) The signal processing method according to any one of (1) to (13), wherein the biological information is information relating to at least one of the pulse rate of the user and the respiration of the user.

[0211] (15) The signal processing method according to any one of (1) to (14), wherein the acquiring step acquires the first signal from captured video data of the user.

[0212] (16) The signal processing method according to any one of (1) to (15), wherein the acquiring step acquires the first signal corresponding to at least one of a signal in a green wavelength region included in the captured video data of the user and a signal in a blue wavelength region included in the captured video data of the user, and a signal in a red wavelength region included in the captured video data of the user.

[0213] (17) A signal processing device comprising: an acquisition unit that acquires a first signal corresponding to a state of a user that changes over time in a first period; a first generation unit that generates a plurality of first processed signals by performing a plurality of first filter processes with different output frequencies on the first signal; a decision unit that decides a first output signal from among a plurality of first processed signals based on the signal values ​​of each of the plurality of first processed signals; and a second generation unit that generates biometric information based on the first output signal.

[0214] (18) A program for causing a computer to execute the following steps: an acquisition step of acquiring a first signal corresponding to a user's state that changes over time during a first period; a first generation step of generating a plurality of first processed signals by performing a plurality of first filter processes with different output frequencies on the first signal; a determination step of determining a first output signal from among a plurality of first processed signals based on the signal values ​​of each of the plurality of first processed signals; and a second generation step of generating biometric information based on the first output signal.

[0215] (Other) A method executed by a computer, the method comprising: acquiring a biosignal of a user; a period during which the biosignal was acquired includes an ith period, where i is 1, ..., n, and n is an integer equal to or greater than 2; the first period, ..., and the nth period are different from one another; the biosignal includes a first biosignal corresponding to the first period, ..., and an nth biosignal corresponding to the nth period; and performing a jth filter process on the ith biosignal, thereby filtering the signal. ｉｊ is generated, j=1, . . . , m, m is an integer equal to or greater than 2, and the j-th filtering process is performed on the i-th biological signal at a sampling period ｊ and sampling is performed at the sampling period １ , ..., the sampling period ｍ are different from each other, and determine the i-th information for the i-th period, and the i-th information is used to determine the signal ｉ１ The maximum value determined based on ｉ１ , ..., the signal ｉｍ The maximum value determined based on ｉmthe determined first information, ..., the determined nth information, being the maximum value among the determined first information, ..., the determined nth information, and generating biometric information of the user based on the determined first information, ..., the determined nth information.

[0216] Although the embodiments have been described above, they are presented as examples and are not intended to limit the scope of the disclosure. The novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the disclosure. The above embodiments are included within the scope and spirit of the disclosure, as well as within the scope of the disclosure and its equivalents as set forth in the claims.

[0217] Although the signal processing method and the like have been described above according to the embodiments, the aspects of the signal processing method and the like are not limited to the embodiments. The present disclosure includes forms obtained by applying various modifications to the embodiments that a person skilled in the art would conceive, and forms realized by arbitrarily combining the components and functions of the embodiments within the scope of the present disclosure.

[0218] For example, a process performed by a specific component in an embodiment may be performed by another component instead of the specific component. The order of multiple processes may be changed, or multiple processes may be performed in parallel. Ordinal numbers such as first and second used in the description may be changed, removed, or newly added as appropriate. These ordinal numbers do not necessarily correspond to a meaningful order and may be used to identify elements.

[0219] 10 Signal processing device 20A Acquisition unit 20D First generation unit 20E Determination unit 20F Second generation unit

Claims

1. A signal processing method executed by a signal processing device, comprising: an acquisition step of acquiring a first signal corresponding to a user's state that changes over time in a first period; a first generation step of generating a plurality of first processed signals by performing a plurality of first filter processes with different output frequencies on the first signal; a determination step of determining a first output signal from among a plurality of first processed signals based on the signal values ​​of each of the plurality of first processed signals; and a second generation step of generating biometric information based on the first output signal.

2. The signal processing method according to claim 1, wherein the acquisition step sequentially shifts a time window of the first period along a time series with respect to the input signal, and acquires the first signal in the first period of the time window each time the time window is shifted; the first generation step generates a plurality of the first processed signals each time the time window is shifted by performing a plurality of the first filter processes on the first signal acquired each time the time window is shifted; and the determination step determines the first output signal from the plurality of first processed signals each time the time window is shifted based on the signal values ​​of each of the plurality of first processed signals.

3. The signal processing method according to claim 2, wherein the second generation step generates the biometric information based on output signal waveforms consisting of a time-series continuous group of the first output signals determined sequentially for each shift of the time window.

4. The signal processing method according to claim 1, wherein the first generation step comprises performing a plurality of first filter processes by inputting the first signal to each of a plurality of comb filters having different emphasis time periods, and generating a plurality of first processed signals to be output from each of the plurality of comb filters.

5. A signal processing method according to claim 4, wherein each of the plurality of comb filters includes a plurality of combs representing sampling timings for sampling a signal at periods corresponding to different frequencies to be output in a time window of the first period, and a total signal value obtained by sampling the first signal at each timing of the plurality of combs is output as the first processed signal.

6. The signal processing method according to claim 5, wherein in each of the plurality of comb filters, the weight value of at least one of the plurality of combs included in the time window is different from the weight values ​​of the other combs.

7. A signal processing method according to claim 5, wherein, in each of the plurality of comb filters, the weight value of the comb in the center of the time axis direction among the plurality of combs included in the time window is greater than the weight values ​​of the other combs other than the comb in the center.

8. The signal processing method according to claim 1, wherein the first generation step executes a plurality of first filter processes that output frequencies within a frequency range according to a predetermined condition related to the user.

9. A signal processing method as described in claim 1, wherein the determination step determines a first output signal from among the plurality of first processed signals, the first processed signals having frequencies within a frequency range corresponding to at least one of a predetermined condition related to the user and a reliability determination result of the biometric information generated based on the first output signal.

10. The signal processing method according to claim 1, wherein said determining step determines, as said first output signal, said first processed signal having the largest signal value among said plurality of first processed signals.

11. A signal processing method as described in claim 1, wherein the first generation step, for each of the plurality of first processed signals generated by execution of the plurality of first filter processes, generates, for each of the plurality of first filter processes, a weighted average result of the first processed signal generated by a previous execution of the first filter process and the first processed signal generated by a current execution of the first filter process as the first processed signal generated by the current execution of the first filter process.

12. A signal processing method as described in claim 11, wherein the first generation step, if the first processed signal generated by the current execution of the first filter processing is greater than the first processed signal generated by the previous execution of the first filter processing, generates a weighted average result as the first processed signal generated by the current execution of the first filter processing, in which the weighting value of the first processed signal generated by the current execution of the first filter processing is made higher than the weighting value of the first processed signal generated by the previous execution of the first filter processing.

13. A signal processing method as described in claim 11, wherein the first generation step, if the first processed signal generated by the current execution of the first filter processing is smaller than the first processed signal generated by the previous execution of the first filter processing, generates a weighted average value as the first processed signal generated by the current execution of the first filter processing, in which the weighting value of the first processed signal generated by the current execution of the first filter processing is lower than the weighting value of the first processed signal generated by the previous execution of the first filter processing.

14. The signal processing method according to claim 1, wherein the biological information is information relating to at least one of the pulse rate of the user and the respiration rate of the user.

15. The signal processing method according to claim 1, wherein the acquisition step acquires the first signal from video data captured by the user.

16. A signal processing method as described in claim 1, wherein the acquisition step acquires the first signal corresponding to at least one of a signal in the green wavelength region contained in the captured video data of the user and a signal in the blue wavelength region contained in the captured video data of the user, and a signal in the red wavelength region contained in the captured video data of the user.

17. A signal processing device comprising: an acquisition unit that acquires a first signal corresponding to a user's state that changes over time during a first period; a first generation unit that generates a plurality of first processed signals by performing a plurality of first filter processes with different output frequencies on the first signal; a decision unit that determines a first output signal from among a plurality of first processed signals based on the signal values ​​of each of the plurality of first processed signals; and a second generation unit that generates biometric information based on the first output signal.

18. A program for causing a computer to execute the following steps: an acquisition step of acquiring a first signal corresponding to a user's state that changes over time during a first period; a first generation step of generating a plurality of first processed signals by performing a plurality of first filter processes with different output frequencies on the first signal; a determination step of determining a first output signal from among a plurality of first processed signals based on the signal values ​​of each of the plurality of first processed signals; and a second generation step of generating biometric information based on the first output signal.

Images