Electronic device, method for controlling electronic device, and program

Abstract

An electronic device includes: a transmission unit that transmits a transmission wave; a receiving unit that receives a reflected wave of the transmission wave from the target; and a signal processing unit that detects the distance position, direction, and speed of the target on the basis of a converted signal obtained by Fourier-converting the beat signal of the transmission wave and the reflected wave. The signal processing unit has an extraction unit that extracts heartbeat information of the subject on the basis of the detected distance, direction, and speed of the subject. And an extraction unit that extracts signal components corresponding to vibrations accompanying the heartbeat of the subject from the converted signal using a plurality of window functions, selects the most suitable heart sound signal as heart sound by applying a learning classifier to the extracted signal components, and extracts the heart sound by performing envelope processing on the most suitable heart sound signal. A value representing the interval of peak time is acquired from the optimal signal, and the heartbeat interval of the target is extracted on the basis of the variance and median value representing the value of the interval.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority from Japanese Patent Application No. 2022-148632 filed in Japan on September 16, 2022, and all disclosed contents of the prior application are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to an electronic device, a control method of an electronic device, and a program. Background Art

[0004] For example, in the automotive-related industries, the technology of measuring the distance between the vehicle and a predetermined object has attracted attention. In particular, in recent years, various radar (RADAR (Radio Detecting and Ranging)) technologies have been studied, which measure the distance to an object by transmitting radio waves such as millimeter waves and receiving reflected waves reflected by objects such as obstacles. It is expected that the importance of such distance measurement technology will increase in the future as technologies for assisting drivers and technologies related to autonomous driving that automates part or all of driving are developed.

[0005] In addition, various proposals have been made for technologies such as detecting the presence of a predetermined object by receiving a reflected wave after the transmitted radio wave is reflected by the predetermined object. For example, Patent Document 1 proposes a device that can detect the presence of a person and the biological information of the person by using microwaves. In addition, Patent Document 2 proposes a device that detects vital signs such as the breathing or heartbeat frequency of a living body based on the reflected signal of a microwave radar.

[0006] Prior art literature

[0007] Patent Literature

[0008] Patent Document 1: Japanese Patent Application Publication No. 2002-71825

[0009] Patent Document 2: Japanese Patent Application Publication No. 2021-32880 Summary of the invention

[0010] An electronic device according to one embodiment includes:

[0011] a transmitting unit, which transmits a transmitting wave;

[0012] a receiving unit that receives a reflected wave of the transmitted wave from a target object; and

[0013] A signal processing unit, which detects the distance position, direction and speed of the target object based on a transformed signal obtained by Fourier transforming a beat signal between the transmission wave and the reflected wave;

[0014] The signal processing unit includes an extraction unit, which extracts heartbeat information of the target object based on the detected distance, direction and speed of the target object.

[0015] The extraction unit,

[0016] A signal component corresponding to the vibration of the heartbeat of the target object is extracted from the transformed signal using a plurality of window functions.

[0017] By applying a learning classifier to the extracted signal components, the most suitable heart sound signal is selected as the heart sound.

[0018] By performing envelope processing on the most suitable heart sound signal, the most suitable best signal is selected as the heartbeat.

[0019] A value indicating an interval of peak times is acquired from the optimal signal, and a heartbeat interval of the target object is extracted based on a variance and a median value of the value indicating the interval.

[0020] A method for controlling an electronic device according to one embodiment includes:

[0021] a step of transmitting a transmission wave from a transmitting unit;

[0022] A step of receiving a reflected wave of the transmitted wave from a target object during reception;

[0023] A step of detecting the distance position, direction and speed of the target object based on a transformed signal after Fourier transforming the beat signal of the transmitted wave and the reflected wave; and

[0024] A step of extracting heartbeat information of the object based on the detected distance, direction and speed of the object;

[0025] In the step of extracting the heartbeat information of the object,

[0026] A signal component corresponding to the vibration of the heartbeat of the target object is extracted from the transformed signal using a plurality of window functions.

[0027] By applying a learning classifier to the extracted signal components, the most suitable heart sound signal is selected as the heart sound.

[0028] By performing envelope processing on the most suitable heart sound signal, the most suitable best signal is selected as the heartbeat.

[0029] A value indicating an interval of peak times is acquired from the optimal signal, and a heartbeat interval of the target object is extracted based on a variance and a median value of the value indicating the interval.

[0030] A program according to one embodiment causes an electronic device to execute the following steps:

[0031] sending a transmission wave from a transmitting unit;

[0032] receiving a reflected wave of the transmitted wave from a target object in receiving;

[0033] Detecting the distance position, direction and speed of the target object based on a transformed signal after Fourier transforming the beat signal of the transmitted wave and the reflected wave; and

[0034] Extracting heartbeat information of the object based on the detected distance, direction and speed of the object;

[0035] In the step of extracting the heartbeat information of the object,

[0036] A signal component corresponding to the vibration of the heartbeat of the target object is extracted from the transformed signal using a plurality of window functions.

[0037] By applying a learning classifier to the extracted signal components, the most suitable heart sound signal is selected as the heart sound.

[0038] By performing envelope processing on the most suitable heart sound signal, the most suitable best signal is selected as the heartbeat.

[0039] A value indicating an interval of peak times is acquired from the optimal signal, and a heartbeat interval of the target object is extracted based on a variance and a median value of the value indicating the interval. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a diagram for explaining how to use an electronic device according to one embodiment.

[0041] Figure 2 This is a functional block diagram schematically showing the structure of an electronic device according to one embodiment.

[0042] Figure 3 This is a diagram for explaining the structure of a signal processed by an electronic device according to one embodiment.

[0043] Figure 4 This is a diagram for explaining signal processing by an electronic device according to one embodiment.

[0044] Figure 5 This is a diagram for explaining signal processing by an electronic device according to one embodiment.

[0045] Figure 6 This is a diagram for explaining signal processing by an electronic device according to one embodiment.

[0046] Figure 7A diagram schematically showing an example of the arrangement of antennas in an antenna array of an electronic device according to an embodiment and an operation principle.

[0047] Figure 8 This is a diagram showing an example of the arrangement of antennas in an antenna array of an electronic device according to an embodiment.

[0048] Fig. 9 This is a diagram showing an example of signal processing by an electronic device according to one embodiment.

[0049] Fig.10 This is a diagram showing an example of signal processing by an electronic device according to one embodiment.

[0050] Fig.11 This is a flowchart illustrating a comparative example of the operation of the electronic device according to one embodiment.

[0051] Fig.12 This is a flowchart illustrating the operation of an electronic device according to one embodiment.

[0052] Fig.13 This is a flowchart illustrating the operation of an electronic device according to one embodiment.

[0053] Fig.14 A diagram illustrating a multi-window process of an electronic device according to an embodiment.

[0054] Fig.15 This is a diagram showing the relationship between the ranks of a target signal and a noise signal in singular value decomposition of an electronic device according to one embodiment.

[0055] Fig.16 FIG. 1 is a diagram schematically showing a multi-resolution analysis based on discrete wavelet transform of an electronic device according to one embodiment.

[0056] Fig.17 This is a diagram schematically illustrating a process of classifying and identifying envelopes of optimal heart sounds by an electronic device according to one embodiment.

[0057] Fig.18 This is a diagram showing the result of one-dimensionalization processing performed by continuous wavelet transform in an electronic device according to one embodiment.

[0058] Fig.19 is a diagram schematically showing machine learning of an electronic device according to one embodiment.

[0059] Fig. 20 This is a diagram showing an example of classification and recognition of signals by an electronic device according to one embodiment.

[0060] Fig.21This is a diagram showing an example of mean absolute percentage error of an envelope of optimal heart sounds by an electronic device according to one embodiment.

[0061] Fig. 22 This is a diagram showing an enlarged view of envelopes of two heart sounds detected by the electronic device according to one embodiment.

[0062] Fig.23 This is a diagram showing a histogram of the peak intervals of heart sounds detected by the electronic device according to one embodiment.

[0063] Fig.24 This is a diagram for explaining extraction of an envelope of heart sounds by an electronic device according to one embodiment.

[0064] Fig.25 This is a diagram illustrating DP matching for RRI estimation in an electronic device based on one embodiment.

[0065] Fig.26 This is a diagram showing a time series waveform of heart sounds obtained by an electronic device according to one embodiment.

[0066] Fig. 27 This is a diagram showing a time series waveform of heart sounds obtained by an electronic device according to one embodiment.

[0067] Fig.28 This is a diagram showing a timing waveform of RRI obtained by an electronic device according to one embodiment.

[0068] Fig.29 This is a diagram showing the power spectrum density of RRI obtained by an electronic device according to one embodiment.

[0069] Fig.30 This is a diagram showing an example of the arrangement of antennas in an antenna array of an electronic device according to an embodiment.

[0070] Fig.31 is a diagram illustrating a negative peak of a reconstruction error output of an autoencoder in processing by an electronic device according to one embodiment.

[0071] Fig.32 is a diagram illustrating a negative peak of a reconstruction error output of an autoencoder in processing by an electronic device according to one embodiment. DETAILED DESCRIPTION

[0072] If it is possible to detect weak vibrations such as the heartbeat of a human body with good accuracy by transmitting and receiving radio waves such as millimeter waves, it can be expected to play a role in various fields. The purpose of the present disclosure is to provide an electronic device, a control method and a program for the electronic device that can detect the heartbeat of a human body with good accuracy by transmitting and receiving radio waves. According to one embodiment, it is possible to provide an electronic device, a control method and a program for the electronic device that can detect the heartbeat of a human body with good accuracy by transmitting and receiving radio waves. Below, one embodiment is described in detail with reference to the accompanying drawings.

[0073] In the present disclosure, an "electronic device" may be a device driven by electricity. In addition, a "user" may be a person (typically a human) or an animal who uses a system and / or an electronic device of an embodiment. A user may also include a person who uses an electronic device of an embodiment to monitor an object such as a human. In addition, an "object" may also be an object (such as a human or an animal) that is monitored using an electronic device of an embodiment. Furthermore, a user may be included in an object.

[0074] In the present disclosure, "heart beat" may be the beating of the heart. In addition, "beat" may be the contraction movement of the heart in rhythm.

[0075] In addition, in the present disclosure, "heart rate" refers to the number of times the heart beats in a certain period of time. For example, the heart rate can be the number of beats per minute. When the heart pumps blood, the arteries pulsate. Therefore, the number of times the arteries pulsate can also be referred to as the pulse number or simply pulse.

[0076] Furthermore, in the present disclosure, "heart sound" may be the sound of heart beating. That is, heart sound may be the sound produced when the heart contracts / relaxes. Here, "heart sound" may be composed of a first sound and a second sound, where a low and long first sound is produced based on the tension of the ventricular muscle, the closing of the mitral valve, the beginning of blood ejection into the artery and / or the acceleration of blood flow, followed by a high and short second sound produced by the closing of the aortic valve and / or the closing of the pulmonary valve.

[0077] In the present disclosure, heart sounds are not necessarily limited to physical sounds based on air vibrations, but may also refer to vibrations caused by the beating (pulsation) of the heart. For example, in the present disclosure, heartbeats sometimes include a vibration source, and heart sounds sometimes include vibrations caused by the vibration source. Furthermore, in the present disclosure, heartbeats may also include heart sounds depending on the situation.

[0078] In addition, in the present disclosure, heart sounds may be vibrations of the body accompanying the movement of the heart. Heart sounds may be, for example, vibrations of the entire body or the head, neck, chest, throat, arms, legs, wrists, or other body parts. Heart sounds may be, for example, vibrations of the surface skin of the entire body or the head, neck, chest, throat, arms, legs, wrists, or other body parts. Heart sounds may be, for example, vibrations of clothes, underwear, glasses, or other accessories worn by the subject accompanying vibrations of the surface skin of the entire body or the head, neck, chest, throat, arms, legs, wrists, or other body parts. Heartbeats may be the pulsation of the heart itself. Heartbeat intervals or heart rate, etc. may also be calculated based on the movement of the heartbeat. In the present disclosure, the pulsation of the heart may also be set to beats.

[0079] An electronic device of one embodiment can detect the heartbeat of objects such as humans that exist around the electronic device. Therefore, it is envisioned that the scene of using an electronic device of one embodiment may be, for example, a specific institution used by people who engage in social activities such as a company, a hospital, a nursing home, a school, a gym, and a nursing institution. For example, if it is a company, it is extremely important to grasp and / or manage the health status of employees, etc. Similarly, if it is a hospital, it is extremely important to grasp and / or manage the health status of patients and medical practitioners, etc., or if it is a nursing home, it is extremely important to grasp and / or manage the health status of residents and staff, etc. The scene of using an electronic device of one embodiment is not limited to the above-mentioned companies, hospitals, nursing homes, etc., but may also be any institution that expects to grasp and / or manage the health status of the object. Any institution may also include non-commercial institutions such as the user's residence. In addition, the scene of using an electronic device of one embodiment is not limited to indoors, but may also be outdoors. For example, the scene of using an electronic device of one embodiment may also be in a moving body such as a tram, a bus, and an airplane, as well as a station and a boarding and alighting station. In addition, the scene where the electronic device of one embodiment is used may be a mobile body such as a car, an aircraft, or a ship; a hotel, a user's home, a living room, a bathroom, a toilet, or a bedroom of a home, etc.

[0080] An electronic device of one embodiment can be used, for example, for detecting or monitoring the heartbeat of a subject such as a care recipient or a care receiver in a nursing institution or the like. In addition, an electronic device of one embodiment can also, for example, issue a prescribed warning to the person and / or others when it is confirmed that the heartbeat of a subject such as a care recipient or a care receiver is abnormal. Therefore, according to an electronic device of one embodiment, for example, the person and / or the staff of a nursing institution can recognize that, for example, the pulse of a subject such as a care recipient or a care receiver is abnormal. On the other hand, an electronic device of one embodiment can also report the notification to the person and / or others when it is not confirmed that the heartbeat of a subject such as a care recipient or a care receiver is abnormal (for example, confirmed to be normal). Therefore, according to an electronic device of one embodiment, for example, the person and / or the staff of a nursing institution can recognize that the pulse of a subject such as a care recipient or a care receiver is normal.

[0081] In addition, the electronic device of one embodiment may detect the pulse of animals other than humans. As an example, the electronic device of one embodiment described below detects the pulse of a human using a sensor based on a technology such as a millimeter wave radar.

[0082] An electronic device of one embodiment can be set on any stationary object or any moving object. An electronic device of one embodiment can send a transmission wave from a transmitting antenna to the surrounding of the electronic device. In addition, an electronic device of one embodiment can receive a reflected wave after the transmission wave is reflected from a receiving antenna. At least one of the transmitting antenna and the receiving antenna can be set on the electronic device, for example, it can also be set on a radar sensor, etc.

[0083] As a typical example, the following describes a case where an electronic device of one embodiment is stationary. On the other hand, the object (human) whose pulse is detected by the electronic device of one embodiment may be stationary, may be moving, or may move the body in a stationary state. The electronic device of one embodiment is similar to a conventional radar sensor, and can measure the distance between the electronic device and the object when the object around the electronic device is movable. In addition, the electronic device of one embodiment can measure the distance between the electronic device and the object even if both the electronic device and the object are stationary.

[0084] Hereinafter, an electronic device according to an embodiment will be described in detail with reference to the drawings. First, an example of detecting an object by an electronic device according to an embodiment will be described.

[0085] Figure 1 This is a diagram for explaining an example of how to use the electronic device according to one embodiment. Figure 1An example of an electronic device having a sensor function including a transmitting antenna and a receiving antenna according to one embodiment is shown.

[0086] like Figure 1 As shown, an electronic device 1 of an embodiment may have a transmitting unit and a receiving unit described later. As described later, the transmitting unit may have a transmitting antenna array 24. In addition, the receiving unit may have a receiving antenna array 31. The specific structures of the electronic device 1, the transmitting unit, and the receiving unit will be described later. For the convenience of description, Figure 1 The electronic device 1 is shown to have a transmitting antenna array 24 and a receiving antenna array 31. In addition, the electronic device 1 may appropriately include a signal processing unit 10 ( Figure 2 ) or at least one of the other functional units. In addition, the electronic device 1 may also have a signal processing unit 10 ( Figure 2 ) or at least one of the other functional parts. Figure 1 In the embodiment, the electronic device 1 may move or remain stationary without moving.

[0087] exist Figure 1 In the example shown, a transmitting unit having a transmitting antenna array 24 and a receiving unit having a receiving antenna array 31 are simplified in the electronic device 1. The electronic device 1 may also have, for example, a plurality of transmitting units and / or a plurality of receiving units. The transmitting unit may have a transmitting antenna array 24 composed of a plurality of transmitting antennas. In addition, the receiving unit may have a receiving antenna array 31 composed of a plurality of receiving antennas. Here, the position where the transmitting unit and / or the receiving unit are arranged in the electronic device 1 is not limited to Figure 1 The position shown may also be appropriately set to other positions. In addition, the number of the transmitting unit and / or the receiving unit may be any number more than one according to various conditions (or requirements) such as the detection range and / or detection accuracy of the heartbeat by the electronic device 1.

[0088] As described later, the electronic device 1 transmits electromagnetic waves as transmission waves from the transmission antenna array 24. For example, in a predetermined object (such as Figure 1 When an object 200 (shown in FIG. 2 ) exists around the electronic device 1, at least a portion of the transmission wave transmitted from the electronic device 1 is reflected by the object and becomes a reflected wave. Then, by receiving such a reflected wave using, for example, the receiving antenna array 31 of the electronic device 1, the electronic device 1 can detect the object as a target.

[0089] The electronic device 1 having the transmitting antenna array 24 can typically be a radar (RADAR (Radio Detecting and Ranging)) sensor that transmits and receives radio waves. However, the electronic device 1 is not limited to a radar sensor. The electronic device 1 of one embodiment can also be a sensor based on technology such as LIDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging) of light waves. Such a sensor can be configured to include a patch antenna, etc. Technologies such as RADAR and LIDAR are already well known, so the detailed description is appropriately simplified or omitted. Furthermore, the electronic device 1 of one embodiment can also be a sensor based on technology such as transmitting and receiving sound waves or ultrasonic waves and detecting objects.

[0090] Figure 1 The electronic device 1 shown receives the reflected wave of the transmission wave transmitted from the transmission antenna array 24 through the reception antenna array 31. In this way, the electronic device 1 can detect the specified object 200 existing within a specified distance from the electronic device 1 as a target. Figure 1 As shown, the electronic device 1 can measure the distance L between the electronic device 1 and the predetermined object 200. In addition, the electronic device 1 can also measure the relative speed between the electronic device 1 and the predetermined object 200. Furthermore, the electronic device 1 can also measure the direction (arrival angle θ) in which the reflected wave from the predetermined object 200 reaches the electronic device 1.

[0091] exist Figure 1 In the example, the XY plane may be a plane substantially parallel to the ground surface. In this case, Figure 1 The positive direction of the Z axis shown can represent vertical upward. Figure 1 In the embodiment, the electronic device 1 can be arranged on a plane parallel to the XY plane. Figure 1 In the figure, the object 200 may be in a state of standing upright on the ground surface which is substantially parallel to the XY plane, for example.

[0092] Here, the object 200 may be, for example, a human being existing around the electronic device 1. In addition, the object 200 may also be a living being other than a human being, such as an animal existing around the electronic device 1. As described above, the object 200 may move, or may stop or remain stationary. In the present disclosure, the objects detected by the electronic device 1 include living beings such as humans, dogs, cats, and other animals such as horses, in addition to inanimate objects such as arbitrary objects. The objects detected by the electronic device 1 of the present disclosure may also include object markers including humans, objects, and animals detected by radar technology. In the present disclosure, object markers may also include humans, objects, and animals. In the following, it is assumed that the object such as the object 200 existing around the electronic device 1 is a human being (or an animal) for explanation. In the following, "object 200" is appropriately represented as "subject 200". In the present disclosure, the object marker may also be the above-mentioned object 200.

[0093] exist Figure 1 In FIG. 1 , the ratio of the size of the electronic device 1 to the size of the object 200 does not necessarily represent an actual ratio. Figure 1 , the transmitting antenna array 24 of the transmitting unit and the receiving antenna array 31 of the receiving unit are shown to be disposed outside the electronic device 1. However, in one embodiment, the transmitting antenna array 24 of the transmitting unit and / or the receiving antenna array 31 of the receiving unit may be disposed at various positions of the electronic device 1. For example, in one embodiment, the transmitting antenna array 24 of the transmitting unit and / or the receiving antenna array 31 of the receiving unit may also be disposed inside the electronic device 1, and not be visible from the appearance of the electronic device 1.

[0094] As a typical example, the transmission antenna of the electronic device 1 transmits radio waves in a frequency band such as millimeter waves (above 30 GHz) or quasi-millimeter waves (for example, around 20 GHz to 30 GHz). On the other hand, the transmission antenna of the electronic device 1 may also transmit radio waves having a frequency bandwidth of 4 GHz, such as 77 GHz to 81 GHz.

[0095] Figure 2 1 is a functional block diagram schematically showing a configuration example of the electronic device 1 according to one embodiment. Hereinafter, an example of the configuration of the electronic device 1 according to one embodiment will be described.

[0096] When measuring distance and the like by means of a millimeter wave radar, a frequency modulated continuous wave radar (hereinafter referred to as FMCW radar (Frequency Modulated Continuous Wave radar)) is mostly used. The FMCW radar generates a transmission signal by scanning the frequency of the transmitted radio waves. Therefore, for example, in a millimeter wave FMCW radar using radio waves in the 79 GHz frequency band, the frequency of the radio waves used is, for example, 77 GHz to 81 GHz, and has a bandwidth of 4 GHz. The radar in the 79 GHz frequency band has the following characteristics: the usable bandwidth is wider than other millimeter wave / quasi-millimeter wave radars such as those in the 24 GHz, 60 GHz, and 76 GHz frequency bands. Below, such an embodiment is described as an example.

[0097] The FMCW radar radar method used in the present disclosure may also include an FCM method (Fast-Chirp Modulation) for sending a chirp signal with a period shorter than usual. The signal generated by the electronic device 1 is not limited to the signal of the FMCW method. The signal generated by the electronic device 1 may also be a signal of various methods other than the FMCW method. The transmission signal sequence stored in any storage unit may be different according to these various methods. For example, in the case of the radar signal of the above-mentioned FMCW method, a signal with an increased frequency and a signal with a decreased frequency may be used for each time sample. The above-mentioned various methods can appropriately apply known technologies, so detailed descriptions are omitted.

[0098] like Figure 2 As shown, an electronic device 1 of one embodiment has a signal processing unit 10. The signal processing unit 10 may include a signal generation processing unit 11, a received signal processing unit 12, a heartbeat extraction unit 13, and a calculation unit 14. The heartbeat extraction unit 13 may, for example, perform a process of extracting a micro-Doppler component. In addition, the heartbeat extraction unit 13 may perform a process of extracting an envelope of the heart sound of the subject 200. The calculation unit 14 may, for example, perform a process of calculating the heartbeat interval (RRI) of the subject 200. In addition, the calculation unit 14 may, for example, also perform a process of calculating the heartbeat of the subject 200. Furthermore, the calculation unit 14 may, for example, perform a process of calculating the heart rate variability (Heart rate validity: HRV) of the subject 200. In this case, the calculation unit 14 may also perform a process of performing a frequency analysis of the extracted time series data of the heartbeat interval of the subject 200. In addition, the calculation unit 14 may perform a process of calculating the heart rate variability of the subject 200 based on the frequency analysis of the time series data of the heartbeat interval. The signal generation processing unit 11, the received signal processing unit 12, the heartbeat extraction unit 13 and the calculation unit 14 will be described later as appropriate. In the present disclosure, the heart sound may be, for example, a vibration waveform of the chest directly observed by a radar (see Fig. 20 The heartbeat is the heartbeat itself. The heartbeat interval or heart rate can be calculated based on the movement of the heartbeat. The heartbeat interval can be the time interval between the heartbeat and the next heartbeat.

[0099] In addition, the electronic device 1 of one embodiment has a transmission DAC 21, a transmission circuit 22, a millimeter wave transmission circuit 23, and a transmission antenna array 24 as a transmission unit. In addition, the electronic device 1 of one embodiment has a reception antenna array 31, a mixer 32, a reception circuit 33, and a reception ADC 34 as a reception unit. The electronic device 1 of one embodiment may not include Figure 2 At least one of the functional units shown may also include Figure 2 Functional parts other than those shown. Figure 2 The electronic device 1 shown may use a circuit having a configuration substantially similar to that of a general radar using electromagnetic waves in the millimeter wave band, etc. On the other hand, in the electronic device 1 of one embodiment, the signal processing by the signal processing unit 10 may include processing different from that of conventional general radars.

[0100] The signal processing unit 10 of the electronic device 1 of one embodiment is represented by controlling the various functional units constituting the electronic device 1, and can control the overall operation of the electronic device 1. In particular, the signal processing unit 10 performs various processes on the electronic device 1 to process the signal. In order to provide control and processing capabilities for performing various functions, the signal processing unit 10 may include at least one processor such as a CPU (Central Processing Unit) or a DSP (Digital Signal Processor). The signal processing unit 10 can be implemented by a single processor, or by several processors, or by separate processors. The processor can be implemented as a single integrated circuit. An integrated circuit is also called an IC (Integrated Circuit). The processor can be implemented as a plurality of integrated circuits and discrete circuits that can be connected in communication. The processor can be implemented based on various other known technologies. In one embodiment, the signal processing unit 10 can be configured as, for example, a CPU (hardware) and a program (software) executed by the CPU. The signal processing unit 10 can appropriately include a storage unit (memory) required for the operation of the signal processing unit 10.

[0101] The signal generation processing unit 11 of the signal processing unit 10 generates a signal transmitted from the electronic device 1. In the electronic device 1 of one embodiment, the signal generation processing unit 11 can generate a transmission signal such as a chirp signal (transmission chirp signal). In particular, the signal generation processing unit 11 can also generate a signal whose frequency changes linearly periodically (linear chirp signal). For example, the signal generation processing unit 11 can also generate a chirp signal whose frequency periodically increases linearly from 77 GHz to 81 GHz over time. In addition, for example, the signal generation processing unit 11 can also generate a signal whose frequency periodically repeats linear increase (up chirp) and decrease (down chirp) from 77 GHz to 81 GHz over time. The signal generated by the signal generation processing unit 11 can also be pre-set in the signal processing unit 10, for example. In addition, the signal generated by the signal generation processing unit 11 can also be pre-stored in any storage unit in the signal processing unit 10, for example. Since the chirp signal used in a technical field such as radar is known, a more detailed description is appropriately simplified or omitted. The signal generated by the signal generation processing unit 11 is supplied to the transmission DAC 21. Therefore, the signal generation processing unit 11 can be connected to the transmission DAC 21.

[0102] The transmission DAC (digital-analog conversion circuit) 21 has a function of converting a digital signal supplied from the signal generation processing unit 11 into an analog signal. The transmission DAC 21 can be configured to include a general digital-analog conversion circuit. The signal converted to analog by the transmission DAC 21 is supplied to the transmission circuit 22. Therefore, the transmission DAC 21 can be connected to the transmission circuit 22.

[0103] The transmission circuit 22 has a function of converting the signal analogized by the transmission DAC 21 into an intermediate frequency (IF) band. The transmission circuit 22 can be configured as a transmission circuit including a general IF band. The signal processed by the transmission circuit 22 is supplied to the millimeter wave transmission circuit 23. Therefore, the transmission circuit 22 can be connected to the millimeter wave transmission circuit 23.

[0104] The millimeter wave transmission circuit 23 has a function of transmitting the signal processed by the transmission circuit 22 as a millimeter wave (RF wave). The millimeter wave transmission circuit 23 can be configured to include a general millimeter wave transmission circuit. The signal processed by the millimeter wave transmission circuit 23 is supplied to the transmission antenna array 24. Therefore, the millimeter wave transmission circuit 23 can be connected to the transmission antenna array 24. In addition, the signal processed by the millimeter wave transmission circuit 23 is also supplied to the mixer 32. Therefore, the millimeter wave transmission circuit 23 can also be connected to the mixer 32.

[0105] The transmitting antenna array 24 arranges a plurality of transmitting antennas in an array. Figure 22 shows a simplified structure of the transmission antenna array 24. The transmission antenna array 24 transmits the signal processed by the millimeter wave transmission circuit 23 to the outside of the electronic device 1. The transmission antenna array 24 may be configured to include a transmission antenna array used in a general millimeter wave radar.

[0106] As described above, the electronic device 1 according to one embodiment includes a transmission antenna (transmission antenna array 24 ), and can transmit a transmission signal (eg, a transmission chirp signal) as a transmission wave from the transmission antenna array 24 .

[0107] For example, Figure 2 As shown, it is assumed that an object such as the subject 200 exists around the electronic device 1. In this case, at least a portion of the transmission wave transmitted from the transmission antenna array 24 is reflected by the object such as the subject 200. Among the transmission waves transmitted from the transmission antenna array 24, at least a portion of the transmission waves reflected by the object such as the subject 200 can be reflected to the receiving antenna array 31.

[0108] The receiving antenna array 31 receives the reflected wave. Here, the reflected wave may be at least a part of the transmission wave transmitted from the transmitting antenna array 24 that is reflected by an object such as the subject 200 .

[0109] The receiving antenna array 31 arranges a plurality of receiving antennas in an array. Figure 2 . The receiving antenna array 31 receives the reflected wave after the transmission wave transmitted from the transmitting antenna array 24 is reflected. The receiving antenna array 31 can be configured to include a receiving antenna array used in a general millimeter wave radar. The receiving antenna array 31 supplies the received signal received as the reflected wave to the mixer 32. Therefore, the receiving antenna array 31 can be connected to the mixer 32.

[0110] The mixer 32 converts the signal (transmission signal) processed by the millimeter wave transmission circuit 23 and the reception signal received by the reception antenna array 31 into a frequency band of an intermediate frequency (IF). The mixer 32 can be configured to include a mixer used in a general millimeter wave radar. The mixer 32 supplies the signal generated as a result of the synthesis to the reception circuit 33. Therefore, the mixer 32 can be connected to the reception circuit 33.

[0111] The receiving circuit 33 has a function of performing analog processing on the signal converted into the IF band by the mixer 32. The receiving circuit 33 can be configured to include a receiving circuit that converts into a general IF band. The signal processed by the receiving circuit 33 is supplied to the receiving ADC 34. Therefore, the receiving circuit 33 can be connected to the receiving ADC 34.

[0112] The receiving ADC (analog-digital conversion circuit) 34 has a function of converting the analog signal supplied from the receiving circuit 33 into a digital signal. The receiving ADC 34 can be configured to include a general analog-digital conversion circuit. The signal digitized by the receiving ADC 34 is supplied to the receiving signal processing unit 12 of the signal processing unit 10. Therefore, the receiving ADC 34 can be connected to the signal processing unit 10.

[0113] The reception signal processing unit 12 of the signal processing unit 10 has a function of performing various processing on the digital signal supplied from the reception DAC 34. For example, the reception signal processing unit 12 calculates (measures the distance) from the electronic device 1 to an object such as the subject 200 based on the digital signal supplied from the reception DAC 34. In addition, the reception signal processing unit 12 calculates (measures the speed) the relative speed of the object such as the subject 200 relative to the electronic device 1 based on the digital signal supplied from the reception DAC 34. Furthermore, the reception signal processing unit 12 calculates (measures the angle) the azimuth of the object such as the subject 200 when viewed from the electronic device 1 based on the digital signal supplied from the reception DAC 34. Specifically, the I / Q converted data may also be input to the reception signal processing unit 12. By inputting such data, the reception signal processing unit 12 performs fast Fourier transform (2D-FFT) in the range direction and the velocity direction, respectively. Afterwards, the received signal processing unit 12 may also suppress and make false alarms probabilistic by removing noise points based on, for example, CFAR (Constant False Alarm Rate). Then, the received signal processing unit 12 may obtain the position of an object such as the subject 200 by estimating the arrival angle of a point that satisfies the CFAR benchmark. Information generated as a result of distance measurement, speed measurement, and angle measurement by the received signal processing unit 12 may be supplied to the heartbeat extraction unit 13.

[0114] The heartbeat extraction unit 13 extracts information related to the heartbeat from the information generated by the reception signal processing unit 12. The operation of extracting information related to the heartbeat by the heartbeat extraction unit 13 will be described later. The information related to the heartbeat extracted by the heartbeat extraction unit 13 may be supplied to the calculation unit 14.

[0115] The calculation unit 14 performs various calculation processes and / or operation processes on the information related to the heartbeat supplied from the heartbeat extraction unit 13. The various calculation processes and / or operation processes performed by the calculation unit 14 will be further described later. The various information calculated and / or operated by the calculation unit 14 can be supplied to, for example, the communication interface 50. Therefore, the calculation unit 14 and / or the signal processing unit 10 can be connected to the communication interface 50. The various information calculated and / or operated by the calculation unit 14 can be supplied to other functional units other than the communication interface 50.

[0116] The communication interface 50 may be configured to include an interface for outputting information supplied from the signal processing unit 10 to, for example, an external device 60. The communication interface 50 may output information on at least one of the position, speed, and angle of an object such as the subject 200 to the external device 60 as a signal such as a CAN (Controller Area Network). For example, at least one of the position, speed, and angle of an object such as the subject 200 may be supplied to the external device 60 via the communication interface 50. Therefore, the communication interface 50 may be connected to the external device 60.

[0117] like Figure 2 As shown, an electronic device 1 of one embodiment is connected to an external device 60 via a communication interface 50 by at least one of wired and wireless means. In one embodiment, the external device 60 may be configured to include any computer and / or any control device, etc. In addition, an electronic device 1 of one embodiment may be configured to include the external device 60. The external device 60 may be configured to have various structures according to the manner in which the information of the heartbeat and / or heart sound detected by the electronic device 1 is used. Therefore, a more detailed description of the external device 60 is omitted.

[0118] Figure 3 1 is a diagram for explaining an example of a chirp signal generated by the signal generation processing unit 11 of the signal processing unit 10 .

[0119] Figure 3 This is a structure indicating the time of one frame when the FCM (Fast-Chirp Modulation) method is used. Figure 3 An example of a received signal using the FCM method is shown. FCM is repeated at short intervals (e.g., longer than the round-trip time of the electromagnetic wave between the radar and the target object calculated based on the maximum range). Figure 3 In FCM, in order to facilitate the signal processing of the received signal, most of them are based on Figure 3 The data is divided into subframes as shown and the sending and receiving processes are performed.

[0120] exist Figure 3 In the example, the horizontal axis represents the elapsed time and the vertical axis represents the frequency. Figure 3 In the example shown, the signal generation processing unit 11 generates a linear chirp signal whose frequency changes linearly and periodically. Figure 3 In , each chirp signal is represented as c1, c2, c3, c4, ..., cn. Figure 3 As shown, in each chirp signal, the frequency increases linearly with the passage of time.

[0121] exist Figure 3 In the example shown, several chirp signals such as c1, c2, c3, c4, ..., cn are included as one subframe. That is, Figure 3 Subframe 1 and subframe 2 shown in the figure are respectively composed of several chirp signals such as c1, c2, c3, c4, ..., cn. Figure 3 In the example shown, several subframes including subframe 1, subframe 2, ..., subframe N are regarded as one frame. Figure 3 The frame shown in FIG. 1 is composed of N subframes. Figure 3 One frame is shown as frame 1, followed by frame 2, frame 3, ..., etc. Each of these frames may be configured to include N subframes similarly to frame 1. In addition, a frame interval of a predetermined length may be included between frames. Figure 3 The length of one frame shown can be set to be about 30 mm to 50 mm, for example.

[0122] In the electronic device 1 of one embodiment, the signal generation processing unit 11 can generate a transmission signal of any number of frames. Figure 3 In this way, the relationship between the time and frequency of the transmission signal generated by the signal generation processing unit 11 can be stored in, for example, a storage unit of the signal processing unit 10.

[0123] In this way, the electronic device 1 according to one embodiment can transmit a transmission signal composed of subframes including a plurality of chirp signals. In addition, the electronic device 1 according to one embodiment can transmit a transmission signal composed of a frame including a predetermined number of subframes.

[0124] The following is sent by electronic device 1 Figure 3 The transmission signal of the frame structure shown in the figure is taken as an example. Figure 3 The frame structure shown is only an example, and the chirp signal contained in a subframe may be arbitrary. That is, in one embodiment, the signal generation processing unit 11 may generate a subframe containing an arbitrary number (e.g., an arbitrary plurality of) chirp signals. In addition, Figure 3 The subframe structure shown is also only an example. For example, the subframes contained in a frame can be arbitrary. That is, in one embodiment, the signal generation processing unit 11 can generate a frame containing any number (for example, any plural number) of subframes. The signal generation processing unit 11 can generate signals of different frequencies. The signal generation processing unit 11 can also generate a plurality of discrete signals with different bandwidths and frequencies f.

[0125] Figure 4 It is shown in other ways Figure 3 A diagram of a portion of a subframe is shown. Figure 4 Shows received Figure 3 Each sample of the received signal of the transmission signal shown is processed by the received signal processing unit 12 ( Figure 2 ) is the result of 2D-FFT (Two Dimensional Fast Fourier Transform).

[0126] like Figure 4 As shown in FIG. 1 , each chirp signal c1, c2, c3, c4, ..., cn is stored in each subframe such as subframe 1, ..., subframe N. Figure 4 In FIG. 1 , each chirp signal c1, c2, c3, c4, ..., cn is composed of samples represented by a grid arranged in the horizontal direction. Figure 4 The received signal shown is composed of Figure 2 The reception signal processing unit 12 shown implements 2D-FFT, CFAR and / or integrated signal processing of each subframe.

[0127] Figure 5 It means in Figure 2 The illustrated diagram is a diagram showing an example of calculating a point group on a range-Doppler (distance-velocity) plane as a result of performing 2D-FFT, CFAR, and integrated signal processing of each subframe in the received signal processing unit 12 .

[0128] exist Figure 5 In the figure, the horizontal direction represents the range (distance) and the vertical direction represents the speed. Figure 5 The black grid s1 shown represents a group of points representing a signal that exceeds the threshold processing of CFAR. Figure 5 The unpainted grid s2 shown represents a bin (2D-FFT sample) without a point group that does not exceed the CFAR threshold. Figure 5 The direction from the radar is calculated by the point group on the range-Doppler plane calculated in the above calculation, and the position and velocity on the two-dimensional plane are calculated as a point group representing an object such as the subject 200. Here, the direction estimation can be calculated by a beamformer and / or a partial space method. Representative algorithms of the partial space method include MUSIC (MUltiple SIgnal Classification) and ESPRIT (Estimation of Signal Parameters via Rotation Invariance Technique).

[0129] Figure 6 It means that after the received signal processing unit 12 has estimated the direction, Figure 5The range-Doppler plane shown in FIG. 1 is an example of the result of transforming the point group coordinates from the XY plane. Figure 6 As shown, the reception signal processing unit 12 can draw a point group PG on the XY plane. Here, the point group PG is composed of points P. In addition, each point P has an angle θ and a velocity Vr in the radial direction in polar coordinates.

[0130] The receiving signal processing unit 12 detects an object that exists within the range where the transmission wave T is transmitted based on at least one of the results of 2D-FFT and angle estimation. The receiving signal processing unit 12 can perform, for example, clustering processing based on the respectively estimated distance information, speed information and angle information, thereby performing object detection. As an algorithm used for clustering data, for example, DBSCAN (Density-based spatial clustering of applications with noise) is known. This is an algorithm that clusters according to density. In the clustering process, for example, the average power of the points that constitute the detected object can also be calculated. The distance information, speed information, angle information and power information of the object detected in the receiving signal processing unit 12 can also be supplied to an external device 60, etc., for example, via the communication interface 50.

[0131] As described above, the electronic device 1 may include a transmitting antenna (transmitting antenna array 24), a receiving antenna (receiving antenna array 31), and a signal processing unit 10. The transmitting antenna array 24 transmits a transmission wave T. The receiving antenna array 31 receives a reflected wave R after the transmission wave T is reflected. Then, the signal processing unit 10 detects an object (such as an object such as the subject 200) that reflects the transmission wave T based on the transmission signal transmitted as the transmission wave T and the reception signal received as the reflected wave R.

[0132] Next, the estimation of the direction of an arriving wave by the antenna array of the electronic device 1 according to one embodiment will be further described.

[0133] Figure 7 The diagram is a diagram for explaining the configuration of the reception antenna array 31 of the electronic device 1 according to one embodiment and the principle of estimating the direction of an arriving wave by the reception antenna array 31 . Figure 7 An example of reception of radio waves by the receiving antenna array 31 is shown.

[0134] like Figure 7 As shown, the receiving antenna array 31 can arrange sensors such as receiving antennas in a straight line. Figure 7 As shown, in one embodiment, the receiving antenna array 31 may be configured to include a plurality of receiving antennas arranged in a straight line. Figure 7In the figure, the receiving antenna array 31 uses small circles to represent antennas x1, x2, x3, ..., x M Such multiple antennas. The receiving antenna array 31 can be composed of any multiple antennas. In addition, Figure 7 As shown in FIG. 1 , the plurality of antennas constituting the receiving antenna array 31 are arranged at intervals of an array spacing d. In this way, a sensor array in which sensors (antennas, ultrasonic transducers, microphones, etc.) corresponding to various physical waves are arranged in an array is also called a uniform linear array (ULA). Figure 7 As shown, physical waves (electromagnetic waves and sound waves, etc.) arrive from various directions such as θ1 and θ2. Here, θ1 and θ2 can be the above-mentioned arrival angles. In this way, a sensor array such as the receiving antenna array 31 can estimate the arrival direction (arrival angle) by using the phase difference generated in the measured values ​​between sensors according to the arrival direction of the physical waves. In this way, the method of estimating the arrival direction of the waves is recorded as arrival angle estimation or arrival direction estimation (Direction of Arrival: DoA).

[0135] In the electronic device 1 of one embodiment, at least one of the transmitting antenna array 24 and the receiving antenna array 31 can configure the plurality of antennas in a straight line. As a result, for example, it is possible to appropriately narrow the directivity when transmitting and receiving radio waves in a millimeter wave radar. When transmitting a transmission wave, the direction of the transmission beam is mostly controlled by a beamformer. On the other hand, when receiving a reflected wave, the arrival direction of the reflected wave is mostly estimated by a partial space method (such as the above-mentioned MUSIC and ESPRIT) compared to a beamformer. In the beamformer and the partial space method, in Figure 7 In the ULA shown, for electromagnetic waves arriving from various directions, a phase difference occurs in the measured values ​​between sensors according to the arrival direction. Therefore, the arrival direction of the reflected wave can be estimated using this phase difference.

[0136] Next, the estimation of the angles of the arriving waves in two directions by the antenna array of the electronic device 1 according to one embodiment will be further described.

[0137] Figure 8 This is a diagram showing an example of antenna arrangement for estimating directions of arrival at two orthogonal angles.

[0138] like Figure 8 As shown, in the electronic device 1 according to one embodiment, the transmitting antenna array 24 and / or the receiving antenna array 31 may be configured as an array including a plurality of patch antenna units.

[0139] exist Figure 8In the illustrated transmitting antenna array 24, one patch antenna unit may include a plurality of elements electrically connected in the direction 1 shown in the figure. In each patch antenna unit, the plurality of elements may be electrically connected, for example, by wiring such as strip lines on a substrate. In each patch antenna unit, the plurality of elements may be spaced apart at intervals d shorter than half the wavelength λ of the transmitting wave. 1,t Separately configured. Figure 8 In the example, each patch antenna unit can electrically connect any number of two or more elements. Figure 8 Part of the electrically connected components is omitted in the figure.

[0140] In addition, if Figure 8 As shown, the transmitting antenna array 24 may be configured such that a plurality of patch antenna elements are arranged in an array in the direction 2 shown in the figure. Each patch antenna element may be arranged at a spacing d shorter than half the wavelength λ of the transmitting wave. 2,t In one embodiment, the transmit antenna array 24 may include any number of patch antenna elements greater than two.

[0141] like Figure 8 As shown, in one embodiment, the receiving antenna array 31 can change the configuration of a plurality of elements in the transmitting antenna array 24. Figure 8 In the receiving antenna array 31 shown, one patch antenna unit may be configured to include a plurality of elements electrically connected in the direction 2 shown in the figure. In each patch antenna unit, the plurality of elements may be electrically connected, for example, by wiring such as a strip line on a substrate. In each patch antenna unit, the plurality of elements may be spaced apart at intervals d shorter than half the wavelength λ of the transmission wave. 2,s Separately configured. Figure 8 In the example, each patch antenna unit can electrically connect any number of two or more elements. Figure 8 Part of the electrically connected components is omitted in the figure.

[0142] In addition, if Figure 8 As shown, the receiving antenna array 31 may be configured such that a plurality of patch antenna elements are arranged in an array in the direction 1 shown in the figure. Each patch antenna element may be arranged at a spacing d shorter than half the wavelength λ of the transmission wave. 1,s In one embodiment, the receiving antenna array 31 may include any number of patch antenna units greater than two.

[0143] The elements included in the transmitting antenna array 24 and the receiving antenna array 31 can all be arranged on the same plane (for example, on the surface layer of the same substrate). In addition, the transmitting antenna array 24 and the receiving antenna array 31 can be arranged close to each other (single station type). Figure 8The shown directions 1 and 2 may be geometrically orthogonal.

[0144] use Figure 8 The transmitting antenna array 24 and receiving antenna array 31 shown can appropriately narrow the directivity of the transmitting antenna and the receiving antenna. Figure 8 The transmitting antenna array 24 shown in the figure controls the direction of each transmitting wave at each time of transmitting the transmitting wave (transmitting signal), so as to achieve the purpose of Figure 8 The beamformer in the direction 2 shown. Figure 8 The receiving antenna array 31 shown can Figure 8 The arrival direction of the reflected wave is estimated in the direction 1 shown. In this way, the arrival direction of the reflected wave can be estimated for two substantially orthogonal angles. Therefore, a point group representing an object such as the subject 200 can be acquired three-dimensionally.

[0145] Next, a method of detecting the heartbeat of the subject 200 using the electronic device 1 according to one embodiment will be described.

[0146] An electronic device 1 according to one embodiment transmits a transmission wave such as a millimeter wave radar to a subject 200, and measures (estimates) the heartbeat of the subject 200 based on the result of receiving a reflected wave reflected from the chest where the heart of the subject 200 is located. As described above, the subject 200 may be a human or an animal. In this case, for example, by frequency filtering the vibration of the position of the subject 200 detected by the radar, it is possible to extract a component of an envelope assumed to be a heartbeat. Then, after extracting the component of the envelope assumed to be a heartbeat, the approximate heartbeat interval can be calculated by setting the interval between the peak values ​​of the envelope as the heartbeat interval. Here, the peak value of the heartbeat envelope can be approximated to be roughly consistent with the R peak value of the electrocardiogram. Therefore, the "heartbeat interval" is the same as the term used in electrocardiograms, etc., and is also recorded as RR interval or RRI (RRinterval).

[0147] Here, a method for estimating the heartbeat interval of the subject 200 based on the results of the above-mentioned 2D-FFT, CFAR processing, and arrival direction estimation is studied. Figure 4 and Figure 5 The following describes how the human heartbeat and body movement are expressed using the results of 2D-FFT performed on the image.

[0148] Fig. 9 2D-FFT is a diagram showing an example of the result of receiving the reflected wave of the transmission wave transmitted to the subject 200 and performing 2D-FFT processing. As a result of 2D-FFT, Fig. 9 A frequency spectrum representing the heart sounds and body movements of the subject 200 is shown. Fig. 9 In the figure, the horizontal axis represents the distance (Range) and the vertical axis represents the speed (Velocity). The signal processing unit 10 (e.g., the heartbeat extraction unit 13) of the electronic device 1 of one embodiment can extract, for example, Fig. 9 The peak value Hm shown is a body movement such as the heartbeat of the subject 200. Fig. 9 The frequency spectrum components shown by the peak value Hm include not only the heart sound of the subject 200 and the envelope of the heart sound, but also body motion. In order to extract the heartbeat interval, it is necessary to extract body motion, etc., so it is assumed that frequency filtering is performed. The frequency filtering performed here, for example, assumes a bandpass filter, a high-pass filter, and / or a low-pass filter with a target of 0.5 Hz or more and 10 Hz or less.

[0149] Fig.10 This is a diagram for explaining a method of detecting a peak based on the envelope waveform of the heart sound obtained by the above-mentioned frequency filtering. Fig.10 The graph shown in FIG. 1 shows an example of the time variation of the envelope waveform of the heart sound extracted by the above-mentioned frequency filtering. Fig.10 The envelope waveform shown contains multiple peaks as shown in the figure. On the other hand, it is known that the beats of the subject 200 converge within a range of about 50 to 130 in one minute. Fig.10 By selecting a peak value of an interval of 0.4 seconds to 0.8 seconds, which is the inverse of the number of beats per minute, from the plurality of peak values ​​shown, it is possible to calculate the approximate heart sound interval of the subject 200. For example, one may select Fig.10 The peaks indicated by downward arrows in FIG. 2 are the approximate heart sound intervals of the subject 200 .

[0150] Fig.11 1 is a flowchart showing an example of the above-mentioned heartbeat interval estimation operation. Fig.11 The above-mentioned heartbeat interval estimation operation will be described probabilistically.

[0151] Fig.11 FIG. 2 shows the operation of the electronic device 1 after receiving the reflected wave according to one embodiment. Fig.11 The premise of the action shown is Figure 2 The electronic device 1 shown transmits a transmission wave (transmission signal) from the transmission antenna array 24. At least a portion of the transmission wave transmitted from the electronic device 1 is reflected by the subject 200 (eg, the chest) to become a reflected wave. Figure 2 The electronic device 1 shown receives such reflected waves from the receiving antenna array 31. Then, the Fig.11 Action shown.

[0152] When to start Fig.11In the operation shown, first, in step S110, the signal processing unit 10 of the electronic device 1 processes the received signal (received signal). The signal processing performed in step S110 may include, for example, the above-mentioned 2D-FFT, CFAR processing and / or arrival direction estimation. Such an operation may be performed, for example, by the received signal processing unit 12 of the signal processing unit 10.

[0153] Next, in step S120, the signal processing unit 10 extracts the vibration source from the information processed in step S110. The signal processing performed in step S120 may include, for example, filtering the data resulting from the 2D-FFT process. In step S120, the signal processing unit 10 may extract only the frequency spectrum component at the position where the subject 200 is located. Here, the position where the subject 200 is located may be determined by various known methods. Such an operation may be performed, for example, by the heartbeat extraction unit 13 of the signal processing unit 10.

[0154] Next, in step S130, the signal processing unit 10 converts the processing result of the previous stage into a vibration waveform. The processing performed in step S130 may include, for example, a process of extracting phase information from IQ data. In step S130, the signal processing unit 10 may include a process of extracting vibration data including heart sounds from the spectrum components of the 2D-FFT processing of the subject 200 extracted in step S120. Such an operation may be performed, for example, by the heartbeat extraction unit 13 of the signal processing unit 10.

[0155] Next, in step S140, the signal processing unit 10 extracts vibration data from the processing result of the previous stage. The processing performed in step S140 may include frequency filtering, for example. In step S140, the signal processing unit 10 extracts a low-frequency signal including an envelope of the heart sound of the subject 200 by performing frequency filtering. Such an operation may be performed, for example, by the heartbeat extraction unit 13 of the signal processing unit 10.

[0156] Next, in step S150, the signal processing unit 10 detects the peak value of the beat of the subject 200 from the processing result of the previous stage to calculate the interval of the peak value, thereby calculating the RR interval (RRI) of the subject 200. In step S150, the signal processing unit 10 can detect the peak value of the low-frequency signal of the envelope of the heart sound of the subject 200. In addition, in step S150, the signal processing unit 10 can calculate and / or extract the interval (interval) of each peak value. Such an action can be performed, for example, by the calculation unit 14 of the signal processing unit 10. As described above, in step S150, the signal processing unit 10 can extract the interval of the heart sound.

[0157] Next, in step S160, the signal processing unit 10 can calculate the heart rate variability (HRV) of the subject 200. In step S160, the signal processing unit 10 can calculate the HRV of the subject 200 by calculating the spectral density of the time series data of RRI. When calculating the power spectral density of the time series data of RRI, the frequency analysis of the time series waveform of RRI can be performed, for example, by using the Welch method. Such an action can be performed, for example, by the calculation unit 14 of the signal processing unit 10. In addition, in step S160, the signal processing unit 10 can also analyze the spectrum from the processing result of the previous stage.

[0158] As described above, the electronic device 1 of one embodiment performs frequency filtering on the vibration at the position of the subject 200 to extract the component of the envelope that is considered to be the heartbeat, and sets the peak interval of the envelope as the heartbeat interval. In this way, the electronic device 1 of one embodiment can calculate the approximate heartbeat interval of the subject 200.

[0159] In the calculation of the heartbeat interval as described above, for example, Fig.10 As shown, there are several peaks in a time period of tens of ms. Therefore, a certain degree of uncertainty is involved in selecting the peak of the heartbeat. Therefore, the accuracy of the calculated heartbeat interval also has an error of about tens of ms. For example, in the above-mentioned calculation of the heartbeat interval, compared with the instantaneous RRI obtained by the electrocardiograph, an error of at least about 20 ms will occur. At this accuracy, it is difficult to calculate the heart rate variability (HRV) to perform autonomic nervous and / or emotional analysis of a person. That is, in the above-mentioned calculation of the heartbeat interval, it is difficult to perform more advanced medical analysis by extracting heart sounds.

[0160] Therefore, for example, it is also possible to consider using frequency filtering to further cut off the peak on the high-frequency side so that the heartbeat time interval is about 0.4 seconds to 0.8 seconds. Fig.10 The accuracy of the information of the signal shown is not exceeded, so it is considered difficult to reduce the error of several 10 ms.

[0161] In addition, in the calculation of the heartbeat intervals described above, heart sounds are not extracted. Therefore, it is considered difficult to obtain information useful for diagnosis (eg, auscultation in medical treatment) based on the acoustic properties of heart sounds themselves in the above method.

[0162] Therefore, the electronic device 1 of one embodiment further improves the above method. Thus, the electronic device 1 of one embodiment extracts heart sounds by analyzing the heart sounds themselves, for example, using a radar using a high frequency band of millimeter waves or higher, to extract accurate heartbeat intervals. Such a method is described below.

[0163] In order to extract the heart sounds of the subject 200 with high precision, the electronic device 1 of one embodiment performs appropriate signal processing on the signal (chirp signal) received by the electronic device 1 in an appropriate order, thereby reducing the partial space and partial space basis where the signal components of the heart sounds of the subject 200 exist. In the electronic device 1 of one embodiment, different basis vectors are used in the linear space where the signal of the heart sounds of the subject 200 exists, an appropriate coordinate system is used to describe the space, and the partial space is extracted based on each coordinate. Through such processing, the electronic device 1 of one embodiment is able to search for the partial space where the heart sounds of the subject 200 exist. In one embodiment, such a coordinate system can use an appropriate coordinate system according to the purpose. For example, a coordinate system of a space in which 2D-FFT is performed, a coordinate system of a time series signal obtained by time reduction of a chirp signal, a coordinate system based on the Fourier transform, or a coordinate system based on a continuous / discrete wavelet, etc. can be used.

[0164] An electronic device 1 according to one embodiment can perform the following characteristic processing on a received signal in a phased manner. The following is an overview of the characteristic processing based on the electronic device 1 according to one embodiment. The electronic device 1 according to one embodiment performs three characteristic processing. That is, the electronic device 1 according to one embodiment performs (1) extraction of partial space, (2) generation of envelope and selection of optimal heart sound, and (3) statistical signal processing for heartbeat interval calculation. The following describes each processing in more detail.

[0165] (1) Extraction of partial space

[0166] First, the electronic device 1 of one embodiment performs processing of omitting data considered unnecessary in the linear space on the received signal. The electronic device 1 of one embodiment performs processing of retaining the necessary partial space by omitting the unnecessary partial space. Specifically, the following processing can be performed.

[0167] (1-1) Stage 1: Dimension Reduction

[0168] In the first stage, a suitable window function is applied to the 2D-FFT processing of the chirp signal received by the electronic device 1, and the arrival direction estimation is used to extract only the micro-Doppler component of the point group where the subject 200 (such as a person or an animal) exists. Here, a plurality of window functions centered on a plurality of distance ranges are generated around the approximate position of the target object, and the plurality of window functions are applied. Hereinafter, the processing of applying a plurality of window functions in this way is referred to as "multi-window processing". Here, a time series signal of a vibration can be generated for one window function, that is, one center distance range.

[0169] (1-2) The second stage: Reduction of the dimensions of some spaces

[0170] In the second stage, principal component analysis and / or singular value decomposition or the like is performed on the micro-Doppler signal on the set of time-series waveforms of the plurality of vibrations extracted in the processing of the first stage.

[0171] (1-3) The third stage: Reduction of the dimensions of some spaces

[0172] In the third stage, frequency filtering is performed by using at least any one of short-time Fourier transform, continuous wavelet transform, and a bandpass filter.

[0173] (1-4) The fourth stage: Reduction of the dimensions of some spaces

[0174] In the fourth stage, a discrete wavelet transform-based multiresolution analysis is performed to extract heart sounds by using a wavelet function and a scaling function suitable for heart sounds.

[0175] (2) Envelope generation and selection of optimal heart sounds

[0176] Next, in order to facilitate the processing of the extracted heart sound data, the electronic device 1 of one embodiment can perform a process of generating an envelope based on the waveform of the extracted heart sound. Furthermore, the electronic device 1 of one embodiment can perform a process of selecting (extracting) the best from a plurality of heart sound data. Specifically, the following process can be performed.

[0177] (2-1) Generation of envelope

[0178] Here, an envelope is generated based on the heart sounds extracted in the extraction of the partial space (1). In the process of generating the envelope, for example, any one of continuous wavelet transform, discrete wavelet transform, wavelet scattering coefficient, Mel-frequency cepstral coefficient, and moving variance is used.

[0179] (2-2) Selection of the best heart sound

[0180] Next, the best heart sound is selected from the generated envelope. The number of heart sounds generated by the extraction of the above-mentioned (1) partial space is equal to the number of window functions. Therefore, here, the best heart sound is extracted (selected) from the heart sounds equal to the number of window functions. First, a previously trained (learned) learning classifier can be used to calculate the score of the heart sound. In this case, the learning classifier can be, for example, an autoencoder, LSTM (Long Short Term Memory), other neural networks, support vector machines, or decision trees. Then, based on the calculated score, the best heart sound is determined. At the same time, the accompanying calculation results can be obtained from the learning classifier. In addition, as a learning classifier, any one of a linear support vector machine, a simple perceptron, a classifier using logistic regression, a decision tree (classification tree), a k-nearest neighbor algorithm, a random forest, a nonlinear support vector machine, a neural network, or a combination thereof can be used.

[0181] (3) Statistical signal processing for heartbeat interval calculation

[0182] Here, the heart sound interval is calculated by statistically processing the envelope of the generated optimal heart sound. In most cases, including healthy people, the heart sound includes the first heart sound (S1) and the second heart sound (S2). If the time interval between the peaks of the heart sound is detected within a certain period of time, the following two intervals exist.

[0183] (i) The interval between the first and second heart sounds (hereinafter also referred to as the interval between S1 and S2)

[0184] (ii) Second heart sound - first heart sound interval (hereinafter also referred to as "S2-S1 interval")

[0185] The distribution of the histogram or probability density of these intervals is a mixed Gaussian distribution or a single Gaussian distribution. Therefore, based on information such as the distribution, accurate heart sound intervals can be calculated by using, for example, simple DP (Dynamic Programming) matching (dynamic programming method).

[0186] According to the electronic device 1 of one embodiment, by obtaining the envelope waveform of the heart sound obtained through the above-mentioned stages of processing, the positive and negative peaks are detected, and by appropriately calculating the interval, the RR interval (RRI / RR interval) can be calculated with high accuracy.

[0187] Next, the operation of the electronic device 1 according to one embodiment will be described in more detail.

[0188] Fig.12 This is a flowchart showing an example of the operation performed by the electronic device 1 according to one embodiment. Fig.13 is a more detailed representation Fig.12 Flowchart of an example of the operation of step S16 in FIG. Fig.12 and Fig.13 The flow of operations performed by the electronic device 1 according to one embodiment will be described.

[0189] Fig.12 The step S11 shown can be Fig.11 The operation in step S110 shown in FIG. 1 is performed in the same manner. Fig.12 When the operation shown starts, first, in step S11, the signal processing unit 10 of the electronic device 1 processes the received signal (received signal). The signal processing performed in step S11 may include, for example, the above-mentioned 2D-FFT, CFAR processing and / or arrival direction estimation. Such an operation may be performed, for example, by the received signal processing unit 12 of the signal processing unit 10.

[0190] In step S12, the signal processing unit 10 performs multi-window processing. In step S12, the signal processing unit 10 may perform processing for determining candidate clusters of humans or animals and / or filtering processing (multi-window processing) on ​​data of 2D-FFT processing based on a plurality of window functions.

[0191] In step S12, as a previous stage of selecting the best candidate for the heart sound, the signal processing unit 10 may perform a multi-window process for generating a plurality of candidates for the heart sound. Here, the signal processing unit 10 may perform a process of clustering point groups corresponding to humans or animals on the data subjected to the 2D-FFT process. In addition, the signal processing unit 10 may perform the multi-window process by applying a plurality of window functions to the selected clusters. The multi-window process performed here may include a process of generating the data subjected to the 2D-FFT process that is the source of the plurality of candidates for the heart sound.

[0192] In step S12, the signal processing unit 10 can extract only the area where the subject 200 exists on the range-Doppler plane calculated by 2D-FFT using an appropriate window function, thereby generating a plurality of candidate heart sounds. In step S12, the signal processing unit 10 can set a plurality of window functions as appropriate window functions. Here, the signal processing unit 10 can use window functions such as a Hanning window, a Hamming window, and a Blackman-Harris window, for example.

[0193] In step S12 , the signal processing unit 10 may detect a vibration source (object marker) including body motion including heart sounds and / or breathing of the subject 200 based on, for example, the following first to third steps.

[0194] (First step)

[0195] The signal processing unit 10 estimates the direction of arrival based on the Figure 5 and Figure 6 ) results, the point groups exceeding the CFAR threshold on the range-Doppler plane are classified into the area of ​​the pre-set angle. Figure 6 If the angle on the xy plane shown is θ, the area can be classified into area A to area C with the angle from θ to θ.

[0196] Area A: -10deg.<θ<10deg.

[0197] Area B: -20 degrees <θ ≤ -10 degrees.

[0198] Area C: 10 degrees ≤ θ < 20 degrees.

[0199] (Second step)

[0200] The signal processing unit 10 processes the point group exceeding the CFAR threshold in the following manner: Figure 5 Clustering is applied on the range-Doppler plane shown in S1 of FIG. 1 , and these point groups belong to the group of regions of a certain angle classified in the first step. Here, as clustering, for example, a method such as DBSCAN can be applied.

[0201] (Third step)

[0202] The signal processing unit 10 assumes that L clusters are processed in the second step, and calculates the deviation D of the number of bins in the Doppler direction. dev [l] with a certain threshold D of the first cluster dev,th As a result of this comparison, the signal processing unit 10 only becomes D dev [l]≥D dev,th The subject is determined to be the subject 200, and the flag HF[1] is set. That is, the signal processing unit 10 can perform the processing shown in the following [pseudo code 1], for example.

[0203] [Pseudo code 1]

[0204] for l=1to L do

[0205] if D dev [l]≥D dev,th

[0206] HF[l]=1

[0207] else

[0208] HF[l]=0

[0209] end if

[0210] end do

[0211] In step S13, the signal processing unit 10 restores the set of chirp signals. In step S13, the signal processing unit 10 may restore the set of chirp signals by performing an inverse discrete Fourier transform (2D-IFFT (inverse fast Fourier transform)) on the plurality of generated 2D-FFT data.

[0212] Fig.14 It is a diagram for explaining a multi-window process of the electronic device 1 according to one embodiment. Fig.14 This is a diagram for explaining multi-window processing, in which a plurality of time series signals are generated from a point group belonging to the cluster selected by the above pseudo code 1 by using a window function that moves in a plurality of range directions.

[0213] exist Fig.14 In the upper area, the horizontal axis represents the distance (Range), and the vertical axis represents the Doppler velocity (Doppler Velocity). Fig.14 The upper area shows the result of clustering the point group of 2D-FFT data at a certain moment. Fig.14 The upper area of ​​represents the result of clustering, where three clusters, Cluster 1, Cluster 2, and Cluster 3, are generated. Fig.14 The middle area of ​​schematically shows a situation where the signal processing unit 10 performs multi-window processing on each of cluster 1 , cluster 2 , and cluster 3 . Fig.14 The middle area of ​​shows the situation where the signal processing unit 10 performs multi-window processing by shifting the window function little by little in the distance (Range) direction and applying it to each cluster. In addition, Fig.14 The lower area of ​​shows the results of calculating the vibration speeds of each of cluster 1 , cluster 2 , and cluster 3 by performing 2D-IFFT on each of clusters 1 , 2 , and 3 by the signal processing unit 10 . Fig.14 The lower part of the figure shows the situation where a vibration time series signal is generated by performing 2D-IFFT on each cluster. In fact, by shifting the window function little by little in the range direction and Fig.14 Each cluster shown is applied multiple times to obtain multiple vibration time series signals.

[0214] exist Fig.11 In step S13, the signal processing unit 10 applies a plurality of window functions to the cluster consisting of the point group selected as the flag HF[l]=1 by the above [pseudo code 1]. Through this processing, 2D-FFT data corresponding to each window function is generated. Here, the 2D-FFT data generated by the first window function is expressed as the following mathematical formula (1).

[0215] [Mathematical formula 1]

[0216]

[0217] In this case, in step S13, the signal processing unit 10 can calculate the time series signal waveform of the vibration velocity according to the following mathematical formula (2): The waveform is restored to a set of chirp signals by two-dimensional inverse fast Fourier transform (2D-IFFT).

[0218] [Mathematical formula 2]

[0219]

[0220] Here, F -1 represents inverse discrete Fourier transform, and M represents the number of chirps subjected to 2D-FFT processing.

[0221] Below, as an example, only one cluster, such as Fig.14 Cluster 1 is shown for illustration.

[0222] In step S14, the signal processing unit 10 performs a principal component analysis of the signal and / or removes noise. Step S14 may be a singular value decomposition (SVD) for preprocessing and eliminating noise (noise) of the signal. This processing may be performed for the purpose of removing low energy noise and / or low level noise mixed by uncertainty of Fourier transform.

[0223] In step S14, the signal processing unit 10 may process the set S of received chirp signals extracted in the 2D-FFT plane. l Perform singular value decomposition. If the matrix of the left singular vectors is recorded as U, the matrix of the diagonal singular values ​​is recorded as Σ, and the matrix of the right singular vectors is recorded as V, then the set S of received chirp signals is used. l The singular value decomposition of can be obtained as follows:

[0224] [Mathematical formula 3]

[0225]

[0226] In the above mathematical formula (3), * indicates Hermite transposition for V. In addition, N indicates the number of samples in one chirp signal, and M indicates the number of snapshots of the chirp.

[0227] Next, the signal processing unit 10 may limit (U ext ) The number of row vectors of the left singular vectors of the above mathematical formula (3). In addition, the signal processing unit 10 can limit (Σ ext ) is the number of diagonal elements of the diagonal matrix of the singular values ​​of the above mathematical formula (3). In this way, unnecessary noise signals and vibration components outside the desired position are removed. As a result, the projection to Sl The signal S in the partial space of the target signal l ext As shown in the following mathematical formula (4).

[0228] [Formula 4]

[0229]

[0230] Fig.15 This is a diagram showing an example of the relationship between the ranks of the target signal and the noise signal in singular value decomposition. Fig.15 The SVD results are arranged in descending order of singular values ​​(equivalent to energy values), with the horizontal axis set to rank and the vertical axis set to singular values. In the present disclosure, such an idea is adopted that when a signal above a certain singular value is taken out, it becomes the target signal. In this case, the rank of the singular value that serves as the dividing line between whether it is the target signal or not is in Fig.15 The dark grey portion of the graph represents the rank region below 30. Fig.15 As shown, the portion of the rank region below 30 becomes the region of the target signal. Fig.15 In the example, the rank portion of the target signal is greater than or equal to 1 and less than or equal to 30. Therefore, in the present disclosure, for example, singular vectors corresponding to left and right singular values ​​of a rank region greater than or equal to 1 and less than or equal to 30 can be extracted.

[0231] In the above example, the rank portion of the target signal is greater than 1 and less than 30. The maximum rank value can be basically determined based on experience. In addition, the maximum rank value (rank 30 in the present disclosure) can also be determined by statistical methods. Fig.15 In the graph of , the singular values ​​decrease sharply in rank 130, and the signals of the subsequent ranks basically become noise. Therefore, the signals of the subsequent ranks may not be needed.

[0232] exist Fig.15 In the graph of , the left and right singular vectors (vectors in the space of the extended signal) corresponding to the singular values ​​in the range of rank 31 to 130 also have some target signal components. The left and right singular vectors are assumed to be mainly caused by unnecessary fine vibrations and / or artificial noise (artifacts) generated by radar signal processing. Therefore, such elements can also be discarded.

[0233] In step S14, when noise is removed by SVD, the signal processing unit 10 converts the result of the noise removal into a signal waveform (step S15). In step S15, the signal processing unit 10 may convert the beat signal (IQ data) into a time series signal of vibration.

[0234] In step S15, the signal processing unit 10 calculates the sum of the column vectors of the chirp signals based on the above equation (4) to obtain a vector represented by the following equation (5).

[0235] [Mathematical formula 5]

[0236]

[0237] In step S15, finally, by obtaining s l ext,sum The deflection angle and differential on the Gaussian plane are used to calculate the vector representing the vibration velocity shown in the following mathematical formula (6).

[0238] [Mathematical formula 6]

[0239]

[0240] Next, in step S16, the signal processing unit 10 extracts heart sounds and / or analyzes heart beat intervals from the obtained plurality of vibration waveforms. Specifically, in step S16, the signal processing unit 10 extracts heart sounds from the vibration waveform shown in the above mathematical formula (6). In step S16, the signal processing unit 10 can perform processing for generating heart sound waveforms, calculating heart beat intervals (RRI) and / or calculating heart rate variability (HRV). Thus, the signal processing unit 10 can calculate heart beat intervals. This processing can be performed on waveforms of a plurality of vibration velocities corresponding to the window functions, respectively.

[0241] As mentioned above, in more detail, Fig.12 The processing of step S16 shown may include Fig.13 At least a portion of the processing of each step from step S21 to step S27 shown in FIG. Fig.13 Each step from step S21 to step S27 shown will be described in more detail.

[0242] First, in Fig.13 In step S21 shown, the signal processing unit 10 performs a process (denoising) of removing noise from the time series waveform. Here, in the process of removing noise, a wavelet method such as the Empirical Bayes method or the Continuous Wavelet Transform (CWT) can be used.

[0243] More specifically, in step S21, the signal processing unit 10 performs a process on the vibration velocity vector v corresponding to the first window function. l vib Preprocessing to further remove unwanted noise.

[0244] In step S21, the signal processing unit 10 may perform noise removal processing by frequency band limiting based on the empirical Bayes method and / or continuous wavelet. In addition, in step S21, the signal processing unit 10 may also perform noise removal processing by frequency subtraction, which uses the noise profile to eliminate the noise caused by Fig.12 Artifact noise etc. introduced by the nonlinear processing in steps S11 to S14 shown.

[0245] Next, in step S22, the signal processing unit 10 extracts the waveform of the target signal, that is, the waveform of the heart sound. In this case, for example, a method based on discrete wavelets such as Maximum overlap discrete Wavelet transform (MODWT) can be used.

[0246] In more detail, in step S22, the signal processing unit 10 may use a wavelet waveform having a waveform similar to a cardiac sound waveform as preprocessing for the denoised signal, and perform multi-resolution analysis based on discrete wavelet transform. Thus, in step S22, the signal processing unit 10 extracts only a portion of the multi-resolution analysis level where the cardiac sound of the subject 200 exists based on experience. Thus, in step S22, the signal processing unit 10 may extract the cardiac sound of the subject 200. Specifically, in order to improve the temporal resolution, the signal processing unit 10 may use maximum repeat multi-resolution analysis (MODWT) or the like. In addition, the wavelet basis suitable for the extraction of the heartbeat may be, for example, Symlet and Daubechies. In addition, these instructions may be set appropriately each time.

[0247] The signal velocity vector v can be l vib The signal subjected to the denoising process in step S21 is assumed to be v l vib,dn , and then the real part Re(v l vib,dn )Implementation Fig.16 Multiresolution analysis shown. Fig.16 : is a diagram schematically showing multi-resolution analysis based on discrete wavelet transform. Then, the signal processing unit 10 performs reconstruction by limiting the empirically appropriate level to j0∈N of the following mathematical formula (7), thereby obtaining the cardiac sound waveform h.

[0248] [Mathematical formula 7]

[0249]

[0250] Here, Φ j0,k , j,k The vectors representing the scale function and wavelet function, ck d j,k represent their respective coefficients.

[0251] In this way, the electronic device 1 of one embodiment can obtain, for example, Fig.26 and Fig. 27 The result can be directly applied to the diagnosis and treatment of heart sounds, etc.

[0252] Next, in step S23, the signal processing unit 10 generates an envelope waveform. In step S23, in order to detect (extract) the peak value of the energy of the target signal (heart sound waveform) extracted in step S22, the signal processing unit 10 performs scalar mapping based on continuous wavelet transform and envelope waveform extraction based on the one-dimensionalization. Here, discrete / continuous wavelet transform, moving variance and / or Hilbert transform, etc. may also be used.

[0253] More specifically, in step S23, in order to obtain the cardiac sound waveform h shown in the above mathematical formula (7), l The signal processing unit 10 can obtain a scalar graph based on continuous wavelet transform. The signal processing unit 10 takes the sum of a certain range f1 to f2 of the frequency axis at each time of the scalar graph. Fig.17 and Fig.18 As shown, the signal processing unit 10 obtains a one-dimensional waveform s l h . Fig.17 and Fig.18 : is a diagram showing an example of the analysis result of continuous wavelet transform. This analysis can be replaced by discrete wavelet transform with due attention to resolution.

[0254] Fig.17 is the waveform of the heart sound h l Graph of the scalar graph of . Fig.17 It is a graph showing the time change of the normalized vibration frequency. Fig.17 The horizontal axis represents the time in the number of samples. Fig.17 The vertical axis represents the normalized vibration frequency at each time.

[0255] in addition, Fig.18 It is represented by the continuous wavelet transform Fig.17 The heart sound waveform h l A scalar graph is a graph showing the result of one-dimensionalization. Fig.18 It is a graph showing the time change of the vibration speed. Fig.18 The horizontal axis represents the time in the number of samples. Fig.18 The vertical axis represents the vibration speed.

[0256] Here, the matrix representing the absolute value of the scalar map satisfies the following mathematical formula (8).

[0257] [Mathematical formula 8]

[0258]

[0259] In this case, the one-dimensional envelope waveform e l h like Fig.18 The signal processing unit 10 can calculate the envelope waveform e by the following [pseudo code 2] l h .

[0260] [Pseudo code 2]

[0261] for m=1:M

[0262] [Mathematical formula 9]

[0263]

[0264] end

[0265] Here, e l h The vector represents the one-dimensional envelope waveform, and f represents the frequencies of the scalar graph.

[0266] Next, in step S24, the signal processing unit 10 selects the best heart sound. In step S24, in order to extract the best heart sound envelope from the envelope waveforms of the plurality of heart sounds extracted in step S23, the signal processing unit 10 may perform recognition processing based on machine learning. Here, machine learning such as an autoencoder, LSTM (Long Short Term Memory), or other neural networks or support vector machines (SVM) may be used.

[0267] More specifically, in step S24, the signal processing unit 10 calculates the vector e obtained as the envelope waveform of the l heart sounds. l h Select the best vector. Here, in order to select the best e l h The signal processing unit 10 can select the minimum cost function by scoring each cost function. Here, the cost function is calculated using the error of reconstruction by an automatic encoder (self-encoder).

[0268] Fig.19 , Fig. 20 (A) and Fig. 20(B) is a diagram schematically illustrating a process of classifying and identifying the envelope of the best heart sound. Fig.19 This is a diagram schematically showing a situation in which an autoencoder is made to learn a plurality of learning data. Fig. 20 (A) and Fig. 20 (B) is a diagram showing an example of error when verifying data reconstructed by the autoencoder. Fig. 20 (A) and Fig. 20 The vertical axis of (B) represents an arbitrary unit determined by a digital scale of signal processing.

[0269] In step S24 , the signal processing unit 10 may perform a heart sound envelope recognition process for selecting the optimal heart sound as in the following sub-steps 241 to 243 .

[0270] <Sub-step S241>

[0271] In sub-step S241, the signal processing unit 10 may prepare a large number of Fig.19 The self-encoder is trained using learning data for a data set corresponding to the envelope of the first and second sounds in the heart sound envelope shown on the upper side. Here, the number of vector dimensions of each learning data is recorded as J. Fig.19 The upper side shows the data augmentation and the automatic extraction of the phoneme suitable for the heart sound envelope. Fig.19 As shown below, a learned autoencoder is generated.

[0272] <Sub-step S242>

[0273] In sub-step S242, if Fig. 20 (A) and Fig. 20 As shown in (B), the signal processing unit 10 can perform reconstruction of the learned autoencoder. In this case, the signal processing unit 10 can be the envelope vector e of the object of classification recognition. l h As the i-th vector e after cutting off the length of J l,i h Next, the signal processing unit 10 calculates the original envelope waveform vector e l,i h The mean absolute percentage error (MAPE) of the reconstructed envelope waveform vector is shown in the following mathematical formula (10).

[0274] [Formula 10]

[0275]

[0276] In this case, as shown in the following [pseudo code 3], the signal processing unit 10 can calculate MAPE while sequentially shifting the window function used for cost function calculation.

[0277] [Pseudo code 3]

[0278] [Mathematical formula 11]

[0279]

[0280] fori=1:I

[0281] [Mathematical formula 12]

[0282]

[0283] end

[0284] However, length is a function that represents the length of the signal vector, and trunc is a function that rounds down to the integer by discarding the decimal point. l h The last part of the length J is discarded and the processing loop is caused not to enter the calculation.

[0285] exist Fig. 20 (A) and Fig. 20 In (B), one of the solid line graph and the dotted line graph represents a waveform based on detected data, and the other of the solid line graph and the dotted line graph represents a waveform based on reconstructed data. Fig. 20 (A) shows data with a MAPE greater than a threshold, i.e., data for which prediction cannot be performed well (cannot be properly reconstructed), and cannot be said to be a relatively good waveform. Fig. 20 (B) shows data where MAPE is not greater than the threshold, that is, data for which prediction is good (can be appropriately reconstructed), which can be said to be a waveform that is relatively good.

[0286] <Sub-step S243>

[0287] Fig.21 (A) and Fig.21 (B) is a graph showing an example of MAPE. Fig.21 (A) and Fig.21 (B) is a diagram for explaining the process of extracting the minimum point of MAPE from the reconstruction result by the autoencoder in order to select the optimal heart sound. Fig.21 (A) and Fig.21 The horizontal axis of (B) represents time in units of sample numbers. Fig.21 (A) and Fig.21 The vertical axis of (B) represents MAPE (unit: %). Fig.21(A) is the MAPE representing the best heart sound envelope l env In addition, Fig.21 (B) is the MAPE showing the non-optimal heart sound envelope l env Example.

[0288] In the electronic device 1 of one embodiment, the signal processing unit 10 may be Fig.21 (A) or Fig.21 (B) Set a threshold MAPE in the MAPE th Furthermore, the signal processing unit 10 can th The cost function is calculated based on the minimum point. For example, Fig.21 (A) and Fig.21 (B) The dashed lines shown below the respective graphs show the MAPE as an example. th = 10. Here, the number of local points below the threshold is used as the cost function. If such a cost function is expressed in a mathematical expression, it is as shown in the following mathematical expression (13).

[0289] [Mathematical formula 13]

[0290]

[0291] In the above mathematical formula (13), LocalMin is a function that lists the minimum points. In the electronic device 1 of one embodiment, the signal processing unit 10 can select the case where the cost function shown in the above mathematical formula (13) is the minimum as the best heart sound. That is, the signal processing unit 10 can select the number l of the best heart sound as shown in the following mathematical formula (14): best .

[0292] [Mathematical formula 14]

[0293]

[0294] Fig.21 (A) shows the MAPE l env Below the threshold MAPE th Therefore, the signal processing unit 10 can Fig.21 The envelope shown in (A) is an example of the envelope of the best heart sound. Fig.21 (B) shows the MAPE l env Below the threshold MAPE th Therefore, the signal processing unit 10 can Fig.21The envelope curve shown in (B) is an example of a non-optimal envelope curve of heart sounds.

[0295] Next, in step S25, the signal processing unit 10 statistically analyzes the best heart sound. In step S25, the signal processing unit 10 can extract the positive peak and negative peak of the envelope of the best heart sound selected in step S24, and perform statistical analysis between the peaks. Thus, information for calculating RRI is generated. In step S25, the signal processing unit 10 can use the estimation of the distribution of the Gaussian Mixture Model (GMM) or the EM algorithm (expectation?maximization algorithm) or the variational Bayes method, etc.

[0296] Below, refer to Fig. 22 and Fig.23 The process of generating a frequency distribution (histogram) of the peak intervals of the envelope of the heart sound will be described. Fig. 22 This is a diagram showing an example of an enlarged view of envelopes of two heart sounds. Fig. 22 The horizontal axis represents the time in the number of samples. Fig. 22 The vertical axis represents the signal level. Fig.23 It is a diagram showing an example of a histogram of peak intervals. Fig.23 The horizontal axis represents the time between peaks. Fig.23 The vertical axis represents degrees.

[0297] Fig. 22 An example of the envelope of the first heart sound (S1) and the second heart sound (S2) is shown. Fig. 22 The time interval between S1 and S2 (between S1 and S2) and the time interval between S2 and S1 (between S2 and S1) are shown in FIG. Fig.23 It is a diagram showing the degree distribution between S1 and S2 and between S2 and S1. Fig.23 The histograms corresponding to the interval S1 - S2 and the interval S2 - S1 are shown.

[0298] The probability density distribution of the normalized histogram becomes a mixed Gaussian distribution of the two clusters. Therefore, in order to estimate the RRI based on this information, it is necessary to estimate the mixed Gaussian distribution. Here, as shown in the following mathematical formulas (15) and (16), the mixed Gaussian distribution refers to a distribution described by the linear sum of multiple Gaussian distributions. Fig.23 As shown in the histogram of the peak intervals, the probability distribution between the peaks of the heart sounds becomes the mixed Gaussian distribution.

[0299] [Mathematical formula 15]

[0300]

[0301] [Formula 16]

[0302]

[0303] Here, x represents the data observation vector of d dimensions. k represents the mixing rate of Gaussian distribution. N represents normal distribution. μ k Represents the expected value of the kth Gaussian distribution. Σ k Represents variance.

[0304] The following describes the process of estimating a mixed Gaussian distribution using a variational Bayesian method. The process may include the following two processes: [first process] and [second process]. The following describes the [first process] and [second process] in more detail.

[0305] [First treatment]

[0306] The envelope of the heart sound (shown in the following mathematical formula (17)) is extracted for each time frame numbered k=1, 2, ..., K at a certain length T.

[0307] [Mathematical formula 17]

[0308]

[0309] However, the envelope of the heart sound represented by the above-mentioned mathematical formula (17) can be extracted while overlapping the envelope waveforms of each time frame.

[0310] In the above-mentioned [first process], the signal processing unit 10 divides the selected heart sound (shown in the above-mentioned mathematical formula (17)) into K time frames as the best heart sound. That is, the signal processing unit 10 generates the extracted envelope waveform as shown in the following mathematical formula (18).

[0311] [Mathematical formula 18]

[0312]

[0313] Fig.24 2 is a diagram for explaining the process of extracting the envelope of heart sounds. Fig.24 In the envelope of the heart sound shown on the upper side of , the time windows extracted for time frames t=1 and t=2 are shown. Fig.24 The envelope of the heart sound shown on the lower side of shows the time windows extracted for the time frames t=T-1 and t=T. Here, T and t are discrete times and represent non-negative integers. Fig.24 The envelope of the heart sound shown on the upper side shows the width w of the time window win , and shows the overlap width wo Here, the moving step of the time window is w step =w win -w o .

[0314] [Second Process]

[0315] The mixed Gaussian distribution latent variable at time t=T is used as the prior distribution, input t=T+1, and the variational Bayes method is used to update the latent variable.

[0316] In the above-mentioned [second process], the signal processing unit 10 performs Bayesian estimation of latent parameters including mixed Gaussian distribution for the time frame t=k using the variational Bayesian method described below.

[0317] Here, an outline of the estimation of the variational Bayesian method is recorded. The variational Bayesian method is a method for estimating the posterior distribution of a latent variable Z. Generally speaking, the variational Bayesian method is a method for approximating a certain approximate posterior distribution q(Z) with the actual posterior distribution p(Z|X) using the variational method. In the variational Bayesian method, latent variables and variables are synonymous. In the variational Bayesian method, all variables are treated as latent variables, i.e., probability variables, whose true values ​​cannot be calculated. Below, the outline of the variational Bayesian method is explained by generalizing the latent variables.

[0318] In the variational Bayesian method, instead of minimizing the Kulback-Leibler divergence (KL divergence) described as in the following equation (19), approximation is performed by maximizing the following equation (20). Equation (20) is also called the variational lower bound (Evidence of Lower Bound: ELBO).

[0319] [Mathematical formula 19]

[0320]

[0321] [Mathematical formula 20]

[0322]

[0323] Here, X is a matrix or vector representing a set of observed data. Z is a matrix or vector representing latent variables. q represents the approximate posterior probability distribution. p(Z|X) and p(X, Z) represent the true posterior distribution and simultaneous distribution, respectively.

[0324] That is, the signal processing unit 10 can obtain the approximate posterior distribution q(Z) of the latent variable Z by solving the following mathematical formula (21).

[0325] [Mathematical formula 21]

[0326]

[0327] Generally, in order to efficiently calculate the above equation (21), the mean field approximation shown in the following equation (22) is performed.

[0328] [Mathematical formula 22]

[0329]

[0330] The mean field approximation is an approximation based on the idea that the posterior probability distributions of the latent variables after several groups are all independent and are therefore represented by their respective products.

[0331] The signal processing unit 10 can solve the above-mentioned mathematical formula (19) under the condition of the above-mentioned mathematical formula (22) to obtain the Euler Larange equation represented by the following mathematical formula (23).

[0332] [Mathematical formula 23]

[0333]

[0334] By solving the equation of the above mathematical formula (23), the signal processing unit 10 can obtain the approximate posterior distribution of the variables of the i-th group as shown in the following mathematical formula (24).

[0335] [Mathematical formula 24]

[0336]

[0337] Among them, f(q i (Z i ),Z i ) is a function that satisfies the following mathematical formula (25).

[0338] [Mathematical formula 25]

[0339] F[q(Z)]=F[q,Z]=∫f(q i (Z i ), Z i )dZ (25)

[0340] Here, equation (24) means the expected value of the logarithm of the simultaneous probability of obtaining an observed value and a latent variable from a latent variable other than itself.

[0341] Hereinafter, a method of applying the variational Bayesian method described so far to the estimation of an actual mixed Gaussian distribution will be described.

[0342] First, the latent variables of the approximate posterior distribution need to be grouped and the mean field approximation is performed. In order to use the variational Bayesian update equation (24), initially, for the latent variables of the mixed Gaussian distribution, the respective dependencies are considered and the Bayesian theorem is used to decompose the simultaneous probability density distribution into individual probability distributions as shown in the following equation (26).

[0343] [Mathematical formula 26]

[0344] p(X,z,π,μ,∑)=p(X|Z,μ,∑)p(Z|π)p(π)p(μ|∑)p(∑) (26)

[0345] Here, the latent variable of the mixed Gaussian distribution can be the mixing rate of the Gaussian distribution π = [π1, π2, ..., π K ], the expected value of Gaussian distribution μ=[μ1,μ2,...,μ K ], Gaussian distribution variance Σ=[Σ1,Σ2,...,Σ K ] and the one-hot vector z of which Gaussian distribution the n data belong to n =[z n1 ,z n2 ,...,z nK ]. For these latent variables of mixed Gaussian distribution, considering their respective dependencies, using Bayes' theorem, as shown in the above mathematical formula (26), the simultaneous probability density distribution is decomposed into each probability distribution. The decomposition shown in the above mathematical formula (26) can be performed after the dependency of the latent variables is clarified using the curve graph model.

[0346] The conjugate prior distribution of the probability distribution of each factor of the above mathematical formula (26) is set as shown in the following mathematical formulas (27) to (31). For a certain likelihood function, when the prior distribution and the posterior distribution have the same functional form, the prior distribution and the posterior distribution are called the conjugate prior distribution of the likelihood function.

[0347] [Mathematical formula 27]

[0348]

[0349] [Mathematical formula 28]

[0350]

[0351] [Mathematical formula 29]

[0352]

[0353] [Math 30]

[0354]

[0355] [Mathematical formula 31]

[0356]

[0357] Here, regarding equation (29), P(π|α) is α=[α1,α2,...,α K ] is a Dirichlet distribution with parameters. In addition, B(α) is a β function extended to multiple variables. Regarding formula (31), W(Σ k |W,ν) is Σ with ν, D and W as parameters k Wishart distribution. In addition, ν is the number of data vectors. D is the number of variables. W is Σ k As shown in the following mathematical formula (32), C(W,ν) is a constant calculated using the Γ function.

[0358] [Math 32]

[0359]

[0360] Next, for the approximate posterior distribution, the one-hot vector z and other variables are grouped to specifically determine equation (24) as shown in the following equation (33).

[0361] [Mathematical formula 33]

[0362] q(z,π,μ,∑)=q(z)q(π,μ,∑) (33)

[0363] Regarding the above equation (33), the variational Bayesian update equation of the above equation (24) is mutually updated for two groups. The two groups may be group 1: z, and group 2: π, μ, Σ.

[0364] Originally, the variational Bayesian method is different from the EM algorithm and does not distinguish between latent variables and general variables. Therefore, in the variational Bayesian method, the distinction between the E-step (Expectation step) and the M-step (Maximization step) has no special meaning. However, for the sake of convenience, the above two steps are defined as follows.

[0365] E-step: Take the expected value for group 2 and update z to approximate the posterior distribution q(z)

[0366] M-step: Take the expected value of q(z) of group 1 in the E-step and update the approximate posterior distribution q(π,μ,Σ) of group 2

[0367] Hereinafter, as a signal processing performed by the signal processing unit 10 in the electronic device 1 according to one embodiment, a variational Bayesian method for processing a mixed Gaussian distribution from time t = T-1 to time t = T will be described. This processing may include the following five sub-steps, namely, sub-step 251 to sub-step 255, sandwiching the above-mentioned E-step and M-step.

[0368] <Sub-step S251>

[0369] In sub-step S251, each latent variable is initialized (at T = 0) or inherits the latent variable of t = T-1 as a prior distribution (T ≥ 1). Here, the latent variable of the posterior distribution calculated at time t = T is provided as the initial value at time t = T. Here, the latent variable of the posterior distribution calculated at time t = T can be set to α| t＝T ,β| t＝T-1 ,ν| t＝T-1 ,m| t＝T-1 , W| t＝T-1 . In addition, the initial value at time t = T can be set to α0| t＝T ,β0| t＝T ,ν0| t＝T ,m0| t＝T , W0| t＝T The subscript 0 indicates the initial value.

[0370] <Sub-step S252>

[0371] In sub-step S251, the approximate posterior distribution q(z) of z is calculated by the E-step, and the burden rate r is calculated using the result. nk (Posterior probability of one-hot vector) Here, as the E-step, the above-mentioned mathematical formula (33) is used to rewrite the above-mentioned mathematical formula (24) into the following mathematical formula (34).

[0372] [Math 34]

[0373] q(z)=Aexp(Elnp(X, π, μ, ∑)) (34)

[0374] Using the above equation (34), the burden rate r is calculated by the following equations (35) and (39): nk (Posterior probability of a one-hot vector).

[0375] [Mathematical formula 35]

[0376]

[0377] [Mathematical formula 36]

[0378]

[0379] [Mathematical formula 37]

[0380]

[0381] [Math 38]

[0382]

[0383] [Math 39]

[0384]

[0385] <Sub-step S253>

[0386] In sub-step S253, as an M-step, the load ratio r is used. nk To update the latent variables of the mixed Gaussian distribution (group 2). Here, n is the index of the data. In addition, k is the index of the Gaussian distribution. Here, as an M-step, the above mathematical formula (24) is rewritten as the following mathematical formula (40).

[0387] [Math 40]

[0388] q(π,μ,∑)=A exp(E ln p(X,z)) (40)

[0389] Use the above equation (40) and the following equations (41) to (48) to update the parameters.

[0390] [Mathematical formula 41]

[0391]

[0392] [Mathematical formula 42]

[0393]

[0394] [Mathematical formula 43]

[0395]

[0396] [Mathematical formula 44]

[0397] α k =α0| t＝T+1 +N k (44)

[0398] [Mathematical formula 45]

[0399] β k =β0| t＝T+1 +N k (45)

[0400] [Mathematical formula 46]

[0401] v k =v0| t＝T+1 +N k (46)

[0402] [Mathematical formula 47]

[0403]

[0404] [Mathematical formula 48]

[0405]

[0406] <Sub-step S254>

[0407] In sub-step S254, sub-steps S252 and S253 are repeated to perform convergence determination, and the processing of sub-steps S252 and S253 is terminated. Specifically, if the increase in the log-likelihood function or the increase in the ELBO is less than the set value or exceeds the maximum number of iterations between iterations, it can be determined to be converged.

[0408] In sub-step S254, ELBO shown in equation (20) can be used as an indicator for convergence determination. When equation (20) is rewritten based on the estimation of mixed Gaussian distribution, it becomes as shown in the following equation (49).

[0409] [Math 49]

[0410]

[0411] For the increment between iterations of the above mathematical formula (49), a threshold ΔF is set th , set the iteration index to l, and if the following mathematical formula (50) is satisfied, stop the iteration of the variational Bayes calculation.

[0412] [Mathematical formula 50]

[0413] Fl[q(z,π,μ,∑)]-F l-1 [q(z,π,μ,∑)]<ΔF th (50)

[0414] In addition, if the number of iterations exceeds a specified number, the variational Bayes calculation can be stopped.

[0415] <Sub-step S255>

[0416] In sub-step S255, for each latent variable, the expected value of each distribution is taken as the representative value to determine the mixed Gaussian model (GMM). The variational Bayes method is a method for calculating the posterior distribution of the mixed Gaussian distribution. Therefore, in order to finally determine the mixed Gaussian distribution, it is necessary to determine the representative value. Here, the mixed Gaussian distribution is determined by taking the expected value of π, μ, Σ in each latent variable. That is, as shown in the following mathematical formulas (51) to (53), the mixed Gaussian distribution is determined by taking the expected value of each as the representative value.

[0417] [Mathematical formula 51]

[0418]

[0419] [Mathematical formula 52]

[0420]

[0421] [Mathematical formula 53]

[0422]

[0423] Based on the above, the mixed Gaussian distribution at time t=T obtained from the representative value of the approximate posterior distribution is as shown in the following mathematical formula (54) and mathematical formula (55). The mixed Gaussian distribution obtained in this way can be used in the next step S26.

[0424] [Mathematical formula 54]

[0425]

[0426] [Mathematical formula 55]

[0427]

[0428] In fact, if Fig. 22 and Fig.23 As shown, in most cases including healthy people, only the first sound (first heart sound (S1)) and the second sound (second heart sound (S2)) are obvious. In such a case, it is simplified to K=2. For subjects with abnormal heartbeats, the third and / or fourth sounds may be obvious. In such a case, K=3 or 4.

[0429] According to the above, [the second process] can be completed and the process of step S25 is also completed.

[0430] Next, in step S26 , the signal processing unit 10 calculates the RR interval (RRI) based on the statistical information of the heart sound interval generated in step S25 .

[0431] More specifically, in step S26, the signal processing unit 10 may estimate the RRI using the mixed Gaussian distribution (the above-mentioned mathematical formulas (54) and (55)) calculated in step S25. In this case, the signal processing unit 10 first estimates the distribution of the RRI using the above-mentioned mathematical formulas (54) and (55).

[0432] For the reproductive property of the normal distribution, that is, two probability variables X and Y that follow the normal distribution, the following mathematical formula (56) holds.

[0433] [Mathematical formula 56]

[0434]

[0435] Therefore, for the above equations (54) and (55), by considering Fig. 22 and Fig.23 , the distribution of RRI is estimated as shown in the following mathematical formula (57). In the mathematical formula (57), μ RRI and Σ RRI As shown in the following mathematical formula (58).

[0436] [Mathematical formula 57]

[0437]

[0438] [Mathematical formula 58]

[0439]

[0440] Here, in Fig. 22 and Fig.23 In the example, if the time of the peak value of the envelope of the picked up heart sound is vectorized and recorded as t peak , then it is as shown in the following mathematical formula (59).

[0441] [Math 59]

[0442]

[0443] Next, if the average value μ of the probability density distribution of RRI defined by the above equations (57) and (58) is arranged RRI Let t be the vector of non-negative integer multiples of targ , then it is as shown in the following mathematical formula (60).

[0444] [Math 60]

[0445] t targ =[μ RRI , 2μ RRI ,…,LμRRI ] (60)

[0446] t targ is an approximate target time for peak A candidate corresponding to the point where the RRI is taken is selected. Next, the peak point of the heart sound envelope considered to be the RRI is extracted by performing DP (Dynamic Programming) matching between the above equations (59) and (60).

[0447] Fig.25 is a diagram illustrating DP matching for RRI estimation. Fig.25 In the horizontal direction, the vector t is arranged peak , arranging vector t in the vertical direction targ .exist Fig.25 In the figure, the circle represents t peak and t targ Match. In addition, Fig.25 In the figure, the cross indicates the point where a match was attempted but not found using the following [pseudo code 4]. peak Is it related to t targ The vector of the flags that any element of matches is set to m RRI . m RRI It can be expressed as the following mathematical formula (61).

[0448] [Mathematical formula 61]

[0449]

[0450] The calculation process to obtain the match is performed through the following [pseudo code 4].

[0451] [Pseudocode 4]

[0452] [Mathematical formula 62]

[0453] D th =a∑ RRI (62)

[0454] for l=1:L (loop of vertical axis)

[0455] for n=1:N (loop along the horizontal axis)

[0456] [Mathematical formula 63]

[0457]

[0458] if D t ≤D th

[0459] [Mathematical formula 64]

[0460]

[0461] (Write 1 in the match flag)

[0462] break (the matching flag becomes 1, gradually breaking the horizontal axis loop)

[0463] else

[0464] [Mathematical formula 65]

[0465]

[0466] (Write 0 in the match flag)

[0467] end

[0468] n=n+1(make Fig.25 The iterative increase of the horizontal axis)

[0469] end

[0470] l=l+1(make Fig.25 The iterative increment of the vertical axis)

[0471] end

[0472] Here, the above mathematical formula (62) means that the variance Σ of the probability density distribution of RRI shown in the following mathematical formula (66) is obtained by RRI Multiply by a constant to determine t peak and t targ The distance threshold D th .

[0473] [Mathematical formula 66]

[0474]

[0475] The above mathematical formula (63) means that t is calculated within the loop. peak and t targ That is, [pseudo code 4] represents the extraction of t that meets the conditions shown in the following mathematical formula (67): peak The elements are treated as matching points.

[0476] [Mathematical formula 67]

[0477]

[0478] As the result of the calculation of [pseudo code 4], m RRI The number of non-zero elements of is set to M. That is, the following mathematical formula (68) holds.

[0479] [Mathematical formula 68]

[0480]

[0481] Next, by taking the matching flag m RRI and t peak By taking the Hadamard product (element product) of , and excluding 0, the time column vector t of the point that becomes the RRI can be calculated RRI That is, only extract Fig.25 The result of marking the points on the grid with circles is shown in the following mathematical formula (69).

[0482] [Math 69]

[0483]

[0484] Here, RmZero is a function that removes 0 from the elements of a vector.

[0485] As shown in the following mathematical formula (70), by taking t RRI The vector Δt representing the RRI data is calculated by taking the difference between the adjacent elements of RRI .

[0486] [Mathematical formula 70]

[0487]

[0488] Finally, in the vector Δt representing the RRI data RRI Defect processing and smoothing processing are required. The matching in the above [pseudo code 4] is Fig.25 In the loop, none of them become D in the first iteration t ≤D th In the case of RRI Defect, Δt RRI The missing / abnormal value processing can be performed as follows: As shown in the following pseudo code 5, for Δt shown in the following mathematical formula (71), RRI The average value is allowed to be up to a certain multiple b. When it is greater than a certain multiple b, it is replaced by the average value.

[0489] [Mathematical formula 71]

[0490]

[0491] [Pseudo code 5]

[0492] for m=1:M-1

[0493] [Mathematical formula 72]

[0494]

[0495] [Mathematical formula 73]

[0496]

[0497] end

[0498] end

[0499] In this way, the RRI data after the missing / outlier processing is added with a prime symbol and recorded as Δt' RRI This enables the moving average filter to further function and to convert the smoothed data Δt" RRI The final RRI data is used as the final RRI data. Fig.26 and Fig. 27 , the Δt" obtained as described above RRI An example of data is shown in Fig.28 . Fig.26 , Fig. 27 and Fig.28 1 is a diagram showing an example of a time series waveform of a heart sound and RRI based on the electronic device 1 according to one embodiment. Fig.26 and Fig. 27 In the figure, the horizontal axis represents time and the vertical axis represents vibration speed. Fig.28 In the figure, the horizontal axis represents time and the vertical axis represents RRI.

[0500] Fig.26 1 is a diagram showing a time series waveform of heart sounds obtained by the electronic device 1 according to one embodiment. Fig. 27 It is enlarged to show Fig.26 Graph of the time span of the area indicated by the dotted line.

[0501] like Fig. 27 As shown, based on the time series waveform of the heart sound obtained by the electronic device 1 of one embodiment, the first heart sound S1 and the second heart sound S2 can be clearly identified. Fig. 27 As shown, based on the timing waveform of the heart sound obtained by the electronic device 1 of one embodiment, the heartbeat interval RRI calculated based on the interval between the first heart sound S1 and the second heart sound S2 can be clearly determined. Here, although RRI and the heartbeat interval are not strictly the same, they can be considered to be approximately the same. Therefore, in the present disclosure, the case where RRI is the same as the heartbeat interval is described.

[0502] Fig.28 The results shown are based on Fig.26 and Fig. 27 The result of the calculation of the heart sounds accurately extracted from Fig.28 The results shown have an accuracy of a few ms.

[0503] Next, in step S27, the signal processing unit 10 calculates the heart rate variability (HRV). In step S27, the signal processing unit 10 may calculate the power spectral density (PSD) by performing frequency analysis on the time series waveform of the RRI. The frequency analysis of the time series waveform of the RRI may be performed using, for example, the Walsh method.

[0504] More specifically, in step S27, the signal processing unit 10 may process the Δt″ estimated in step S26 by using a temporally repeated window function, such as the Walsh method. RRI FFT and power spectrum calculation are performed. Thus, the signal processing unit 10 can obtain the power spectrum density (PSD) of the RRI.

[0505] Fig.29 : is a diagram showing an example of the power spectral density (PSD) of the RRI obtained by the signal processing unit 10. Fig.29 In FIG. 1 , the horizontal axis represents frequency and the vertical axis represents power spectrum density (PSD). The calculation method of the power spectrum such as the Walsh method can adopt a known general method. Therefore, a more detailed description of the calculation method of the power spectrum such as the Walsh method is omitted.

[0506] Fig.29 It means according to Fig.28 The timing waveform of RRI shown is a diagram of an example of power spectral density calculated using the Walsh method. According to an embodiment of the electronic device 1, it is possible to calculate the power spectral density of RRI based on RRI with an accuracy of several ms. Therefore, according to an embodiment of the electronic device 1, it is also possible to accurately calculate the 0.15 Hz to 0.4 Hz component of HF called heartbeat PSD.

[0507] As described above, according to the electronic device 1 of one embodiment, it is possible to obtain, for example, Fig.26 and Fig. 27 The detailed cardiac waveform shown, Fig.28 The accurate RRI timing data shown and Fig.29 The power spectral density (PSD) of the heartbeat interval shown. According to an electronic device 1 of one embodiment, it is possible to detect the heartbeat of a human body or the like with good accuracy by using the transmission and reception of radio waves. Therefore, according to an electronic device 1 of one embodiment, it is possible to detect weak vibrations such as the heartbeat of a human body or the like with good accuracy by using the transmission and reception of radio waves such as millimeter waves, and it is expected to play a role in various fields.

[0508] Here, in Figure 26 to Figure 29In the example, only the first heart sound S1 and the second heart sound S2 which are obvious in a healthy person are described. However, according to the electronic device 1 of one embodiment, even when the third heart sound and / or the fourth heart sound due to an abnormal heartbeat occur, it is possible to perform the same processing.

[0509] (Other embodiments)

[0510] Other embodiments will be described below.

[0511] In other embodiments, the electronic device 1 of one embodiment may calculate the heart rate based on the RRI data instead of calculating the PSD based on the RRI data. In this case, the signal processing unit 10 of the electronic device 1 may, for example, Fig.13 In step S27 shown in the figure, a process of calculating the heart rate from the RRI data is executed. In this case, the signal processing unit 10 can perform the following processes 1 to 3 in step S27, for example.

[0512] 1. Low-pass filter (allows frequencies below 1 Hz to pass)

[0513] 2. Processing of taking the inverse of RRI

[0514] 3. Round the reciprocal obtained in 2 above to a natural number

[0515] By using these processes, the heart rate can be calculated based on the RRI. In one embodiment, the electronic device 1 of one embodiment can perform LPF processing, reciprocal calculation and / or rounding to natural numbers, etc. as Fig.13 The process of step S27 is shown.

[0516] In other embodiments, an electronic device 1 of an embodiment is performing Fig.13 The processing shown in the figure and the processing 1 to 3 above can also be changed. Fig.13 The processing shown in step S24 is shown. For example, as Fig.13 In step S24 shown, the electronic device 1 of one embodiment may also perform classification and recognition based on an autoencoder, template matching based on a cross-correlation function, classification and recognition based on LSTM, and / or classification and recognition based on other dimensional vectors of SVM.

[0517] In other embodiments, the electronic device 1 of one embodiment may also be Fig.13 In such processing, the processing of step S25 is performed using the EM algorithm. However, the EM algorithm is not a Bayesian estimation method. Therefore, the EM algorithm does not have a mechanism for calculating the posterior distribution based on the prior distribution. When used for estimating the mixed Gaussian distribution, the variables π, μ, and Σ describing the mixed Gaussian part are point estimated.

[0518] Therefore, when using the EM algorithm to perform Fig.13 In the case of step S25 shown, any of the following methods is adopted.

[0519] For each time frame, a mixed Gaussian distribution was estimated using the EM algorithm without considering the relationship between time frames.

[0520] For each time frame, after estimating the mixed Gaussian distribution using the EM algorithm, the variables π, μ, and Σ of each mixed Gaussian distribution are averaged before and after, or time series filtering is performed based on the Kalman filter.

[0521] In one embodiment, the transmitting antenna array 24 and / or the receiving antenna array 31 of the electronic device 1 are not limited to Figure 8 For example, in one embodiment, the receiving antenna array 31 of the electronic device 1 may be Fig.30 The structure shown. Fig.30 is a diagram showing an example of a URA (Uniform rectangular array) receiving antenna. Fig.30 The URA receiving antenna shown does not change the directivity by a beam former in the transmitting antenna array 24, and the arrival direction of two angles is estimated only by the URA receiving antenna.

[0522] exist Figure 2 In the electronic device 1 shown, the signal processing unit 10 is described as having functional units such as the heartbeat extraction unit 13 and the calculation unit 14. However, in one embodiment, the processing performed by the heartbeat extraction unit 13 and / or the calculation unit 14 may be performed by an external computer or processor.

[0523] Additionally, in one embodiment, Fig.12 The processing in step S14 shown (processing of analyzing singular value decomposition / principal components) may be replaced by other partial space methods.

[0524] Additionally, in one embodiment, Fig.12 The processing in step S14 (processing of analyzing singular value decomposition / principal components) shown in FIG. 1 may be omitted if the number of samples N of one chirp can be set to a larger number. Fig.12 The processing in step S14 (processing of analyzing singular value decomposition / principal components) shown above may be omitted when mixing of other noises can be eliminated by a hardware method or the like.

[0525] exist Fig.13In the process of step S23 (generation of heart sound envelope waveform), the result of continuous wavelet transform is added to the frequency axis direction to calculate using [pseudo code 2]. However, the envelope waveform may be generated using other methods such as moving average or Hilbert transform.

[0526] The electronic device 1 of one embodiment may also be Fig.29 In the power spectral density (PSD) of the RRI shown, the RRI is calculated by taking the interval between the negative peaks of the output of the reconstruction error of the autoencoder. That is, the electronic device 1 of one embodiment may also replace Fig.24 The heart sound envelope is shown, while the curve representing the reconstruction error of the autoencoder is used Fig.29 , and then set the point equivalent to RRI as Fig.29 The negative peak in the image. The time resolution at this time is limited by the moving step of the window in [pseudo code 3]. In order to reduce the time resolution, just reduce the moving step of the window in [pseudo code 3].

[0527] Fig.31 and Fig.32 is a diagram illustrating the negative peak of the reconstruction error output of the autoencoder. Fig.31 and Fig.21 (A) is a graph showing the MAPE of the envelope of the best heart sound in the same manner. Figure 3 The horizontal axis represents the time in the number of samples. Fig.31 The vertical axis represents MAPE (in %). Fig.32 It is enlarged to show Fig.31 The part surrounded by the dotted line in . Fig.32 As shown in FIG. 1 , according to the graph representing the reconstruction error of the autoencoder, a negative peak can be obtained when the templates of the heart sounds match.

[0528] The electronic device 1 of one embodiment can also be used when estimating a mixed Gaussian distribution based on a variational Bayes method. Fig.13 That is, the electronic device 1 of one embodiment may also use only Fig.13 The process in step S25 is replaced by Fig. 22 and Fig.23 In this case, the mixed Gaussian distribution becomes a single peak, that is, a Gaussian distribution. In this embodiment, the heart sound interval calculated by the above method can also be Fig.31 and Fig.32 The time intervals between negative peaks are histogrammed for Fig.13 The calculation of step S26 shown in FIG. 1 is equivalent to the heart sound interval. The histogram of this embodiment can also be a single packet characteristic. However, in this embodiment, in the case of containing some noise, the mixed Gaussian distribution ( Fig.32 The time between the negative peaks and the time interval between other noise peaks are separated to find the expected Fig.32 The time between negative peaks.

[0529] The electronic device 1 of one embodiment may also be implemented using Markov Chain Monte Carlo methods (MCMC method). Fig.13 The processing in step S25.

[0530] As described above, an electronic device 1 according to one embodiment uses, for example, a millimeter wave sensor having a plurality of transmitting antennas and a plurality of receiving antennas to detect weak vibrations such as a heartbeat. When no object is detected, an electronic device 1 according to one embodiment changes the transmission phase of the antenna to perform a beamforming pattern of the transmitting antenna and detect the body movement of the object. On the other hand, once the body movement of the object is detected, the electronic device 1 according to one embodiment performs beamforming in the direction of the body movement to detect the heartbeat. In this way, according to an electronic device 1 according to one embodiment, the signal quality can be improved by automatically detecting the direction of the human body. Therefore, according to an electronic device 1 according to one embodiment, the detection accuracy and / or detection range of the heartbeat can be improved. Therefore, according to an electronic device 1 according to one embodiment, the human heartbeat can be detected with high accuracy.

[0531] The present disclosure is described based on the drawings and embodiments, but it should be noted that it is easy for those skilled in the art to make various deformations or modifications based on the present disclosure. Therefore, it should be noted that these deformations or modifications are also included in the scope of the present disclosure. For example, the functions contained in each functional unit can be reconfigured in a logically non-contradictory manner. Multiple functional units can be combined into one or divided. The above-mentioned embodiments of the present disclosure are not limited to faithfully implementing the respective embodiments described, and can be implemented by appropriately combining each feature or omitting a part. That is, the content of the present disclosure can be variously deformed and modified based on the present disclosure as long as the technical personnel of the present disclosure. Therefore, these deformations and modifications are included in the scope of the present disclosure. For example, in each embodiment, each functional unit, each technical solution, each step, etc. can be added to another embodiment in a logically non-contradictory manner, or can be replaced by each functional unit, each technical solution, each step, etc. of another embodiment. In addition, in each embodiment, multiple functional units, each technical solution, each step, etc. can be combined into one, or divided. In addition, each embodiment of the present disclosure described above is not limited to implementing each embodiment described individually faithfully, and each feature can be appropriately combined or partially omitted to implement.

[0532] The above embodiment is not limited to being implemented as the electronic device 1. For example, the above embodiment may also be implemented as a control method of a device such as the electronic device 1. Furthermore, the above embodiment may also be implemented as a program executed by a device such as the electronic device 1 or a storage medium or recording medium having the program recorded thereon.

[0533] In addition, the electronic device 1 of the above embodiment includes components such as the transmitting antenna array 24 and the receiving antenna array 31 that constitute a so-called radar sensor. However, the electronic device of one embodiment may also be configured as, for example, a signal processing unit 10. In this case, the signal processing unit 10 may be implemented as, for example, a processing unit having a function of processing signals handled by the transmitting antenna array 24 and the receiving antenna array 31.

[0534] Description of Reference Numerals

[0535] 1 Electronic equipment

[0536] 10 Signal Processing Unit

[0537] 11 Signal generation and processing unit

[0538] 12. Received signal processing unit

[0539] 13 Heartbeat extraction unit

[0540] 14 Computing Department

[0541] 21 Send DAC

[0542] 22 Transmitting Circuit

[0543] 23 mmWave Transmitter Circuit

[0544] 24 Transmitting Antenna Array

[0545] 31 Receiving Antenna Array

[0546] 32 Mixer

[0547] 33 Receiving Circuit

[0548] 34 Receive ADC

[0549] 50 Communication interface

[0550] 60 External devices

Claims

1. An electronic device, wherein: have: The transmitting unit transmits the transmitting wave; a receiving unit that receives a reflected wave of the transmitted wave from a target object; and A signal processing unit, which detects the distance position, direction and speed of the target object based on a transformed signal obtained by Fourier transforming a beat signal between the transmission wave and the reflected wave; The signal processing unit includes an extraction unit, which extracts heartbeat information of the target object based on the detected distance, direction and speed of the target object. The extraction unit extracts a signal component corresponding to the vibration of the heartbeat accompanying the target object from the transformed signal using a plurality of window functions. The extracting unit selects the most suitable heart sound signal as the heart sound by applying a learning classifier to the extracted signal component. The extracting unit performs envelope processing on the most suitable heart sound signal, thereby selecting the most suitable best signal as the heartbeat. The extraction unit acquires a value indicating an interval of peak times from the optimal signal, and extracts a heartbeat interval of the target object based on a variance and a median value of the value indicating the interval.

2. The electronic device according to claim 1, wherein: The extraction unit extracts the signal component by focusing on a plurality of positions of the transformed signal. The extraction unit performs a process of removing noise from the extracted signal component using singular value decomposition, and the process uses the frequency of the extracted signal component.

3. The electronic device according to claim 1, wherein: The extraction unit performs the following processing: Applying a plurality of window functions to the transformed signal to extract a micro-Doppler component of a point group where the target object exists; performing at least one of principal component analysis and singular value decomposition on the extracted micro-Doppler component; performing frequency filtering using at least one of short-time Fourier transform, continuous wavelet transform, and bandpass filter on the result of at least one of the principal component analysis and singular value decomposition; The frequency-filtered result is subjected to multi-resolution analysis including discrete wavelet transform using a wavelet function and a scaling function suitable for heart sounds.

4. The electronic device according to claim 1, wherein: The extraction unit determines the optimality of a signal component corresponding to a vibration accompanying a heartbeat of the target object as a heart sound signal using a learned classifier. For the signal component determined to be optimal by the learning classifier, a moving variance process is used to extract an envelope signal.

5. The electronic device according to claim 1, wherein: The extraction unit extracts an envelope signal from the signal component using any one of continuous wavelet transform, discrete wavelet transform, wavelet scattering coefficient, Mel-frequency cepstral coefficient, and moving variance processing. The extraction unit determines an optimal envelope signal based on a score calculated for the heart sound of the signal component using a trained learning classifier for the envelope signal.

6. The electronic device according to claim 5, wherein: The extracting unit obtains the interval value of the peak time for the extracted optimal envelope signal, and estimates the variance and median value of the value set of the interval value, The extracting unit creates a reference for extracting heartbeat intervals from the envelope signal based on the estimated variance and median value. The extraction unit extracts the heartbeat interval from the envelope signal using the reference.

7. The electronic device according to claim 6, wherein: The extracting unit detects the first heart sound and the second heart sound based on the extracted optimal envelope signal, The extracting unit calculates the heartbeat interval of the target object by using DP matching based on a histogram or probability density distribution of the interval between the first heart sound and the second heart sound and the interval between the second heart sound and the first heart sound.

8. A method for controlling an electronic device, in, include: a step of transmitting a transmission wave from a transmitting unit; A step of receiving a reflected wave of the transmitted wave from a target object during reception; A step of detecting the distance position, direction and speed of the target object based on a transformed signal after Fourier transforming the beat signal of the transmitted wave and the reflected wave; and A step of extracting heartbeat information of the object based on the detected distance, direction and speed of the object; In the step of extracting the heartbeat information of the object, A signal component corresponding to the vibration of the heartbeat of the target object is extracted from the transformed signal using a plurality of window functions. By applying a learning classifier to the extracted signal components, the most suitable heart sound signal is selected as the heart sound. By performing envelope processing on the most suitable heart sound signal, the most suitable best signal is selected as the heartbeat. A value indicating an interval of peak times is acquired from the optimal signal, and a heartbeat interval of the target object is extracted based on a variance and a median value of the value indicating the interval.

9. A program, wherein: Have the electronic device perform the following steps: sending a transmission wave from a transmitting unit; receiving a reflected wave of the transmitted wave from a target object in receiving; Detecting the distance position, direction and speed of the target object based on a transformed signal after Fourier transforming the beat signal of the transmitted wave and the reflected wave; as well as Extracting heartbeat information of the object based on the detected distance, direction and speed of the object; In the step of extracting the heartbeat information of the object, A signal component corresponding to the vibration of the heartbeat of the target object is extracted from the transformed signal using a plurality of window functions. By applying a learning classifier to the extracted signal components, the most suitable heart sound signal is selected as the heart sound. By performing envelope processing on the most suitable heart sound signal, the most suitable best signal is selected as the heartbeat. A value indicating an interval of peak times is acquired from the optimal signal, and a heartbeat interval of the target object is extracted based on a variance and a median value of the value indicating the interval.

Images