Sleep sign monitoring method based on millimeter wave radar

Abstract

The invention discloses a sleep sign monitoring method based on a millimeter-wave radar, which comprises the following steps: firstly, presetting acquisition parameters of the millimeter-wave radar, transmitting signals of the millimeter-wave radar, capturing reflected signals by a receiving antenna, sampling to obtain intermediate-frequency digital signals, and sending a sampling result to a rear-end computing device; extracting features related to the sleeping posture and the breathing frequency from the collected intermediate frequency digital signals; and recognizing the sleep posture of the user by using the point cloud features and the breathing features obtained after feature extraction. And then, effective reflection points are extracted, and the respiratory rate is accurately estimated by using the reflection points through a weighted average method. The invention aims to realize an accurate and reliable vital sign monitoring system by using commercial millimeter wave radar equipment which is low in cost and easy to deploy, stably recognize six sleeping postures and accurately measure the respiratory rate; rich and stable point cloud information is obtained; the characterization of the point cloud on the sleep posture is enhanced; and a more accurate respiratory rate estimation result is obtained.

Description

Technical Field

[0001] The present invention relates to the field of wireless sensing technology, and in particular to a sleep sign monitoring method based on millimeter wave radar. Background Art

[0002] Sleep, as an indispensable and important part of human life, has a profound impact on health and well-being. In recent years, research on monitoring sleep signs (including sleeping posture, breathing and heartbeat, etc.) has attracted much attention. The development of this field has important background and significance in health care, biomedical research and personal health management. By monitoring sleep signs, individuals can understand their sleep quality and habits, so as to carry out more accurate health management. In addition, sleep sign monitoring also provides valuable opportunities for early disease screening and monitoring, helping doctors and patients to detect and intervene in potential health problems early. For example, sleeping posture is an important indicator for diagnosing postural obstructive sleep apnea syndrome (P-OSA); pressure sores are chronic wounds that are prone to recurrence. In hospitals and communities, caregivers need to ensure that patients change their postures regularly to prevent long-term squeezing of one side of the skin to cause pressure sores; for epilepsy patients, incorrect sleeping postures increase the risk of sudden death. Applying sleep sign monitoring data to clinical research will help promote a deeper understanding of sleep physiology, reveal the intricate relationship between sleep and health, and promote the development of medicine and biomedicine.

[0003] Millimeter wave radar refers to a radar system that uses the millimeter wave frequency band (usually between 30GHz and 300GHz). At present, linear frequency modulated continuous wave (FMCW) radar is widely used in the civilian field. The radar system generates a local oscillator signal through oscillators and other devices. The local oscillator signal is frequency modulated, phase modulated, and power amplified before emitting electromagnetic waves from the transmitting antenna. The characteristic of this electromagnetic wave is that the frequency changes linearly with time. After being reflected by an object, it is captured by the receiving antenna, and then the intermediate frequency digital signal is obtained through the front-end low noise amplifier, mixing, sampling and other processing. By performing a series of processing on the intermediate frequency digital signal, the distance, speed, angle and other information of the target can be estimated, and the point cloud can be further calculated and generated. By processing and analyzing the phase of the reflected signal, the micro-motion information of the target, such as breathing rate and heart rate, can be estimated.

[0004] Convolutional Neural Network (CNN) is a deep learning algorithm that uses deep learning methods, especially convolutional neural network (CNN) and Transformer encoder for signal processing and classification. CNN is responsible for local feature extraction, such as edge and texture recognition, while Transformer processes sequence data and captures global and local relationships. Together, they improve the accuracy of posture recognition.

[0005] Current research on sleep sign monitoring based on millimeter-wave radar can only monitor breathing and heartbeat, or can only identify a few categories of sleeping positions. However, comprehensive monitoring of vital signs and sleeping positions can reveal the relationship between various physiological indicators and provide more accurate health diagnosis and prediction. Therefore, it is of great significance to study a non-contact technical solution that can simultaneously monitor vital signs and identify a large number of sleeping positions. Summary of the invention

[0006] The purpose of the present invention is to make up for the defects of the existing technology and provide a sleep sign monitoring method based on millimeter wave radar. The present invention extracts point cloud data from the signal received by the radar, removes noise points through multi-frame azimuth probability clustering analysis, and obtains stable point cloud features. The phase changes caused by breathing behavior are analyzed through phase analysis, breathing features are extracted, and sleeping posture is described in combination with point cloud.

[0007] The present invention is achieved through the following technical solutions:

[0008] A sleep sign monitoring method based on millimeter wave radar specifically comprises the following steps:

[0009] Step 1-1, data collection: first, the acquisition parameters of the millimeter wave radar are preset, the millimeter wave radar transmits a signal according to the preset parameters, the receiving antenna captures the reflected signal, the millimeter wave radar RF front end amplifies, mixes, and filters the received signal, obtains an intermediate frequency digital signal after sampling, and sends the sampling result to the back-end computing device;

[0010] Step 1-2, feature extraction: extracting features related to sleeping posture and breathing rate from the collected intermediate frequency digital signal, namely, point cloud feature generation and breathing feature extraction;

[0011] Step 1-3, sleeping posture recognition: using the point cloud features and breathing features obtained after feature extraction to recognize the user's sleeping posture.

[0012] Step 1-4, respiratory rate estimation: extract effective reflection points from the point cloud data using specific parameters according to different sleeping postures, and use these reflection points to accurately estimate the respiratory rate through a weighted average method based on an exponential function.

[0013] The acquisition parameters of the preset millimeter-wave radar described in step 1-1 specifically include the selectively enabled transceiver antenna parameters, the bandwidth of the transmitted FM wave, the FM time, the number of chirps, the frame length, the ADC sampling frequency and the number of sampling points.

[0014] The steps 1-2 are as follows: extracting features related to sleeping posture and breathing rate from the collected intermediate frequency digital signal, i.e., generating point cloud features and extracting breathing features:

[0015] In the point cloud feature generation, the probability clustering analysis of multi-frame azimuth angles is applied. For each distance bin, the azimuth angles are clustered with a certain error range as the radius to obtain the azimuth angle estimation. Finally, the corresponding altitude angle is estimated under the selected azimuth angle, and the point cloud in the Cartesian coordinate system is generated by combining the reflection distance calculation.

[0016] Step 1-2-1, after receiving the data returned by the radar, the back-end computing device converts its format into {ADC, Chirp, Horizontal, Vertical, Frame}, where ADC is the sampling result of the intermediate frequency digital signal, Chirp represents the snapshot dimension, Horizontal represents the horizontal dimension of the two-dimensional virtual antenna array, Vertical represents the height dimension, and Frame represents the frame dimension;

[0017] Step 1-2-2, use Range-FFT to estimate the frequency components of the intermediate frequency signal, further calculate the distance information, and convert the data format into {Range, Chirp, Horizontal, Vertical, Frame}, where Range represents the distance from the radar to the target; then, extract the signal data within the effective sensing range through the preset scenario;

[0018] Step 1-2-3, use Doppler-FFT to estimate the speed information and convert the data format into {Range, Doppler, Horizontal, Vertical, Frame}, where Doppler represents the speed of the target. When a person is sleeping, the body is in a relatively static state, and almost only the upper body moves periodically with breathing. This slow movement speed is very low, less than the speed range represented by a Doppler bin, that is, it is included in the zero Doppler bin. Therefore, by extracting the data of the zero Doppler bin, the moving targets within the perception range can be filtered out, and at the same time, the signal reflected by the human body in the sleeping state can be retained.

[0019] Step 1-2-4, use Angle-FFT to estimate the reflection angle. After performing Angle-FFT in the azimuth antenna dimension, the data format becomes {Range, Azimuth, Vertical, Frame}, where Azimuth represents the azimuth angle and Vertical represents the altitude angle. The range-azimuth angle estimation result can be expressed as:

[0020]

[0021] In the formula, y r,θ represents the reflected signal of the target at a distance of r and an angle of θ, A is the amplitude, j is the imaginary unit, and f cis the starting frequency of the chirp, c is the speed of light, d r is the distance from the reflector to the reference element, Δd θ is the path difference due to θ, and R is the perceived distance. Then, the actual incident angle of the reflected signal at r is estimated, and the angular spectrum of r is ρ = |y r |Perform peak search and obtain n peaks where θ 1 and θ 2 Respectively represent the angle values ​​corresponding to the first peak and the second peak. This paper adopts a threshold method based on signal-to-noise ratio to extract reliable peaks, and sets an effective azimuth extraction range to filter out some background static target interference.

[0022] Step 1-2-5, purify the azimuth using an algorithm based on the probability clustering of multi-frame azimuth angles. The angle estimated from a single snapshot is easily affected by random noise and multipath reflection interference and produces deviations, which will cause the subsequently generated point cloud features to become unreliable. In order to suppress this interference, the present invention designs an algorithm based on the probability clustering of multi-frame azimuth angles. The basic idea of ​​the algorithm is that in the estimation results of different frames, the arrival angle of the real target will appear more times, and the angle will only fluctuate within a small range. The angle estimation results of false targets caused by noise interference, multipath reflection, etc., have large numerical variations between different frames. Moreover, estimating the breathing characteristics of the human body requires multiple frames of data, so the angle purification algorithm improves the utilization rate of frame data. The process of the purification algorithm based on the probability of multi-frame azimuth angles is as follows:

[0023] Let the total number of frames be N frame , perform Angle-FFT on the snapshot of each range bin under the zero Doppler bin, and perform the same operation on each frame to obtain the azimuth of all frames First, count θ r The probability of each angle appearing in , and arrange them in descending order. Then, the angle with the largest probability is used as the cluster center and a new category C is assigned to it 1 , and use the preset noise interference tolerance as the cluster radius γ, and then traverse the remaining angles. If the difference Δθ between the currently traversed angle and the cluster center is less than the cluster radius γ, the angle is considered to belong to the cluster and its category is changed to C 1 After traversing all the remaining angles, the first round of clustering is completed and the category is C 1 The angle from θ r Then, the angle with the highest probability in the current angle set is used as the new cluster center, and the category cluster is set to C 2 , a new round of clustering is performed, and after traversing the remaining angles, the 2All angles. Repeat this process until the angle pool is empty. At this point, all estimated azimuth angles have been purified into several categories. The more angles a category contains, the more reliable the angles contained in the category. Sort the categories according to the number of angles they contain, and finally, take the central angles of the categories containing the most angles as the extraction results. In order to ensure that the number of points contained in the point cloud input into the deep neural network is the same, if the number of categories estimated by the current sampled signal is not enough, copy the category with the most angles to fill it. Then, perform Angle-FFT in the vertical antenna dimension to estimate the elevation angle Specifically, first extract the Angle-FFT result corresponding to each valid azimuth angle, and then use the peak search algorithm based on the signal-to-noise ratio threshold to estimate the effective elevation angle. As with the azimuth angle, the effective elevation angle is still extracted by limiting the pre-known perception range to filter out irrelevant object reflections. At this point, the data of {Range, Azimuth, Elevation, Frame} is obtained. Finally, the distance R, azimuth angle θ and elevation angle are combined to obtain the effective elevation angle. The Cartesian coordinates of the reflection point can be expressed as: The same process is performed on each range bin to generate the entire point cloud.

[0024] In the extraction of breathing features, the frame is used as the sampling interval, and the phase information of each valid reflection point in the 2D-Angle diagram is extracted to obtain the phase waveform of each point cloud; the breathing signal is obtained by unwrapping and bandpass filtering the phase waveform, and then the spectrum of the breathing signal is analyzed using FFT, and its peak value is taken as the breathing feature of the point cloud; the breathing feature of the point cloud generated by a single measurement is normalized to obtain the relative breathing feature.

[0025] The breathing feature here is used to combine with the point cloud to estimate the breathing feature of the sleeping posture, which is a positive correlation representation of the breathing amplitude. Areas that are significantly affected by breathing behavior, such as the torso, have a larger breathing amplitude, and the corresponding breathing feature is also a larger value. However, reflection points such as the legs and the bed are not affected by breathing, and the phase of the reflection signal will not fluctuate, and the corresponding breathing feature is also a value close to zero.

[0026] When generating point cloud features, 2D-Angle-FFT is used to extract the arrival angle of the reflection point. After determining the reliable incident angle of the reflection point, the snapshot complex signal sampling value of the reflection point can be obtained from the 2D-Angle-Map generated by FFT. Then, with a sampling method of one snapshot per frame, sampling in multiple frames will obtain the complex signal sequence of the reflection point. Let the i-th reflection point P i The signal sequence is Breathing behavior affects the phase change of the signal, while individual differences, such as body shape, clothing, etc., can affect the reflected signal strength. Therefore, the signal strength value is not a reliable feature, and the present invention extracts the phase of the signal to identify fluctuations.

[0027] First, the phase unwrapping technique is used to solve the problem of periodic jumps when the phase is close to 0 or 2π, so as to ensure that the overall phase presents a smooth and continuous curve φ i .

[0028] After that, a bandpass filter is used to filter out irrelevant high-frequency and low-frequency noise to highlight the frequency range related to respiratory activity. The typical human respiratory frequency (0.1-0.6Hz) is used as the passband. i Bandpass filtering is performed on the respiratory phase curve φ i ′.

[0029] φ i The absolute value of the difference between the peak and valley values ​​of ′Δφ i ′ can reflect the small displacement ΔR caused by human breathing, where Compared with static target reflection points, the phase of reflection points affected by breathing has obvious fluctuations. Therefore, through this obvious difference, the breathing feature can be used to distinguish the point cloud reflected by a part of the human body from the static reflection points in the environment. This helps the subsequent recognition model to quickly find the position of the human body and give it more judgment weight.

[0030] Then, the spectrum of the respiratory signal is analyzed using FFT, and the maximum value of the spectrum peak is taken to represent the respiratory intensity and used as the respiratory feature. The spectrum peak is a comprehensive representation of the respiratory intensity. It filters out the influence of weak frequency components on the waveform amplitude and is more reliable than time domain features.

[0031] In order to overcome individual differences, the present invention proposes the concept of using relative breathing features. Using the reflection point number in the point cloud as an index, the breathing feature intensities of these points are combined into a vector. In the vector, the present invention normalizes the breathing feature of a single measurement value and divides it by the maximum breathing intensity of all points in the point cloud to obtain the relative breathing feature.

[0032] Steps 1-3 describe the use of point cloud features and breathing features obtained after feature extraction to identify the user's sleeping posture, as follows: first, a lightweight deep neural network is used to further extract the spatial distribution information of the point cloud containing the breathing features, and two layers of spatial convolution are performed to initially extract features for sleeping posture recognition; then, a weighted average algorithm is used through the Transformer encoder to perform deep feature fusion to obtain the breathing frequency information of multiple reflection points, and finally, the two-layer Flatten flattened features are used to obtain the sleeping posture recognition result, and the final sleeping posture classification result and the corresponding breathing frequency estimation value are output.

[0033] In the method for estimating the respiratory frequency described in steps 1-4, the key to accurately estimating the respiratory frequency of the human body is to find a reflection point that is strongly correlated with breathing. The present invention uses the continuity of the phase value as an important indicator of whether the reflection point is reliable. A method based on the point density and phase value range on the IQ circle is proposed to extract effective reflection point signals. The method is as follows:

[0034] Step 1-4-1, a complex signal sampling point of the present invention is Ae jα , the amplitude A is the modulus of the vector, the phase α is the angle between the vector and the real part I axis, and each sampling point is divided by its modulus to obtain the unit IQ circle information of the sampling signal; suppose the frame sampling signal of the reflection point is Where N frame is the total number of frames, is the signal sampling value of the reflection point of the i-th frame, then the phase signal of unit amplitude is Right now At this time, all coordinate points of the p signal in the IQ domain fall on the unit circle with an amplitude of 1.

[0035] Step 1-4-2, obtain the reliable threshold T under different postures. The Rayleigh index is used to determine whether the observation points are evenly distributed on a circle or ring. The calculation formula of the Rayleigh index is:

[0036]

[0037] Where m is the total number of points, is the unit vector of each point, given by the point's polar coordinate angle α i From formula (2), we can see that the value range of the Rayleigh index is between 0 and 1. A Rayleigh index closer to 0 indicates that the sampling points are more evenly distributed on the circumference, while a Rayleigh index closer to 1 indicates an uneven distribution.

[0038] Step 1-4-3, calculate the Rayleigh index of the sampling signal y of each reflection point. When the index value is less than the threshold T, it is considered to be an effective reflection point affected by breathing behavior, and the interference of noise on the reflection point signal can be ignored.

[0039] Step 1-4-4, phase unwrapping and bandpass filtering are performed on the phase sequence of each reflection point to obtain a signal in the respiratory frequency band; finally, FFT is used to perform spectrum analysis on the signal, and the frequency index corresponding to the maximum peak of the FFT result is taken as the respiratory frequency estimate of the reflection point. In a sleeping posture, there are often multiple reflection points affected by breathing that meet the threshold T, so the present invention uses an exponential function for weighted average calculation. The calculation formula is expressed as follows:

[0040]

[0041] Where f is the final respiratory rate estimate, n p is the total number of effective reflection points, Rayleigh i is the Rayleigh index value of the i-th reflection point, f i is the respiratory frequency estimation result of the i-th reflection point. Finally, f is used as the respiratory frequency estimation.

[0042] The advantages of the present invention are: (1) the present invention aims to use low-cost, easy-to-deploy commercial millimeter-wave radar equipment to achieve an accurate and reliable vital signs monitoring system that can stably identify six sleeping postures and accurately measure respiratory rate;

[0043] (2) The present invention proposes a point cloud purification method based on multi-frame azimuth occurrence probability clustering analysis to obtain rich and stable point cloud information;

[0044] (3) The present invention obtains the breathing characteristics of the point cloud by extracting the phase change of each reflection point, thereby enhancing the characterization of the sleeping posture by the point cloud;

[0045] (4) The present invention proposes a determination method based on unit IQ circumferential point density to extract reliable reflection points, and uses these reflection points to estimate the respiratory frequency through a weighted average method based on an exponential function, thereby obtaining a more accurate respiratory frequency estimation result. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is a schematic diagram of the deployment scenario;

[0047] Figure 2 It is the system module diagram;

[0048] Figure 3 This is an example diagram of the workflow of sleeping posture recognition;

[0049] Figure 4 This is a diagram of 6 sleeping positions. DETAILED DESCRIPTION

[0050] The present invention is based on a scenario equipped with a millimeter wave radar. Figure 1 It is a schematic diagram of the deployment scenario of the present invention.

[0051] The deployment process of the system is simple. The system includes a millimeter-wave radar module, a data preprocessing module, and a sleep sign recognition module. Figure 2 The specific steps are as follows:

[0052] (1) Sleeping posture recognition method

[0053] Figure 3 Schematic diagram of the working process of the sleeping posture recognition method.

[0054] Step 1-1, data acquisition. First, the radar acquisition parameters need to be preset, including the selection of enabled transceiver antenna parameters, the bandwidth of the transmitted FM wave, the FM time, the number of chirps, the frame length, the ADC sampling frequency, the number of sampling points, etc. The millimeter wave radar transmits the signal according to the preset parameters, and the receiving antenna captures the reflected signal. The radar RF front end amplifies, mixes, filters, and processes the received signal, obtains the intermediate frequency digital signal after sampling, and sends the sampling result to the back-end computing device via Gigabit Ethernet.

[0055] Step 1-2, feature extraction. This part is to extract features related to sleeping posture and breathing rate from the collected data. It is mainly divided into two parts, point cloud feature generation and breathing feature extraction.

[0056] The present invention proposes an angle purification algorithm based on the probability clustering of multi-frame azimuth angles. For each range bin, multi-frame cumulative clustering is performed on the azimuth angle with a certain error range as the radius, so that a more reliable azimuth angle estimation can be obtained. Finally, the corresponding altitude angle is estimated at the selected azimuth angle, and the point cloud in the Cartesian coordinate system is further generated by combining the reflection distance calculation.

[0057] The system uses frames as sampling intervals to extract the phase information of each valid reflection point in the 2D-Angle diagram, thereby obtaining the phase waveform of each point cloud. The breathing signal is obtained by unwrapping and bandpass filtering the phase waveform. Then, the system uses FFT to analyze the spectrum of the breathing signal and takes its peak value as the breathing feature of the point cloud. In order to eliminate individual differences, the system normalizes the breathing features of the point cloud generated by a single measurement to obtain the relative breathing features.

[0058] In the point cloud feature generation, the probability clustering analysis of multi-frame azimuth angles is applied. For each distance bin, the azimuth angles are clustered with a certain error range as the radius to obtain the azimuth angle estimation. Finally, the corresponding altitude angle is estimated under the selected azimuth angle, and the point cloud in the Cartesian coordinate system is generated by combining the reflection distance calculation.

[0059] Step 1-2-1, after receiving the data returned by the radar, the back-end computing device converts its format into {ADC, Chirp, Horizontal, Vertical, Frame}, where ADC is the sampling result of the intermediate frequency digital signal, Chirp represents the snapshot dimension, Horizontal represents the horizontal dimension of the two-dimensional virtual antenna array, Vertical represents the height dimension, and Frame represents the frame dimension;

[0060] Step 1-2-2, use Range-FFT to estimate the frequency components of the intermediate frequency signal, further calculate the distance information, and convert the data format into {Range, Chirp, Horizontal, Vertical, Frame}, where Range represents the distance from the radar to the target; then, extract the signal data within the effective sensing range through the preset scenario;

[0061] Step 1-2-3, use Doppler-FFT to estimate the speed information and convert the data format into {Range, Doppler, Horizontal, Vertical, Frame}, where Doppler represents the speed of the target. When a person is sleeping, the body is in a relatively static state, and almost only the upper body moves periodically with breathing. This slow movement speed is very low, less than the speed range represented by a Doppler bin, that is, it is included in the zero Doppler bin. Therefore, by extracting the data of the zero Doppler bin, the moving targets within the perception range can be filtered out, and at the same time, the signal reflected by the human body in the sleeping state can be retained.

[0062] Step 1-2-4, use Angle-FFT to estimate the reflection angle. After performing Angle-FFT in the azimuth antenna dimension, the data format becomes {Range, Azimuth, Vertical, Frame}, where Azimuth represents the azimuth angle and Vertical represents the altitude angle. The range-azimuth angle estimation result can be expressed as:

[0063]

[0064] In the formula, y r,θ represents the reflected signal of the target at a distance of r and an angle of θ, A is the amplitude, j is the imaginary unit, and f c is the starting frequency of the chirp, c is the speed of light, dr is the distance from the reflector to the reference element, Δd θ is the path difference due to θ, and R is the perceived distance. Then, the actual incident angle of the reflected signal at r is estimated, and the angular spectrum of r is ρ = |y r |Perform peak search and obtain n peaks where θ 1 and θ 2 Respectively represent the angle values ​​corresponding to the first peak and the second peak. This paper adopts a threshold method based on signal-to-noise ratio to extract reliable peaks, and sets an effective azimuth extraction range to filter out some background static target interference.

[0065] Step 1-2-5, purify the azimuth using an algorithm based on the probability clustering of multi-frame azimuth angles. The angle estimated from a single snapshot is easily affected by random noise and multipath reflection interference and produces deviations, which will cause the subsequently generated point cloud features to become unreliable. In order to suppress this interference, the present invention designs an algorithm based on the probability clustering of multi-frame azimuth angles. The basic idea of ​​the algorithm is that in the estimation results of different frames, the arrival angle of the real target will appear more times, and the angle will only fluctuate within a small range. The angle estimation results of false targets caused by noise interference, multipath reflection, etc., have large numerical variations between different frames. Moreover, estimating the breathing characteristics of the human body requires multiple frames of data, so the angle purification algorithm improves the utilization rate of frame data. The process of the purification algorithm based on the probability of multi-frame azimuth angles is as follows:

[0066] Let the total number of frames be N frame , perform Angle-FFT on the snapshot of each range bin under the zero Doppler bin, and perform the same operation on each frame to obtain the azimuth of all frames First, count θ r The probability of each angle appearing in , and arrange them in descending order. Then, the angle with the largest probability is used as the cluster center and a new category C is assigned to it 1 , and use the preset noise interference tolerance as the cluster radius γ, and then traverse the remaining angles. If the difference Δθ between the currently traversed angle and the cluster center is less than the cluster radius γ, the angle is considered to belong to the cluster and its category is changed to C 1 After traversing all the remaining angles, the first round of clustering is completed and the category is C 1 The angle from θ r Then, the angle with the highest probability in the current angle set is used as the new cluster center, and the category cluster is set to C 2 , a new round of clustering is performed, and after traversing the remaining angles, the 2All angles. Repeat this process until the angle pool is empty. At this point, all estimated azimuth angles have been purified into several categories. The more angles a category contains, the more reliable the angles contained in the category. Sort the categories according to the number of angles they contain, and finally, take the central angles of the categories containing the most angles as the extraction results. In order to ensure that the number of points contained in the point cloud input into the deep neural network is the same, if the number of categories estimated by the current sampled signal is not enough, copy the category with the most angles to fill it. Then, perform Angle-FFT in the vertical antenna dimension to estimate the elevation angle Specifically, first extract the Angle-FFT result corresponding to each valid azimuth angle, and then use the peak search algorithm based on the signal-to-noise ratio threshold to estimate the effective elevation angle. As with the azimuth angle, the effective elevation angle is still extracted by limiting the pre-known perception range to filter out irrelevant object reflections. At this point, the data of {Range, Azimuth, Elevation, Frame} is obtained. Finally, the distance R, azimuth angle θ and elevation angle are combined to obtain the effective elevation angle. The Cartesian coordinates of the reflection point can be expressed as: The same process is performed on each range bin to generate the entire point cloud.

[0067] In the extraction of breathing features, the frame is used as the sampling interval, and the phase information of each valid reflection point in the 2D-Angle diagram is extracted to obtain the phase waveform of each point cloud; the breathing signal is obtained by unwrapping and bandpass filtering the phase waveform, and then the spectrum of the breathing signal is analyzed using FFT, and its peak value is taken as the breathing feature of the point cloud; the breathing feature of the point cloud generated by a single measurement is normalized to obtain the relative breathing feature.

[0068] The breathing feature here is used to combine with the point cloud to estimate the breathing feature of the sleeping posture, which is a positive correlation representation of the breathing amplitude. Areas that are significantly affected by breathing behavior, such as the torso, have a larger breathing amplitude, and the corresponding breathing feature is also a larger value. However, reflection points such as the legs and the bed are not affected by breathing, and the phase of the reflection signal will not fluctuate, and the corresponding breathing feature is also a value close to zero.

[0069] When generating point cloud features, 2D-Angle-FFT is used to extract the arrival angle of the reflection point. After determining the reliable incident angle of the reflection point, the snapshot complex signal sampling value of the reflection point can be obtained from the 2D-Angle-Map generated by FFT. Then, with a sampling method of one snapshot per frame, sampling in multiple frames will obtain the complex signal sequence of the reflection point. Let the i-th reflection point P i The signal sequence is Breathing behavior affects the phase change of the signal, while individual differences, such as body shape, clothing, etc., can affect the reflected signal strength. Therefore, the signal strength value is not a reliable feature, and the present invention extracts the phase of the signal to identify fluctuations.

[0070] First, the phase unwrapping technique is used to solve the problem of periodic jumps when the phase is close to 0 or 2π, so as to ensure that the overall phase presents a smooth and continuous curve φ i .

[0071] After that, a bandpass filter is used to filter out irrelevant high-frequency and low-frequency noise to highlight the frequency range related to respiratory activity. The typical human respiratory frequency (0.1-0.6Hz) is used as the passband. i Bandpass filtering is performed on the respiratory phase curve φ i ′.

[0072] φ i The absolute value of the difference between the peak and valley values ​​of ′Δφ i ′ can reflect the small displacement ΔR caused by human breathing, where Compared with static target reflection points, the phase of reflection points affected by breathing has obvious fluctuations. Therefore, through this obvious difference, the breathing feature can be used to distinguish the point cloud reflected by a part of the human body from the static reflection points in the environment. This helps the subsequent recognition model to quickly find the position of the human body and give it more judgment weight.

[0073] Then, the spectrum of the respiratory signal is analyzed using FFT, and the maximum value of the spectrum peak is taken to represent the respiratory intensity and used as the respiratory feature. The spectrum peak is a comprehensive representation of the respiratory intensity. It filters out the influence of weak frequency components on the waveform amplitude and is more reliable than time domain features.

[0074] In order to overcome individual differences, the present invention proposes the concept of using relative breathing features. Using the reflection point number in the point cloud as an index, the breathing feature intensities of these points are combined into a vector. In the vector, the present invention normalizes the breathing feature of a single measurement value and divides it by the maximum breathing intensity of all points in the point cloud to obtain the relative breathing feature.

[0075] Step 1-3, sleep posture recognition. The purpose of this module is to use the point cloud containing breathing features obtained after signal processing to identify the user's sleeping posture. The system uses a designed lightweight deep neural network to further extract the spatial distribution information of the point cloud containing breathing features. The network first performs two layers of spatial convolution to initially extract features, then performs deep feature fusion through the Transformer encoder, and finally obtains the sleep posture recognition results through two layers of Flatten flattening features. There are 6 results in total, as shown below: Figure 4 shown.

[0076] (2) Respiratory rate estimation method.

[0077] Step 1-4, respiratory rate estimation. The present invention proposes a determination method based on unit IQ circumference point density, extracts effective reflection points from point cloud data using specific threshold parameters according to different sleeping postures, and uses these reflection points to accurately estimate the respiratory rate through a weighted average method based on an exponential function.

[0078] The present invention uses the continuity of phase value as an important indicator of whether the reflection point is reliable. A method based on point density and phase value range on the IQ circle is proposed to extract effective reflection point signals. The method is as follows:

[0079] Step 1-4-1, a complex signal sampling point of the present invention is Ae jα , the amplitude A is the modulus of the vector, the phase α is the angle between the vector and the real part I axis, and each sampling point is divided by its modulus to obtain the unit IQ circle information of the sampling signal; suppose the frame sampling signal of the reflection point is Where N frame is the total number of frames, is the signal sampling value of the reflection point of the i-th frame, then the phase signal of unit amplitude is Right now At this time, all coordinate points of the p signal in the IQ domain fall on the unit circle with an amplitude of 1.

[0080] The present invention compares the performance of three unit amplitude phase signals in the IQ domain, namely: the signal of the reflection point affected by breathing, the signal of the reflection point affected by breathing but also affected by large noise, and the signal of the static reflection point. When the signal of the reflection point is a signal affected by breathing, its phase value is theoretically continuously distributed, that is, it is relatively evenly distributed on the circumference of the unit circle, and the corresponding time domain waveform is also uniform. Due to the phase disturbance caused by noise interference, the distribution of the sampling points of the signal on the IQ circle will become scattered, and the time domain waveform of the phase value will have some jumps, resulting in tight peaks and valleys. If it is a completely static reflection point signal, its phase will not change evenly, the phase value will only fluctuate in a small range, and the phase time domain waveform will also change in a small range. Based on the above facts, the present invention transforms the problem of judging the continuity of the phase into the problem of judging whether the sampling points on the circumference of the unit circle in the IQ domain are evenly distributed.

[0081] Step 1-4-2, obtain the reliable threshold T under different postures. The Rayleigh index is used to determine whether the observation points are evenly distributed on a circle or ring. The calculation formula of the Rayleigh index is:

[0082]

[0083] Where m is the total number of points, is the unit vector of each point, given by the point's polar coordinate angle α i From formula (2), we can see that the value range of the Rayleigh index is between 0 and 1. A Rayleigh index closer to 0 indicates that the sampling points are more evenly distributed on the circumference, while a Rayleigh index closer to 1 indicates an uneven distribution.

[0084] However, due to the different reflection surfaces of the human body relative to the radar in different sleeping postures, the uniformity of the phase point varies to a certain extent. Therefore, the present invention obtains a threshold T of the Rayleigh index for each sleeping posture to ensure the reliability of subsequent respiratory frequency calculation.

[0085] The present invention collects 10 sets of radar sampling data from multiple volunteers in each posture, and records the actual breathing frequency corresponding to each set of data. For each posture sample, manually distinguish whether each sampling point is a valid reflection point affected by breathing, and record its corresponding Rayleigh index. Whether each sampling point is a valid reflection point affected by breathing is uniformly determined by combining the time domain and frequency domain characteristics of the phase signal;

[0086] In terms of time domain features: First, the phase time domain waveform of the sampling point should be relatively smooth, and there should be obvious periodicity between adjacent peaks. Secondly, the minimum absolute value of the peak-to-valley difference of the waveform should exceed a certain threshold (for example, it should exceed 15 in the supine position and exceed 8 in the side-lying position). This is to filter out the weak signal that falls between the respiratory frequency bands due to bandpass filtering (usually the absolute value of the peak-to-valley difference is less than 3). In terms of frequency domain features: For the respiratory waveform of each sampling point, its spectrum composition is calculated by FFT, and the frequency corresponding to its maximum peak is taken as the calculated respiratory frequency. When the difference between the calculated respiratory frequency and the actual respiratory frequency is less than 0.5Hz, it is considered to meet the frequency domain feature requirements. Only reflection points that meet both the time domain feature and the frequency domain feature requirements will be considered valid, otherwise they are considered invalid.

[0087] When calculating the threshold T corresponding to each posture, the maximum value of the Rayleigh index of the valid points and the minimum value of the Rayleigh index of the invalid points in all volunteer data are selected, and the average value of the two is calculated as the threshold T under the posture.

[0088] Step 1-4-3, calculate the Rayleigh index of the sampling signal y of each reflection point. When the index value is less than the threshold T, it is considered to be an effective reflection point affected by breathing behavior, and the interference of noise on the reflection point signal can be ignored.

[0089] Step 1-4-4, phase unwrapping and bandpass filtering are performed on the phase sequence of each reflection point to obtain a signal in the respiratory frequency band; finally, FFT is used to perform spectrum analysis on the signal, and the frequency index corresponding to the maximum peak of the FFT result is taken as the respiratory frequency estimate of the reflection point. In a sleeping posture, there are often multiple reflection points affected by breathing that meet the threshold T, so the present invention uses an exponential function for weighted average calculation. The calculation formula is expressed as follows:

[0090]

[0091] Where f is the final respiratory rate estimate, n p is the total number of effective reflection points, Rayleigh i is the Rayleigh index value of the i-th reflection point, f i is the respiratory frequency estimation result of the i-th reflection point. Finally, f is used as the respiratory frequency estimation.

[0092] The present invention demonstrates a sleep sign monitoring system based on millimeter wave radar. The system consists of a millimeter wave radar, data preprocessing and sleep sign recognition module. The core innovation is that a point cloud purification method based on multi-frame azimuth occurrence probability clustering analysis is proposed, rich and stable point cloud information is obtained, and a determination method based on unit IQ circular point density is proposed to obtain a more accurate respiratory rate estimation result. The present invention can use millimeter wave radar to identify the user's sleep signs. The present invention has many applications for identifying sleeping posture and respiratory rate. Methods modified and improved based on the present invention also fall within the scope of protection of the present invention.

Claims

1. A sleep sign monitoring method based on millimeter wave radar, Features: The specific steps include: Step 1-1, data collection: first, the acquisition parameters of the millimeter wave radar are preset, the millimeter wave radar transmits a signal according to the preset parameters, the receiving antenna captures the reflected signal, the millimeter wave radar RF front end amplifies, mixes, and filters the received signal, obtains an intermediate frequency digital signal after sampling, and sends the sampling result to the back-end computing device; Step 1-2, feature extraction: extracting features related to sleeping posture and breathing rate from the collected intermediate frequency digital signal, namely, point cloud feature generation and breathing feature extraction; Step 1-3, sleep posture recognition: using the point cloud features and breathing features obtained after feature extraction to recognize the user's sleep posture; Step 1-4, respiratory rate estimation: extract effective reflection points from the point cloud data using specific parameters according to different sleeping postures, and use these reflection points to accurately estimate the respiratory rate through a weighted average method based on an exponential function.

2. The sleep sign monitoring method based on millimeter wave radar according to claim 1, Features: The acquisition parameters of the preset millimeter-wave radar described in step 1-1 specifically include the selectively enabled transceiver antenna parameters, the bandwidth of the transmitted FM wave, the FM time, the number of chirps, the frame length, the ADC sampling frequency and the number of sampling points.

3. The sleep sign monitoring method based on millimeter wave radar according to claim 1, Features: The steps 1-2 are as follows: extracting features related to sleeping posture and breathing rate from the collected intermediate frequency digital signal, i.e., generating point cloud features and extracting breathing features: In the generation of point cloud features, multi-frame azimuth probability clustering analysis is applied. For each distance bin, multi-frame cumulative clustering is performed on the azimuth with a certain error range as the radius to obtain the azimuth estimation. Finally, the corresponding altitude angle is estimated under the selected azimuth, and the point cloud in the Cartesian coordinate system is generated by combining the reflection distance calculation.

4. The sleep sign monitoring method based on millimeter wave radar according to claim 3, Features: In the point cloud feature generation, multi-frame azimuth occurrence probability clustering analysis is applied. For each distance bin, multi-frame cumulative clustering is performed on the azimuth with a certain error range as the radius to obtain the azimuth estimation. Finally, the corresponding altitude angle is estimated under the selected azimuth, and the point cloud in the Cartesian coordinate system is generated in combination with the reflection distance calculation. The specific steps are as follows: Step 1-2-1, after receiving the data returned by the millimeter wave radar, the back-end computing device converts its format into {ADC, Chirp, Horizontal, Vertical, Frame}, where ADC is the sampling result of the intermediate frequency digital signal, Chirp represents the snapshot dimension, Horizontal represents the horizontal dimension of the two-dimensional virtual antenna array, Vertical represents the height dimension, and Frame represents the frame dimension; Step 1-2-2, use Range-FFT to estimate the frequency components of the intermediate frequency signal, further calculate the distance information, and convert the data format into {Range, Chirp, Horizontal, Vertical, Frame}, where Range represents the distance from the radar to the target; then, extract the signal data within the effective sensing range through the preset scenario; Step 1-2-3, use Doppler-FFT to estimate the speed information and convert the data format into {Range, Doppler, Horizontal, Vertical, Frame}, where Doppler represents the speed of the target. When a person is sleeping, the body is in a relatively static state, and only the upper body is moving up and down periodically with breathing. This slow movement speed is very low, less than the speed range represented by a Doppler bin, that is, it is included in the zero Doppler bin. Therefore, the data of the zero Doppler bin is extracted to filter out the moving targets within the sensing range, while not losing the signal reflected by the human body in the sleeping state; Step 1-2-4, use Angle-FFT to estimate the reflection angle. After performing Angle-FFT in the azimuth antenna dimension, the data format becomes {Range, Azimuth, Vertical, Frame}, where Azimuth represents the azimuth angle and Vertical represents the altitude angle. The range-azimuth angle estimation result is expressed as: In the formula, y r,θ represents the reflected signal of the target at a distance of r and an angle of θ, A is the amplitude, j is the imaginary unit, and f c is the starting frequency of the chirp, c is the speed of light, d r is the distance from the reflector to the reference element, Δd θ is the path difference caused by θ, R is the perceived distance, then, the actual incident angle of the reflected signal at r is estimated, and the angular spectrum of r is ρ = |y r |Perform peak search and obtain n peaks where θ 1 and θ 2 Respectively represent the angle values ​​corresponding to the first peak and the second peak, use the threshold method based on the signal-to-noise ratio to extract reliable peaks, and set the effective azimuth extraction range to filter out some background static target interference; Step 1-2-5, use the algorithm based on the probability clustering of multi-frame azimuth angles to purify the azimuth angle. The process of the purification algorithm based on the probability of multi-frame azimuth angles is as follows: Let the total number of frames be N frame , perform Angle-FFT on the snapshot of each range bin under the zero Doppler bin, and perform the same operation on each frame to obtain the azimuth of all frames First, count θ r The probability of each angle appearing in the cluster is calculated and they are arranged in descending order. Then, the angle with the largest probability is used as the cluster center and a new category C is assigned to it. 1 , and use the preset noise interference tolerance as the cluster radius γ, and then traverse the remaining angles. If the difference Δθ between the currently traversed angle and the cluster center is less than the cluster radius γ, the angle is considered to belong to the cluster and its category is changed to C 1 After traversing all the remaining angles, the first round of clustering is completed and the category is C 1 The angle from θ r Then, the angle with the highest probability in the current angle set is used as the new cluster center, and the category cluster is set to C 2 , a new round of clustering is performed, and after traversing the remaining angles, the 2 Repeat this process for all angles until the angle pool is empty. At this point, all estimated azimuth angles have been purified into several categories. The more angles a category contains, the more reliable the angles contained in the category are. The categories are sorted according to the number of angles they contain. Finally, the central angles of the categories containing the most angles are used as the extraction results. In order to ensure that the number of points contained in the point cloud input to the deep neural network is the same, if the number of categories estimated by the current sampled signal is not enough, the category with the most angles is copied for filling. Then, Angle-FFT is performed in the vertical antenna dimension to estimate the elevation angle. Specifically, first extract the Angle-FFT result corresponding to each valid azimuth angle, then use the peak search algorithm based on the signal-to-noise ratio threshold to estimate the effective altitude angle, and extract the effective altitude angle through the predicted perception range limitation to filter out irrelevant object reflections. At this point, the data of {Range, Azimuth, Elevation, Frame} is obtained; finally, combine the distance R, azimuth angle θ and elevation angle The Cartesian coordinates of the reflection point are expressed as: Performing the same process on each range bin generates the entire point cloud; In the extraction of breathing features, the frame is used as the sampling interval, and the phase information of each valid reflection point in the 2D-Angle diagram is extracted to obtain the phase waveform of each point cloud; the breathing signal is obtained by unwrapping and bandpass filtering the phase waveform, and then the spectrum of the breathing signal is analyzed using FFT, and its peak value is taken as the breathing feature of the point cloud; the breathing feature of the point cloud generated by a single measurement is normalized to obtain the relative breathing feature.

5. The sleep sign monitoring method based on millimeter wave radar according to claim 4, Features: When generating point cloud features, 2D-Angle-FFT is used to extract the arrival angle of the reflection point. After determining the reliable incident angle of the reflection point, the snapshot complex signal sampling value of the reflection point is obtained from the 2D-Angle-Map generated by FFT. Then, with a sampling method of one snapshot per frame, sampling is performed in multiple frames to obtain the complex signal sequence of the reflection point. Suppose the i-th reflection point P i The signal sequence is Breathing behavior affects the phase change of the signal, and individual differences affect the reflected signal intensity, so the phase of the signal is extracted to identify fluctuations; First, the phase unwrapping technique is used to solve the problem of periodic jumps when the phase is close to 0 or 2π, so as to ensure that the overall phase presents a smooth and continuous curve φ i ; After that, a bandpass filter is used to filter out irrelevant high-frequency and low-frequency noise to highlight the frequency range related to respiratory activity, with the typical human respiratory frequency (0.1-0.6 Hz) as the passband. i Bandpass filtering is performed on the respiratory phase curve φ i ′; φ i The absolute value of the difference between the peak and valley values ​​of ′Δφ i ′ reflects the small displacement ΔR caused by human breathing, where Then, the spectrum of the respiratory signal is analyzed using FFT, and the maximum value of the spectrum peak is taken to represent the respiratory intensity and used as the respiratory feature; Using the reflection point number in the point cloud as the index, the breathing feature intensities of these points are combined into a vector, in which the breathing feature of a single measurement value is normalized and divided by the maximum breathing intensity of all points in the point cloud to obtain the relative breathing feature.

6. The sleep sign monitoring method based on millimeter wave radar according to claim 1, Features: Steps 1-3 describe the use of point cloud features and breathing features obtained after feature extraction to identify the user's sleeping posture, as follows: first, a lightweight deep neural network is used to further extract the spatial distribution information of the point cloud containing the breathing features, and two layers of spatial convolution are performed to initially extract features for sleeping posture recognition; then, a weighted average algorithm is used through the Transformer encoder to perform deep feature fusion to obtain the breathing frequency information of multiple reflection points, and finally, the two-layer Flatten flattened features are used to obtain the sleeping posture recognition result, and the final sleeping posture classification result and the corresponding breathing frequency estimation value are output.

7. The sleep sign monitoring method based on millimeter wave radar according to claim 1, Features: The respiratory rate estimation method described in steps 1-4 is as follows: Step 1-4-1, a complex signal sampling point is Ae jα , the amplitude A is the modulus of the vector, the phase α is the angle between the vector and the real part I axis, and each sampling point is divided by its modulus to obtain the unit IQ circle information of the sampling signal; suppose the frame sampling signal of the reflection point is Where N frame is the total number of frames, is the signal sampling value of the reflection point of the i-th frame, then the phase signal of unit amplitude is Right now At this time, the coordinate points of the p signal in the IQ domain all fall on the unit circle with an amplitude of 1; Step 1-4-2, obtain the reliable threshold T under different postures. The Rayleigh index is used to determine whether the observation points are evenly distributed on a circle or ring. The calculation formula of the Rayleigh index is: Where m is the total number of points, is the unit vector of each point, given by the point's polar coordinate angle α i From formula (2), we know that the value range of the Rayleigh index is between 0 and 1. A Rayleigh index closer to 0 indicates that the sampling points are more evenly distributed on the circumference, while a Rayleigh index closer to 1 indicates an uneven distribution. Due to the different reflection surfaces of the human body relative to the radar in different sleeping postures, the uniformity of the phase points varies to a certain extent. Therefore, a threshold T of the Rayleigh index is obtained for each sleeping posture to ensure the reliability of the subsequent respiratory rate calculation. Step 1-4-3, calculate the Rayleigh index of the sampling signal y of each reflection point. When the index value is less than the threshold T, it is considered to be an effective reflection point affected by breathing behavior, and the interference of noise on the reflection point signal can be ignored. Step 1-4-4, phase unwrapping and bandpass filtering are performed on the phase sequence of each reflection point to obtain the signal in the respiratory frequency band; finally, FFT is used to perform spectrum analysis on the signal, and the frequency index corresponding to the maximum peak of the FFT result is taken as the respiratory frequency estimation of the reflection point; in a sleeping posture, there are multiple reflection points affected by breathing that meet the threshold T, so an exponential function is used for weighted average calculation, and the calculation formula is expressed as follows: Where f is the final respiratory rate estimate, n p is the total number of effective reflection points, Rayleigh i is the Rayleigh index value of the i-th reflection point, f i is the respiratory frequency estimation result of the i-th reflection point, and finally f is used as the respiratory frequency estimation.

Images