Method for sleep apnea type discrimination based on millimeter wave radar

Abstract

The application discloses a sleep apnea type distinguishing method and device based on a millimeter wave radar and a computer readable storage medium, and the method comprises the following steps: acquiring a millimeter wave radar echo signal of a monitoring object in a monitoring area; extracting a body movement signal feature, a chest cavity reflection signal energy feature and a breathing signal amplitude feature from the echo signal; constructing a time sequence feature sequence based on the extracted body movement signal feature, the chest cavity reflection signal energy feature and the breathing signal amplitude feature; inputting the feature sequence into a pre-trained classification model based on a self-attention mechanism to perform dynamic correlation analysis, and determining a sleep apnea type of the monitoring object. The application has the advantages of non-contact and accurate sleep apnea type distinguishing.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of millimeter wave radar, in particular to a sleep apnea type distinguishing method and device based on millimeter wave radar and a computer readable storage medium. BACKGROUND

[0002] Sleep apnea syndrome (SAS) is a disease characterized by repeated apnea at night, which can be divided into obstructive sleep apnea (OSA), central sleep apnea (CSA) and mixed sleep apnea (MSA).

[0003] Currently, in the field of diagnosis of sleep apnea syndrome, the main technical solutions include contact type polysomnography (PSG) and non-contact monitoring technology. Among them, PSG is the "gold standard" of clinical diagnosis, which records physiological signals such as electroencephalogram, electrocardiogram, respiratory airflow and blood oxygen saturation by wearing multiple sensors on the human body to evaluate the disease. As an alternative, non-contact technology based on millimeter wave radar has also appeared, which remotely monitors the body surface micro-movement caused by breathing and heartbeat by emitting electromagnetic waves to the human body and analyzing the echo signals.

[0004] However, the above-mentioned existing technical solutions all have their own defects and deficiencies. For polysomnography (PSG), the device is usually very complex and costly, and multiple electrode patches or sensors need to be pasted on the user's body, which not only is cumbersome to operate, but also seriously interferes with the user's normal sleep, affecting the naturalness and authenticity of the monitoring data. As for the existing millimeter wave radar technology, its function is relatively limited, most of the solutions can only realize the identification of apnea events, that is, to judge "yes" or "no" apnea, but cannot effectively distinguish different types of apnea such as obstructive (OSA), central (CSA) and mixed (MSA), and some technologies can only identify a single type. Since different types of apnea have different causes and treatment methods, the lack of classification ability limits its clinical guidance value.

[0005] Therefore, there is an urgent need in the industry for a new sleep apnea monitoring technology that not only has the advantages of millimeter wave radar non-contact and non-disturbance, suitable for long-term, comfortable home environment monitoring, but also breaks through the limitations of existing radar technology, realizes accurate classification of different types of sleep apnea, and provides more targeted health warnings and medical references for users. SUMMARY

[0006] The embodiment of the present application provides a sleep apnea type distinguishing method based on a millimeter wave radar, which aims to accurately distinguish the type of an apnea event in a sleep process in a non-contact manner.

[0007] To achieve the above object, the embodiment of the present application provides a sleep apnea type distinguishing method based on a millimeter wave radar, which comprises the following steps:

[0008] Obtaining a millimeter wave radar echo signal of a monitoring object in a monitoring area;

[0009] Extracting the following physiological features from the echo signal:

[0010] A body motion signal feature representing a change in a whole trunk motion state of the monitoring object, a thoracic cavity reflection signal energy feature representing a change in a thoracic cavity motion energy of the monitoring object, and a respiration signal amplitude feature representing a change in a respiration airflow of the monitoring object;

[0011] Constructing a feature sequence with time sequence based on the extracted body motion signal feature, thoracic cavity reflection signal energy feature and respiration signal amplitude feature;

[0012] Inputting the feature sequence into a pre-trained classification model based on a self-attention mechanism to perform dynamic correlation analysis, and determining the sleep apnea type of the monitoring object.

[0013] In an embodiment, before extracting the three types of physiological features from the echo signal, the method further comprises:

[0014] Adopting an adaptive wavelet to perform denoising processing on the echo signal;

[0015] Performing fast Fourier transform processing on the denoised echo signal to obtain a range profile;

[0016] Adopting a constant false alarm rate detection to process the range profile to locate a target range unit corresponding to a thoracic cavity region of the monitoring object;

[0017] Performing unwrapping and difference processing on phase information of the target range unit to extract a micro-motion signal containing the physiological features.

[0018] In an embodiment, the body motion signal feature representing a change in a whole trunk motion state of the monitoring object comprises:

[0019] Based on a frequency domain representation of the echo signal, calculating an energy proportion of a frequency band to which the body motion signal belongs in total signal energy and a frequency range of the frequency band to which the body motion signal belongs, and taking the energy proportion of the body motion signal and the frequency range as the body motion signal feature.

[0020] In an embodiment, the thoracic cavity reflection signal energy feature representing energy change of the thoracic cavity movement of the monitoring subject is extracted, including:

[0021] The mean value, the standard deviation and the kurtosis of the energy value sequence corresponding to the thoracic cavity region of the monitoring subject in the echo signal within a preset time window are calculated, which are collectively used as the thoracic cavity reflection signal energy feature.

[0022] In an embodiment, the respiratory signal amplitude feature representing respiratory airflow change of the monitoring subject is extracted, including:

[0023] The echo signal is subjected to band-pass filtering processing to obtain a respiratory signal waveform;

[0024] The maximum amplitude, the minimum amplitude and the peak number of the respiratory signal waveform within a preset time window are extracted, which are collectively used as the respiratory signal amplitude feature.

[0025] In an embodiment, a feature sequence with time sequence is constructed, including:

[0026] The body movement signal feature, the thoracic cavity reflection signal energy feature and the respiratory signal amplitude feature are spliced into a multi-dimensional feature vector;

[0027] The multi-dimensional feature vector arranged in time sequence is subjected to windowing processing with a preset sliding window to form an input sequence;

[0028] The input sequence is subjected to position encoding to represent time sequence information in the feature sequence.

[0029] In an embodiment, the pre-trained classification model based on the self-attention mechanism is a Transformer network model;

[0030] The feature sequence is analyzed based on the Transformer network model, including:

[0031] In a multi-head self-attention layer, attention weights are calculated based on the feature sequence to capture dynamic correlations between features in the sequence;

[0032] In a feedforward network layer, the output of the multi-head self-attention layer is subjected to nonlinear transformation;

[0033] In a classification layer, the Softmax function is used to normalize the result processed by the feedforward network layer to output probability values corresponding to different sleep apnea types.

[0034] In an embodiment, after determining the sleep apnea type of the monitoring subject, the method further includes:

[0035] When it is determined that the apnea duration of the monitoring object exceeds a preset time threshold, a preset alarm event is executed.

[0036] To achieve the above object, the embodiment of the present application further proposes a sleep apnea type distinguishing device based on millimeter wave radar, comprising a memory, a processor, and a sleep apnea type distinguishing program based on millimeter wave radar stored on the memory and executable on the processor, wherein the processor implements the sleep apnea type distinguishing method based on millimeter wave radar as described in any of the above embodiments when executing the sleep apnea type distinguishing program based on millimeter wave radar.

[0037] To achieve the above object, the embodiment of the present application further proposes a computer readable storage medium, wherein the computer readable storage medium stores a sleep apnea type distinguishing program based on millimeter wave radar, and the sleep apnea type distinguishing method based on millimeter wave radar as described in any of the above embodiments is implemented when the sleep apnea type distinguishing program based on millimeter wave radar is executed by a processor.

[0038] In summary of the above embodiments, the sleep apnea type distinguishing method based on millimeter wave radar proposed in the present application has the following beneficial effects:

[0039] 1. High-precision sleep apnea type distinguishing is achieved, greatly improving the diagnostic value:

[0040] The technical solution of the present application extracts three types of complementary information, including body movement signal features, chest cavity reflection signal energy features, and respiratory signal amplitude features, to construct a feature sequence, and combines a classification model based on a self-attention mechanism to analyze the feature sequence. Thus, obstructive sleep apnea (OSA), central sleep apnea (CSA), and mixed sleep apnea (MSA) can be accurately distinguished.

[0041] 2. Completely non-contact comfortable monitoring is achieved, which is particularly suitable for long-term home monitoring:

[0042] The technical solution of the present application uses millimeter wave radar to obtain signals, and the entire process does not require any sensors or electrode patches to be worn or attached to the monitoring object, thereby fundamentally solving the pain points of traditional polysomnography (PSG) devices, such as complexity, sleep disturbance, and poor user compliance. This non-invasive and non-disturbing monitoring method is particularly suitable for children who are sensitive to touch or elderly people who need long-term and normal monitoring.

[0043] 3. The monitoring system has strong robustness and reliable data basis:

[0044] The technical scheme of the application contains a complete and rigorous signal preprocessing procedure before extracting physiological characteristics. The noise is filtered out through adaptive wavelet denoising, and the chest position of the monitoring object is automatically and accurately locked through fast Fourier transform combined with constant false alarm rate detection, and then the micro-motion signal with high signal-to-noise ratio is extracted through phase unwrapping and difference, which ensures that the data source of subsequent feature extraction and model analysis is pure and reliable, thereby improving the stability and adaptability to environmental changes of the whole system.

[0045] 4. The alarm function is intelligent and highly practical.

[0046] The alarm function of the technical scheme of the application is not simply triggered, but the duration of apnea is judged. Only when the duration of apnea exceeds a preset threshold with clinical significance, the system will execute the preset alarm event. This design can effectively filter out short and harmless respiratory pauses, avoid unnecessary false alarms to disturb the user or guardian, and make the early warning function more accurate, reliable and humanized. BRIEF DESCRIPTION OF DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description only show some embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor based on the drawings shown.

[0048] Figure 1 A module structure diagram of an embodiment of the present application based on a millimeter wave radar sleep apnea type distinguishing device; Figure 2 A flowchart of an embodiment of the present application based on a millimeter wave radar sleep apnea type distinguishing method.

[0049] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the drawings. DETAILED DESCRIPTION

[0050] It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0051] In order to better understand the above technical solutions, the exemplary embodiments of the present disclosure will be described in more detail below with reference to the drawings. Although the exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0052] It should be noted that in the claims, any reference signs placed between brackets shall not be construed as limiting the claims. The presence of "comprising" in the text does not exclude the presence of components or steps not listed in the claims. The quantifier "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The present invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim that lists several means, several of these means may be embodied by the same hardware item. The use of "first", "second", and "third" etc. does not indicate any order and these words may be interpreted as names.

[0053] like Figure 1 As shown, Figure 1 It is a structural diagram of a server 1 (also called a sleep apnea type differentiation device based on millimeter-wave radar) in the hardware operating environment involved in an embodiment of the present invention.

[0054] The server of the embodiment of the present invention is a device with display function such as "Internet of Things devices", smart air conditioners, smart lights, smart power supplies with networking functions, AR / VR devices with networking functions, smart speakers, self-driving cars, PCs, smart phones, tablet computers, e-book readers, portable computers, etc.

[0055] like Figure 1 As shown, the server 1 includes: a memory 11 , a processor 12 and a network interface 13 .

[0056] The memory 11 includes at least one type of readable storage medium, including flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 11 may be an internal storage unit of the server 1, such as the hard disk of the server 1. In other embodiments, the memory 11 may also be an external storage device of the server 1, such as a plug-in hard disk equipped on the server 1, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash memory card, etc.

[0057] Furthermore, the memory 11 may include both an internal storage unit of the server 1 and an external storage device. The memory 11 can be used not only to store application software installed on the server 1 and various data, such as the code of the millimeter-wave radar-based sleep apnea type classification program 10, but also to temporarily store data that has been output or is about to be output.

[0058] The processor 12 can be, in some embodiments, a Central Processing Unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip, for running program codes stored in the memory 11 or processing data, such as executing the sleep apnea type differentiation program 10 based on millimeter wave radar, etc.

[0059] The network interface 13 can optionally include standard wired interfaces, wireless interfaces (such as WI-FI interfaces), and is generally used to establish a communication connection between the server 1 and other electronic devices.

[0060] The network can be the Internet, a cloud network, a Wireless Fidelity (Wi-Fi) network, a Personal Area Network (PAN), a Local Area Network (LAN), and / or a Metropolitan Area Network (MAN). Various devices in the network environment can be configured to connect to the communication network according to various wired and wireless communication protocols. Examples of such wired and wireless communication protocols can include, but are not limited to, at least one of the following: Transmission Control Protocol and Internet Protocol (TCP / IP), User Datagram Protocol (UDP), HyperText Transfer Protocol (HTTP), File Transfer Protocol (FTP), ZigBee, EDGE, IEEE 802.11, Light Fidelity (Li-Fi), 802.16, IEEE 802.11s, IEEE 802.11g, multi-hop communication, Wireless Access Point (AP), device-to-device communication, cellular communication protocol, and / or Blue Tooth communication protocol, or a combination thereof.

[0061] Optionally, the server can also include a user interface, which can include a display, an input unit such as a keyboard, and optionally a standard wired interface, a wireless interface. Optionally, in some embodiments, the display can be an LED display, a liquid crystal display, a touch liquid crystal display, and an OLED (Organic Light-Emitting Diode) touch, etc. Among them, the display can also be referred to as a display screen or a display unit, which is used to display information processed in the server 1 and to display a visualized user interface.

[0062] Figure 1 Only the server 1 with components 11-13 and the sleep apnea type differentiation program 10 based on millimeter wave radar is shown, and those skilled in the art can understand that, Figure 1 The structure shown does not constitute a limitation on the server 1, and can include fewer or more components than shown, or combine certain components, or different component arrangements.

[0063] In this embodiment, the processor 12 may be configured to call the millimeter-wave radar-based sleep apnea type differentiation program stored in the memory 11 and perform the following operations:

[0064] Acquire millimeter-wave radar echo signals of monitored objects within the monitoring area;

[0065] The following physiological features are extracted from the echo signal:

[0066] Body motion signal characteristics that characterize changes in the overall trunk motion state of the monitored subject;

[0067] The energy characteristics of the chest cavity reflection signal representing the energy changes of the chest cavity motion of the monitored subject, and the amplitude characteristics of the respiratory signal representing the changes of the respiratory airflow of the monitored subject;

[0068] Constructing a time-series feature sequence based on the extracted body motion signal features, the chest cavity reflection signal energy features, and the respiratory signal amplitude features;

[0069] The feature sequence is input into a pre-trained classification model based on a self-attention mechanism for dynamic correlation analysis to determine the sleep apnea type of the monitored subject.

[0070] Based on the hardware architecture of the millimeter-wave radar-based sleep apnea classification device described above, an embodiment of the present invention's millimeter-wave radar-based sleep apnea classification method is proposed. This millimeter-wave radar-based sleep apnea classification method aims to accurately distinguish apnea events during sleep using a non-contact method.

[0071] Reference Figure 2 , Figure 2 This is an embodiment of a method for distinguishing sleep apnea types based on millimeter wave radar according to the present invention. The method for distinguishing sleep apnea types based on millimeter wave radar includes the following steps:

[0072] S10: Acquire the millimeter-wave radar echo signal of the monitored object in the monitoring area.

[0073] Specifically, a frequency-modulated continuous wave (FMCW) millimeter-wave radar can be deployed in the sleeping environment, for example, fixed to the wall directly above the headboard, 1.5-1.6 meters above the ground, with the radar aimed at the chest area of ​​the monitored subject at an angle of approximately 45°. The radar continuously samples within a set range at a preset sampling interval, obtaining raw echo signals that reflect the physiological activity of the monitored subject. Before formal monitoring begins, the system can also first determine whether the monitoring bed is occupied. For example, by clustering the radar point cloud and performing tracking algorithms, it can confirm that a target has entered the set monitoring bed position range before initiating subsequent signal processing and analysis.

[0074] S20, from the echo signal, extract the following physiological features: body motion signal feature representing the overall torso motion state change of the monitoring subject, thoracic cavity reflection signal energy feature representing the thoracic cavity motion energy change of the monitoring subject, and respiratory signal amplitude feature representing the respiratory airflow change of the monitoring subject.

[0075] For the three types of physiological features in step S20, first, the respiratory signal amplitude feature is the basis for judging whether an apnea event occurs. Regardless of the type of apnea, the commonality is the stop of respiratory airflow. Therefore, by analyzing the amplitude change of the respiratory signal (e.g., whether the amplitude approaches zero), it can be first determined whether there is an apnea event or not.

[0076] Secondly, the body motion signal feature is the key to distinguishing central sleep apnea (CSA) from obstructive sleep apnea (OSA). The cause of central sleep apnea is that the central nervous system does not issue a breathing command, resulting in complete lack of movement of respiratory-related muscles such as the diaphragm and intercostal muscles, which is manifested as the entire thoraco-abdominal and even the torso being still. While in obstructive sleep apnea, the central command is normal, and the patient is still trying to breathe, but the upper respiratory tract is blocked, so there is continuous or even more intense movement of the thoraco-abdominal. By analyzing the motion state of the overall torso, central sleep apnea with "complete stillness" and obstructive sleep apnea with "still struggling" can be effectively distinguished.

[0077] Thirdly, the thoracic cavity reflection signal energy feature is a strong evidence and refinement of obstructive sleep apnea (OSA). When obstructive sleep apnea occurs, due to airway obstruction, the monitoring subject will unconsciously increase respiratory effort to try to overcome the obstruction, which will be manifested as a dramatic and regular fluctuation in the energy of thoracic cavity motion (i.e., radar echo signal energy). By quantifying this energy change (e.g., analyzing its fluctuation amplitude or statistical distribution), obstructive sleep apnea can be more reliably identified.

[0078] That is, by combining the above three types of physiological features, a complete logical chain can be formed: first, use "respiratory amplitude" to determine whether there is an apnea, then use "body motion feature" to distinguish whether there is respiratory effort (distinguish CSA and OSA), and then use "thoracic cavity energy" to confirm the pattern of respiratory effort, thereby achieving comprehensive and accurate distinction of the three types.

[0079] In some embodiments, in order to obtain pure physiological information from the original radar echo full of noise and environmental clutter that can be used for analysis, a systematic preprocessing procedure needs to be performed on the echo signal before the three types of physiological features are formally extracted. The procedure aims to transform the original, complex echo signal into a time series signal that accurately reflects the thoracic cavity micro-motion of the monitoring subject. Specifically, the preprocessing procedure includes the following steps:

[0080] S51, the echo signal is denoised by using adaptive wavelet. Specifically, the system uses adaptive wavelet to denoise the echo signal. Circuit noise and environmental electromagnetic interference are inevitably mixed in the original radar signal, and adaptive wavelet denoising is an efficient processing method. By selecting an appropriate wavelet basis (for example, Daubechies 6, that is, db6), and decomposing the signal into different scales, the noise threshold of each layer of wavelet coefficients can be dynamically calculated according to the statistical characteristics of the signal itself, so as to effectively suppress noise while maximizing the preservation of non-stationary signal details generated by human physiological activities.

[0081] For example, when using adaptive wavelet denoising, the noise standard deviation estimate The calculation formula is:

[0082] ;

[0083] Where D1 is the first layer wavelet detail coefficient, 0.6745 is the empirical correction coefficient of Gaussian distribution, and the adaptive soft threshold T is calculated by the formula:

[0084] ;

[0085] Where N is the signal length, and the noise standard deviation calculated above can dynamically adjust the threshold T of each layer. Further, the wavelet coefficients are effectively shrunk, the effective signal is preserved, and the noise is suppressed.

[0086] S52, the denoised echo signal is processed by fast Fourier transform (FFT) to obtain a range spectrum. Specifically, the frequency-modulated continuous wave signal transmitted by the radar is reflected by the monitored object, and the round-trip delay is proportional to the distance of the monitored object. By performing FFT (for example, 128-point FFT) on the echo signal, the time-domain signal can be transformed into the distance domain to form a "distance-amplitude" spectrum , where is the distance frequency, and r is the distance unit. On this distance spectrum, different peaks correspond to reflected objects at different distances within the radar detection range.

[0087] S53, the range spectrum is processed by using constant false alarm rate (CFAR) detection to locate a target range unit corresponding to a chest cavity region of the monitoring object. Specifically, after obtaining the range spectrum, the system needs to automatically and accurately find which peak value is from the chest cavity of the monitoring object. CFAR is an adaptive detection algorithm which traverses the entire range spectrum through a sliding window. For the "to-be-detected unit" at the center of the window, the algorithm calculates the average noise level of the "reference units" on both sides, and sets a dynamic detection threshold according to the noise level. When the signal strength of the to-be-detected unit is significantly higher than the dynamic threshold, the system determines that there is a target at this position. In this way, the distance unit (bin) where the chest cavity of the monitoring object is located can be robustly locked without being disturbed by other stationary objects in the room, such as furniture.

[0088] For example, the guard unit length L = 16 and the reference unit length M = 32 (16 units on both sides) are used in the CFAR detector.

[0089] S54, the phase information of the target range unit is unwrapped and differentially processed to extract a micro-motion signal containing the physiological features. Specifically, the phase of the radar echo signal is extremely sensitive to the small displacement of the target. The millimeter-level or sub-millimeter-level movement of the chest cavity due to breathing and heartbeat will cause regular changes in the phase. However, the original phase value is wrapped in the range of -π to +π, and there is a "2π ambiguity" phenomenon. Therefore, it is necessary to first restore it to a continuously changing true phase through a phase unwrapping algorithm. Then, by calculating the difference between the continuous phases of adjacent radar frames, the static phase offset can be eliminated, and finally a time series signal reflecting only the small displacement of the chest cavity between two sampling intervals is obtained. This signal is the "micro-motion signal", which carries all the physiological information required for subsequent analysis.

[0090] For example, the formula for calculating the phase difference between two adjacent frames is:

[0091] ;

[0092] wherein, is the continuous phase.

[0093] It can be understood that through a series of continuous signal processing steps such as adaptive wavelet denoising, fast Fourier transform, constant false alarm rate detection, and phase unwrapping and differencing, the chest cavity position of the monitoring target can be automatically and robustly located from the original radar echo full of noise and clutter, and a micro-motion signal with high signal-to-noise ratio and accurately reflecting subtle physiological activities such as breathing and heartbeat can be extracted, providing a high-quality and reliable data foundation for subsequent high-precision physiological feature extraction and sleep apnea type differentiation.

[0094] In some embodiments, extracting body motion signal features that characterize changes in the overall trunk motion state of the monitored object includes: based on the frequency domain representation of the echo signal, calculating the proportion of the energy of the frequency band to which the body motion signal belongs to the total signal energy and the frequency range of the frequency band to which the body motion signal belongs, and taking the energy proportion and frequency range of the body motion signal together as the body motion signal features.

[0095] Specifically, large-scale body movements (such as rolling over or moving limbs) and steady breathing exhibit significantly different characteristics in the frequency domain of radar echo signals. Therefore, by transforming the signals into the frequency domain for analysis, these two types of movements can be effectively distinguished.

[0096] Specifically, the system first performs time-frequency analysis on the preprocessed micro-motion signal. For example, the short-time Fourier transform (STFT) technique can be used. A window function (such as a Hanning window) of a specific length (e.g., 256 points) is slid along the time axis to segment the micro-motion signal and perform a Fourier transform. This generates a time-frequency spectrum that simultaneously displays the temporal variations of the signal's frequency components. This time-frequency spectrum is the so-called "frequency domain representation of the echo signal." Like a map, it clearly depicts how energy is distributed across different frequencies at each point in time.

[0097] Once the time-spectrogram is obtained, the proportion of the energy in the frequency band to which the body motion signal belongs to the total signal energy can be calculated, and its frequency range can be determined. First, based on prior knowledge or experimental data, the frequency ranges corresponding to different physiological activities must be defined. For example, the frequency of steady breathing typically ranges from 0.1Hz to 0.8Hz, while the Doppler shift generated by large body movements falls into a relatively higher frequency band, such as 0.8Hz to 5Hz. Then, within each time window, by integrating or summing the time-spectrogram data, the total signal energy and the signal energy falling within the predefined "body motion signal frequency band" (e.g., 0.8Hz-5Hz) can be calculated. Dividing the latter by the former yields the energy proportion of the body motion signal. Furthermore, by searching for the lowest and highest frequencies with non-zero energy within the body motion signal frequency band, the actual frequency range of the body motion signal within that time window can be determined.

[0098] For example, the energy ratio of the body motion signal is extracted , frequency range The calculation formula is:

[0099] ;

[0100] ;

[0101] in They represent the starting frequency to be calculated and the starting frequency and cutoff frequency of the body motion signal to be calculated respectively; frequency domain amplitudes representing the respiratory signal; maximum and minimum frequency values in the body motion signal.

[0102] It can be understood that through time-frequency analysis methods such as short-time Fourier transform, the signal can be unfolded in time and frequency dimensions, and the frequency band belonging to large-amplitude limb or trunk movement can be accurately divided. Therefore, by calculating the energy proportion and frequency range of the specific frequency band, whether the monitored object has a whole movement such as turning over or limb twitching, which is different from the respiratory micro-movement, can be captured with high sensitivity and quantitatively.

[0103] In addition, the body motion signal feature can also be extracted by distinguishing the high frequency / high energy part and the low frequency / low energy part in the echo signal.

[0104] In some embodiments, a chest reflection signal energy feature representing the energy change of the chest movement of the monitored object is extracted, including: calculating the mean, standard deviation and kurtosis of the energy value sequence corresponding to the chest region of the monitored object in the echo signal within a preset time window, which are collectively used as the chest reflection signal energy feature.

[0105] Specifically, the system first needs to obtain a sequence representing the change of chest movement energy over time. This sequence is derived from the "target distance unit" located in the previous signal processing. By calculating the echo signal intensity or power at each sampling time in this unit, an energy value sequence within a preset time window (for example, 60 seconds) can be formed. This energy value sequence intuitively records the intensity of chest movement.

[0106] After obtaining the energy value sequence, in order to describe it from multiple dimensions, the system will calculate three key statistics of the sequence: mean, standard deviation and kurtosis. The "mean" of the energy reflects the average energy level of the chest movement within a time window. The "standard deviation" of the energy quantifies the dispersion degree of the energy sequence. A higher standard deviation means that the energy fluctuation of the chest movement is very intense, which is highly related to the struggle state during obstructive apnea. The "kurtosis" of the energy describes the peak shape of the energy value distribution. A higher kurtosis value usually means that there are some short but extremely high intensity spikes in the energy sequence, which corresponds to the sudden and explosive respiratory effort of the monitored object. These three statistics together constitute a multi-dimensional "fingerprint" of the chest movement energy pattern.

[0107] Exemplarily, in the chest reflection signal energy feature, the mean , standard deviation , and kurtosis of the energy are extracted, and the calculation formulas are as follows:

[0108] ;

[0109] ;

[0110] ;

[0111] wherein N is the number of sampling points within the window, represents the energy size of the sampling point where the time-domain waveform of the respiratory signal is located.

[0112] It can be understood that by calculating the mean, standard deviation and kurtosis of the energy sequence, the energy change characteristics of the chest movement can be comprehensively quantified from three dimensions of average level, fluctuation amplitude and distribution form, so that the normal and stable breathing and the violent and irregular respiratory effort under the blocked apnea can be effectively captured and distinguished, thereby providing rich and high-discrimination judgment information for accurately identifying the blocked sleep apnea event.

[0113] In addition, the chest reflection signal energy feature can also be extracted by calculating and analyzing the energy time sequence change of the signal segment representing the chest movement.

[0114] In some embodiments, the respiratory signal amplitude feature representing the respiratory airflow change of the monitored object is extracted, including: performing band-pass filtering processing on the echo signal to obtain a respiratory signal waveform; and extracting the maximum amplitude, the minimum amplitude and the number of peaks of the respiratory signal waveform within a preset time window, which are collectively used as the respiratory signal amplitude feature.

[0115] Specifically, the system first separates the waveform of the respiratory signal from the echo signal containing multiple information such as heartbeat, body shaking and environmental noise. This can be achieved by a frequency-selective band-pass filtering process. Considering that the respiratory frequency of normal adults in a resting state is mainly concentrated in the very low frequency band of 0.1-0.8 Hz, the system can apply a digital band-pass filter with the same passband range (0.1-0.8 Hz) to filter the micro-motion signal obtained in the previous processing. In this way, the heartbeat signal and body shaking signal with higher frequency, as well as the signal drift with lower frequency, will be effectively filtered out, thereby obtaining a relatively smooth periodic “respiratory signal waveform” that can clearly reflect each inhalation and exhalation process.

[0116] After obtaining the pure respiratory signal waveform, the system can quantify the morphology of the waveform within a preset time window (e.g., 30 seconds) to extract the respiratory signal amplitude features. This includes extracting the maximum and minimum values of the waveform amplitude within the time window, which intuitively reflect the depth of respiration, and when respiration stops, these two values will be very close to zero. At the same time, the system will also count the number of peaks (or troughs) of the respiratory waveform within the time window through a peak detection algorithm, and this "peak number" directly corresponds to the number of breaths in that time period, which is the basis for calculating the respiratory frequency. When apnea occurs, the peak number in the apnea period will be zero.

[0117] It can be understood that by band-pass filtering the radar echo signal in a specific frequency band (such as 0.1-0.8 Hz), the pure respiratory signal waveform that directly reflects the respiratory rhythm and depth can be effectively separated from the mixed micro-motion signal. Therefore, by directly measuring the maximum / minimum amplitude and peak number of the waveform, the presence or absence, speed, and strength of respiration can be intuitively and accurately quantified, providing the most basic and direct basis for the occurrence of all types of apnea events, and is a necessary prerequisite for subsequent type differentiation.

[0118] In addition, the respiratory signal amplitude features can also be obtained by filtering the pure respiratory waveform and measuring its morphology to extract.

[0119] S30, based on the extracted body motion signal features, chest cavity reflection signal energy features, and respiratory signal amplitude features, a feature sequence with time sequence is constructed.

[0120] Specifically, after extracting the body motion, chest cavity energy, and respiratory amplitude physiological features, the discrete information extracted at each independent time point needs to be integrated and constructed into a structured data format that can be understood and processed by a time series analysis model (such as Transformer). The purpose of this step is to convert instantaneous features into feature sequences with time dimension, i.e., feature sequences with time sequence.

[0121] In some embodiments, constructing a feature sequence with time sequence includes the following steps:

[0122] S31, splice the body motion signal feature, the chest cavity reflection signal energy feature and the respiratory signal amplitude feature into a multi-dimensional feature vector. Specifically, at any sampling time, the system obtains different feature values belonging to three categories, for example, the body motion feature can be 2-dimensional, the chest cavity energy feature is 3-dimensional, and the respiratory amplitude feature is also 3-dimensional. By arranging and connecting (i.e., splicing) these different source feature values in a predetermined order, a higher-dimensional, unified feature vector can be formed, for example, an 8-dimensional feature vector. This multi-dimensional feature vector comprehensively describes the comprehensive physiological state of the monitoring object at the sampling time.

[0123] S32, window processing is performed on the multi-dimensional feature vectors arranged in time sequence by using a preset sliding window to form an input sequence. Specifically, a single time point feature vector is not sufficient for time series analysis, so it is necessary to observe the continuous change in a period of time. The sliding window technique is used to achieve this purpose. The system sets a fixed time length window (for example, 60 seconds), and lets the window slide along the time axis. At each position, the window will "frame" all the continuous multi-dimensional feature vectors in this time period, and collect them into a two-dimensional matrix, which is an "input sequence". Then, the window will slide forward a small step according to a preset step (for example, 15 seconds), and "frame" the next input sequence again. This sliding method with overlap helps to ensure that the feature information at the window boundary is not missed.

[0124] S33, position encoding is performed on the input sequence to represent the time sequence information in the feature sequence. Specifically, models like Transformer do not have the ability to perceive the order of elements in the sequence, and they need to be explicitly informed of the position of each feature vector in the time window. Position encoding is used to inject such time sequence information. The system generates a unique "position encoding vector" for each position in the sequence (for example, from the 1st time point to the 60th time point), which has the same dimension as the feature vector. This position encoding vector is usually generated by periodic functions such as sine and cosine, and its value is determined only by its position in the sequence. Adding or splicing this position encoding vector with the feature vector at the corresponding position can form the final feature sequence containing rich physiological information and time sequence information, which is sent to the subsequent classification model.

[0125] Exemplarily, with a 60-second window, a sliding window in time sequence (step length 15 seconds) is generated to generate an input sequence, and a sine-cosine function is used to generate a position encoding to ensure that the time sequence of the data is not lost. The specific encoding implementation is:

[0126]

[0127] ​;

[0128] wherein pos is the position index; i represents the dimension of the encoding vector (half of the number of features); is the hidden layer dimension of the model.

[0129] It can be understood that through a series of processes such as feature splicing, sliding windowing, and position encoding, the instantaneous physiological features of different modalities and different dimensions are integrated into sequence data that can reflect the time evolution process and have a unified format, and the absolute or relative position information of each time step in the sequence is injected, thereby providing an ideal input data format for the subsequent classification model based on the self-attention mechanism, so that the model can not only analyze the physiological state at each time, but also understand and learn the dependence, sequence, and dynamic change law of the states in the time dimension to accurately distinguish different types of sleep apnea.

[0130] S40, inputting the feature sequence into a pre-trained classification model based on the self-attention mechanism to perform dynamic correlation analysis and determine the sleep apnea type of the monitored object.

[0131] In this embodiment, the pre-trained classification model is a Transformer network model, and its core advantage is that it can efficiently process long sequence data and capture long-distance dependencies between elements in the sequence. The internal processing procedure of the model is mainly completed through its core encoder structure, which can specifically include steps such as multi-head self-attention calculation, feedforward network nonlinear transformation, and final classification normalization.

[0132] Specifically, the analysis of the feature sequence based on the Transformer network model includes the following steps:

[0133] S41, in the multi-head self-attention layer, attention weights are calculated based on the feature sequence to capture the dynamic correlation between features in the sequence.

[0134] Specifically, the feature sequence is first fed into a multi-head self-attention layer in the model, in which attention weights are calculated based on the feature sequence to capture dynamic correlations among features in the sequence. The core of the self-attention mechanism is that, for each time point (i.e., each feature vector) in the sequence, it calculates the correlation strength or "attention score" between this time point and all other time points. This means that the model can autonomously learn which other time points in the sequence should be "paid more attention to" when judging the state of the current time. For example, when judging a mixed apnea event, the model can pay attention to both the central apnea features at the front end of the sequence and the obstructive apnea features at the back end of the sequence. The "multi-head" mechanism allows the model to perform multiple such attention calculations in parallel in different representation subspaces, so as to capture richer and more diverse feature correlation patterns from different perspectives.

[0135] For example, the self-attention weight calculation formula is:

[0136] ;

[0137] wherein query represents a query matrix, key represents a key matrix, T represents transposition, hidden_dim represents a matrix dimension, and value represents a value matrix.

[0138] S42, in the feedforward network layer, performing a nonlinear transformation on the output of the multi-head self-attention layer.

[0139] Specifically, the output processed by the multi-head self-attention layer is fed into the feedforward network layer for nonlinear transformation. The output of the self-attention layer is a new sequence obtained by weighted fusion of context information, wherein each position vector contains correlation information of the entire sequence. The subsequent feedforward network layer (usually a fully connected network composed of two linear layers and an activation function such as ReLU) independently performs a nonlinear mapping on each position vector in the new sequence. The purpose of this step is to increase the nonlinear representation capability of the model, so that the model can learn more complex and abstract feature combinations. Generally, a complete Transformer encoder layer is composed of a multi-head self-attention layer and a feedforward network layer (as well as a residual connection between them and layer normalization). In some implementations of the present application, the feature sequence is sequentially passed through 4 such encoder layers for layer-by-layer deep feature extraction.

[0140] S43, in the classification layer, using a Softmax function to normalize the result processed by the feedforward network layer to output probability values corresponding to different sleep apnea types respectively.

[0141] Specifically, when the feature sequence passes through all the encoder layers in turn, the final representation result thereof is sent to a separate classification layer. In the classification layer, a Softmax function is used to normalize the result to output probability values corresponding to different sleep apnea types respectively. The Softmax function can convert an arbitrary numerical vector into a probability distribution vector whose sum of all elements is 1. For example, the original output of the classification layer can be a vector containing 4 real values, corresponding to the scores of four classifications (normal, obstructive, central, and mixed). After the Softmax function processing, the 4 scores are converted into 4 probability values between 0 and 1, and their sum is 1, and the classification with the highest score is the sleep apnea type or non-sleep apnea identified.

[0142] For example, the Softmax normalization calculation formula is:

[0143] ;

[0144] wherein z is a vector, 、 represents the i, j-th element thereof.

[0145] For example, a 60x8-dimensional feature sequence is input into a Transformer model containing 4 encoder layers. In the multi-head self-attention sublayer of the first encoder layer, the model calculates that there is a high attention weight between the 10th time point and the 45th time point in the sequence. After deep processing by the 4-layer encoder, the final representation of the sequence is sent to the classification layer. The original score vector output by the classification layer can be [-1.5, 0.5, 3.0, -2.0]. After normalization by the Softmax function, the vector is converted into a probability distribution vector, for example: {normal: 0.01, obstructive: 0.08, central: 0.90, mixed: 0.01}. According to this probability distribution, the system determines that the sleep apnea type in this time window is "central" because the corresponding probability value 0.90 is the highest.

[0146] It can be understood that, due to the use of a deep processing architecture based on the multi-head self-attention mechanism and the feedforward network, and in particular due to the ability of the multi-head self-attention mechanism to learn the dependency between any two positions in the sequence from different subspaces in parallel, the model can dynamically calculate the influence weight of different physiological features in the time dimension, thereby overcoming the local dependency limitation of traditional time series models, effectively capturing complex long-distance and cross-time patterns that are crucial for sleep apnea type judgment, and greatly improving the understanding ability of the model for time series dynamic changes and the final classification accuracy.

[0147] It is worth mentioning that in other embodiments of the present application, recurrent neural networks, long short-term memory networks, convolutional neural networks, support vector machines, and other classification models can also be used.

[0148] In some embodiments, in order to improve the method provided by the present application from a simple classification system to a real-time monitoring tool with practical warning function, after determining the sleep apnea type of the monitoring object, the system further performs an intelligent alarm process, that is, when it is determined that the apnea duration of the monitoring object exceeds the preset time threshold, the preset alarm event is executed. This process aims to timely remind the respiratory events that may endanger health, while avoiding unnecessary interference to the user or guardian due to excessive sensitivity.

[0149] Specifically, the alarm process is not triggered immediately every time apnea is detected, but a judgment logic for event duration is introduced. The system maintains a timer in the background. When the classification model first outputs one of the obstructive, central, or mixed types, the timer starts counting. In each subsequent analysis window, if the classification result is still one of the three apnea types, the timer continues to accumulate; otherwise, if the classification result changes to the normal state of "normal", the timer will be immediately cleared and reset.

[0150] Only when the accumulated duration of the timer exceeds a clinically meaningful, preset time threshold, the system will finally execute the preset alarm event. This time threshold is crucial because physiologically, short (e.g. a few seconds) respiratory pauses can occur in normal sleep and are not necessarily dangerous. By setting a reasonable threshold (for example, according to medical guidelines, a respiratory stop lasting more than 10 seconds is usually defined as an apnea event, and in one specific implementation of the present application, it can be set to 20 seconds), it can be ensured that the system only alarms for those events with longer duration, which can cause a decrease in blood oxygen and have potential health risks. The so-called "execution of the preset alarm event" can include various forms, such as pushing an alarm notification to the bound smart phone APP through WiFi or Bluetooth, driving a sound and light alarm at the bedside to make a sound or flash light, or recording a serious event that needs attention in the monitoring log of the system.

[0151] It can be understood that, since the alarm step of the present application does not immediately trigger an alarm every time apnea is detected, but introduces the logic of judging the duration of the apnea event, and sets a presettable time threshold (such as 20 seconds), it can effectively filter out short breath pauses that are not clinically significant, significantly reduce the false alarm rate of the alarm, avoid unnecessary disturbance to the user or the guardian, make the alarm function more practical and reliable, and only when a longer apnea event occurs that may pose a threat to health, an effective reminder is issued.

[0152] From the above embodiments, it can be seen that the sleep apnea type distinguishing method based on millimeter wave radar proposed in the present application has the following beneficial effects:

[0153] 1. High-precision sleep apnea type distinguishing is achieved, greatly improving the diagnostic value:

[0154] The technical scheme of the present application extracts three types of complementary information, body movement signal features, chest reflection signal energy features and respiratory signal amplitude features, to construct a feature sequence, and combines a classification model based on a self-attention mechanism to analyze the feature sequence. Thus, obstructive sleep apnea (OSA), central sleep apnea (CSA) and mixed sleep apnea (MSA) can be accurately distinguished.

[0155] 2. Completely non-contact comfortable monitoring is achieved, especially suitable for long-term home monitoring:

[0156] The technical scheme of the present application uses millimeter wave radar to obtain signals, and the entire process does not require any sensors or electrode patches to be worn or attached to the monitored object, thereby fundamentally solving the pain points of traditional polysomnography (PSG) devices, which are complex, interfere with sleep and result in poor user compliance. This non-invasive and non-disturbing monitoring method is especially suitable for children who are sensitive to touch or the elderly who need long-term and normal monitoring.

[0157] 3. The monitoring system is robust and the data basis is reliable:

[0158] The technical scheme of the present application includes a complete and rigorous signal preprocessing procedure before extracting physiological features. Noise is filtered out by adaptive wavelet denoising, and the chest position of the monitored object is automatically and accurately locked by fast Fourier transform combined with constant false alarm rate detection, and then high signal-to-noise ratio micro-motion signals are extracted by phase unwrapping and differencing. This series of processing ensures that the data source for subsequent feature extraction and model analysis is pure and reliable, thereby improving the stability and adaptability to environmental changes of the entire system.

[0159] 4. The alarm function is intelligent and highly practical:

[0160] The alarm function of the technical solution of the present application is not simply triggered, but the duration of apnea is judged, and only when the duration of apnea exceeds a preset threshold value with clinical significance, the system will execute the preset alarm event. This design can effectively filter out short and harmless respiratory pauses, avoid unnecessary false alarms to disturb the user or guardian, and make the early warning function more accurate, reliable and humanized.

[0161] In addition, the embodiment of the present application also proposes a computer readable storage medium, which can be any one or any combination of a hard disk, a multimedia card, an SD card, a flash memory card, an SMC, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disc read-only memory (CD-ROM), a USB memory, etc. The computer readable storage medium includes the sleep apnea type distinguishing program 10 based on the millimeter wave radar. The specific implementation of the computer readable storage medium of the present application is substantially the same as the specific implementation of the sleep apnea type distinguishing method based on the millimeter wave radar and the server 1, and will not be repeated here.

[0162] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0163] The present application is described with reference to flowcharts and / or block diagrams of the method, device (system), and computer program product according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a device that implements the flow Figure 1 The function of one flow or multiple flows and / or blocks Figure 1 The function of one flow or multiple flows and / or blocks

[0164] These computer program instructions can also be stored in a computer readable storage medium that can guide the computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer readable storage medium produce a product including instruction devices that implement the flow Figure 1one or more processes and / or blocks Figure 1 the function specified in the one or more blocks.

[0165] These computer program instructions can also be loaded into computer or other programmable data processing devices to cause a series of operational steps to be performed on the computer or other programmable devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable devices provide steps for implementing the flow Figure 1 one or more processes and / or blocks Figure 1 the function specified in the one or more blocks.

[0166] While the preferred embodiments of the application have been described, additional variations and modifications can be made to the preferred embodiments by those skilled in the art once they learn of the basic inventive concepts. Therefore, the appended claims are intended to cover all such additional variations and modifications as fall within the scope of the application. What is claimed is:

[0167] Obviously, numerous modifications and variations of the present application are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. A method for distinguishing sleep apnea types based on millimeter wave radar, characterized in that: include: Acquire millimeter-wave radar echo signals of monitored objects within the monitoring area; Based on the frequency domain representation of the echo signal, calculating the proportion of energy of the frequency band to which the body motion signal belongs to the total signal energy and the frequency range of the frequency band to which the body motion signal belongs, and using the energy proportion and frequency range of the body motion signal together as body motion signal features representing changes in the overall trunk motion state of the monitored subject; Calculating the mean, standard deviation, and kurtosis of the energy value sequence corresponding to the chest region of the monitored subject in the echo signal within a preset time window, and using these values ​​together as energy features of the chest reflection signal representing energy changes of the chest movement of the monitored subject; performing bandpass filtering on the echo signal to obtain a respiratory signal waveform, and extracting the maximum amplitude, minimum amplitude, and number of peaks of the respiratory signal waveform within a preset time window as respiratory signal amplitude features representing changes in respiratory airflow of the monitored subject; splicing the body motion signal feature, the chest cavity reflection signal energy feature, and the respiratory signal amplitude feature into a multidimensional feature vector; Performing windowing processing on the multidimensional feature vectors arranged in chronological order using a preset sliding window to form an input sequence; Performing position encoding on the input sequence to construct a feature sequence with temporal sequence; The feature sequence is input into a pre-trained classification model based on a self-attention mechanism for dynamic correlation analysis to determine the sleep apnea type of the monitored subject.

2. The method for distinguishing sleep apnea types based on millimeter wave radar according to claim 1, wherein: Before extracting three types of physiological features from the echo signal, the method further includes: Adopting adaptive wavelet to perform denoising on the echo signal; Perform fast Fourier transform on the denoised echo signal to obtain the range spectrum; Processing the range spectrum using constant false alarm rate detection to locate a target range unit corresponding to the chest area of ​​the monitored object; The phase information of the target distance unit is unwrapped and differentially processed to extract the micro-motion signal containing the physiological characteristics.

3. The method for distinguishing sleep apnea types based on millimeter wave radar according to claim 1, wherein: The pre-trained classification model based on the self-attention mechanism is a Transformer network model; Analyzing the feature sequence based on the Transformer network model includes: In the multi-head self-attention layer, attention weights are calculated based on the feature sequence to capture the dynamic correlation between features in the sequence; In the feedforward network layer, a nonlinear transformation is performed on the output of the multi-head self-attention layer; In the classification layer, the Softmax function is used to normalize the results processed by the feedforward network layer to output probability values ​​corresponding to different sleep apnea types.

4. The method for distinguishing sleep apnea types based on millimeter wave radar according to claim 1, wherein: After determining the sleep apnea type of the monitored subject, the method further includes: When it is determined that the duration of the apnea of ​​the monitored object exceeds a preset time threshold, a preset alarm event is executed.

5. A sleep apnea type differentiation device based on millimeter wave radar, characterized in that: The invention comprises a memory, a processor, and a millimeter-wave radar-based sleep apnea type differentiation program stored in the memory and executable on the processor, wherein when the processor executes the millimeter-wave radar-based sleep apnea type differentiation program, the millimeter-wave radar-based sleep apnea type differentiation method according to any one of claims 1 to 4 is implemented.

6. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a millimeter-wave radar-based sleep apnea type differentiation program, which, when executed by a processor, implements the millimeter-wave radar-based sleep apnea type differentiation method according to any one of claims 1 to 4.

Images