Sleep monitoring method and device, storage medium and product
By combining millimeter-wave radar and deep learning models, the problems of high cost and privacy leakage of human sleep monitoring in existing technologies are solved, and efficient and accurate sleep monitoring is achieved. It is suitable for tablets, personal computers, mobile phones and other devices.
Patent Information
- Application Number
- CN202510826293.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-18
- Publication Date
- 2025-09-23
AI Technical Summary
In the existing technology, human sleep monitoring methods are costly and difficult to promote widely, and computer vision methods have problems such as privacy leakage and easily affected monitoring effects.
Millimeter-wave radar is used to obtain multiple frames of initial point cloud data within a preset time length. By removing static clutter data and noise points, point cloud data fusion is performed. A deep learning model is used to monitor sleeping posture, combined with a respiratory rate detection algorithm to achieve efficient and low-cost sleep monitoring.
It achieves efficient, low-cost and accurate sleep monitoring, can monitor sleep posture and breathing rate in real time, and reduces equipment complexity and privacy risks.
Smart Images
Figure CN120689329A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and in particular to a sleep monitoring method, device, storage medium and product. Background Art
[0002] Among related technologies, human sleep monitoring is crucial in many contexts, such as sleep hygiene, bedridden patient care, and chronic disease diagnosis. Research has shown that sleep monitoring is crucial for sleep hygiene and the diagnosis of some chronic diseases. For example, sleeping posture is a key indicator for diagnosing positional obstructive sleep apnea.
[0003] Related technologies have limited means of monitoring users' sleep. Wearable devices rely on specialized medical equipment and professionals, are costly, and difficult to widely promote. Some use computer vision to detect sleep, but this method not only has privacy issues, but is also easily affected by low light, obstacles, etc.
[0004] Therefore, how to monitor human sleep conditions efficiently and at low cost is a problem that needs to be solved urgently. Summary of the Invention
[0005] The main purpose of this application is to provide a sleep monitoring method, device, storage medium and product, aiming to solve the technical problem of how to monitor the human body's sleep status efficiently and at low cost.
[0006] To achieve the above objectives, the present application proposes a sleep monitoring method, which includes: Obtain multiple frames of initial point cloud data within a preset time period; Determining static clutter data and dynamic point cloud data for each frame of initial point cloud data; the dynamic point cloud data is obtained by removing the corresponding static clutter data from the initial point cloud data; Superimposing multiple frames of dynamic point cloud data to obtain fused point cloud data; In the fused point cloud data, noise points are removed to obtain target point cloud data; Based on the target point cloud data, the sleeping posture of the target user is monitored.
[0007] In some embodiments, before removing noise points from the fused point cloud data to obtain target point cloud data, the method further includes: Superimposing a plurality of the static clutter data to obtain a superimposed static clutter matrix; Clustering the superimposed static clutter matrix to obtain at least one clutter cluster, at least one of the clutter clusters including a noise cluster; The step of removing noise points from the fused point cloud data to obtain target point cloud data includes: In the fused point cloud data, points in the noise cluster are removed to obtain target point cloud data.
[0008] In some embodiments, determining static clutter data for each frame of initial point cloud data includes: For each frame of initial point cloud data, calculating the average value of multiple points in the initial point cloud data in the Doppler dimension; Based on the average value, a static clutter matrix is established as static clutter data of the corresponding frame, wherein the value of each element in the static clutter matrix in the Doppler dimension is the average value.
[0009] In some embodiments, obtaining multiple frames of initial point cloud data continuously within a preset time period includes: receiving raw analog-to-digital converter data via a universal asynchronous receiver-transmitter protocol; wherein the millimeter-wave radar is configured to communicate with the sleep monitoring device via the universal asynchronous receiver-transmitter protocol, and a distance fast Fourier transform function of the millimeter-wave radar is in a disabled state; Based on the original analog-to-digital converter data, a plurality of continuous frames of initial point cloud data within the preset time length are obtained.
[0010] In some embodiments, obtaining a plurality of frames of initial point cloud data within the preset time period based on the original analog-to-digital converter data includes: For each frame of the raw analog-to-digital converter data, obtain corresponding first point cloud data by decoding; Performing a distance fast Fourier transform on the first point cloud data to obtain second point cloud data; determining a distance range based on a sleeping position of the target user; performing a Doppler fast Fourier transform on the second point cloud data within the distance range to obtain third point cloud data; Calculating a Doppler sampling interval based on a preset respiratory parameter, and determining a portion of the third point cloud data that is within the Doppler sampling interval as fourth point cloud data; determining a portion of the fourth point cloud data whose energy is higher than a preset energy threshold as fifth point cloud data; Based on the antenna arrangement of the millimeter-wave radar, an angular fast Fourier transform is performed on the fifth point cloud data to obtain the initial point cloud data.
[0011] In some embodiments, the method further comprises: extracting an interest matrix from the second point cloud data; wherein a distance parameter value of the interest matrix is within a selected distance range; Executing a constant false alarm rate algorithm on the time series data to extract a respiratory time series signal; wherein the time series data is calculated by the average value of the matrix of interest in the distance dimension; Segmenting the respiratory timing signal using a Hamming window to obtain a plurality of windows; Calculating an average power density of the plurality of windows based on the power spectral density corresponding to each of the windows; The frequency corresponding to the maximum value of the power spectrum density is determined as the breathing frequency of the target user.
[0012] In some embodiments, monitoring the sleeping posture of the target user based on the target point cloud data includes: Global features are extracted by a multi-layer perception network layer of a deep learning model; wherein the multi-layer perception network layer resamples the standard point cloud data to obtain key points, groups all the key points based on the neighborhood range of each key point to obtain multiple grouping results, performs maximum pooling on each of the grouping results, extracts a first main feature, performs maximum pooling on each of the first main features, extracts a second main feature, and sequentially fuses and performs maximum pooling on the first main features and the second main features to obtain the global feature; The global features are classified through the fully connected layer of the deep learning model to obtain the sleeping posture prediction result of the target user.
[0013] In addition, to achieve the above-mentioned purpose, the present application also proposes a sleep monitoring device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the sleep monitoring method described above.
[0014] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium. A computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the sleep monitoring method described above are implemented.
[0015] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps of the sleep monitoring method described above are implemented.
[0016] One or more technical solutions proposed in this application have at least the following technical effects: By obtaining multiple frames of continuous initial point cloud data within a preset time length; by removing static clutter data, determining the static clutter data and dynamic point cloud data of each frame of initial point cloud data, and superimposing multiple frames of dynamic point cloud data to obtain fused point cloud data, the multi-frame point cloud data is subjected to feature fusion and noise removal, thereby facilitating subsequent efficient and accurate monitoring of the target user's sleeping posture based on the target point cloud data. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.
[0019] Figure 1 A flowchart of a sleep monitoring method according to an embodiment of the present application is shown; Figure 2 A firmware architecture diagram of a millimeter-wave radar provided by an exemplary embodiment of the present application is shown; Figure 3 A schematic diagram of a process for collecting data by a millimeter-wave radar in a related technology provided by an exemplary embodiment of the present application is shown; Figure 4 A schematic diagram of a process for unpacking a data packet provided by an exemplary embodiment of the present application is shown; Figure 5 A flowchart of a sleep monitoring method provided by another embodiment of the present application is shown; Figure 6 A schematic structural diagram of a sleep monitoring device provided in an embodiment of the present application is shown.
[0020] The purpose, features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0021] It should be understood that the specific embodiments described herein are merely used to explain the technical solutions of the present application and are not intended to limit the present application.
[0022] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.
[0023] The main solution of the embodiment of the present application is: obtaining multiple frames of continuous initial point cloud data within a preset time length; determining static clutter data and dynamic point cloud data for each frame of initial point cloud data; obtaining dynamic point cloud data by removing corresponding static clutter data from the initial point cloud data; superimposing multiple frames of dynamic point cloud data to obtain fused point cloud data; clustering the superimposed static clutter data to obtain at least one clutter cluster, at least one cluster including a noise cluster; superimposing static clutter data by superimposing multiple static clutter data; removing noise clusters from the fused point cloud data to obtain target point cloud data; and monitoring the sleeping posture of a target user based on the target point cloud data.
[0024] Among related technologies, human sleep monitoring is crucial in many contexts, such as sleep hygiene, bedridden patient care, and chronic disease diagnosis. Research has shown that sleep monitoring is crucial for sleep hygiene and the diagnosis of some chronic diseases. For example, sleeping posture is a key indicator for diagnosing positional obstructive sleep apnea.
[0025] Related technologies have limited means of monitoring user sleep. Wearable devices rely on specialized medical equipment and professionals, are costly, and difficult to widely promote. Some use computer vision to detect sleep, but this method not only poses privacy risks but is also easily affected by low light, obstacles, and other factors. Some use radio frequency signals to produce sheets with integrated wireless radio frequency identification (RFID) technology to monitor user sleep, but these sheets are not very comfortable.
[0026] In summary, how to monitor human sleep conditions efficiently and at low cost.
[0027] Based on this, the present application provides a solution to apply millimeter-wave radar to sleep monitoring scenarios, realizing a low-cost, high-efficiency, and high-accuracy sleep monitoring method.
[0028] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, mobile phone, etc., or a sleep monitoring device capable of performing the above functions. The following uses a sleep monitoring device as an example to illustrate this embodiment and the following embodiments.
[0029] Reference Figure 1 , Figure 1 The sleep monitoring method is shown. The sleep monitoring method can be applied to a sleep monitoring device, and includes the following steps S110 to S160: Step S110 , obtaining multiple frames of initial point cloud data within a preset time period.
[0030] In some implementations, the raw analog-to-digital converter data may be received via a universal asynchronous receiver-transmitter protocol.
[0031] The millimeter-wave radar is configured to communicate with the sleep monitoring device via a universal asynchronous receiver-transmitter protocol, and the distance fast Fourier transform function of the millimeter-wave radar is in a turned-off state.
[0032] The raw data collected by the millimeter-wave radar through the analog-to-digital converter (ADC) contains complete information of the captured reflected signal. In this embodiment, the ADC data collected by the millimeter-wave radar is the basis for subsequent processing.
[0033] In related technologies, radar serial ports based on the Universal Asynchronous Receiver / Transmitter (UART) protocol have a limited maximum rate (e.g., 3.125 Mbps), which is far from sufficient to transmit high-sampling-rate ADC data. Therefore, when acquiring data from millimeter-wave radar, a low-voltage differential signaling (LVDS) interface is typically used. This requires the millimeter-wave radar to be connected to a field-programmable gate array (FPGA) adapter board. The FPGA processes the data and then transmits the ADC data via a network cable to a terminal (e.g., a computer). This connection method is very complex in terms of device layout and connection, but it ensures relatively high-speed transmission.
[0034] However, in the application scenario of this embodiment, during sleep, the respiratory rate of an adult is 12-20 times per minute, roughly within the range of (0.2Hz, 0.33Hz). Therefore, even in extreme cases, this embodiment does not require a very high transmission rate.
[0035] To facilitate the arrangement, this embodiment modifies the radar firmware program to use only a single radar board to transmit 1 Hz sampled ADC data.
[0036] Specifically, this embodiment is modified based on the existing millimeter wave radar firmware. Taking the Ti model millimeter wave radar in the related art as an example, it is based on Figure 2 In the architecture shown, the parts that need to be modified are the DPU part in the DSP sub-system and the UART transmission part of the Master sub-system.
[0037] like Figure 3As shown in the figure, in related technologies, a host computer (such as a computer) often sends a control signal to activate the radar chip and related programs. The radar front end then acquires the analog signal and converts it into a digital signal. A range-dimensional Fast Fourier Transform (FFT) is then performed on the digital signal to extract the range information and store it in the chip's L3 memory. A Doppler FFT is then performed to extract the velocity information and store it in the L3 memory.
[0038] However, in this embodiment, the subsequent steps use the original ADC data, and there is no need to perform a range-dimensional FFT in advance. To this end, this embodiment is designed to disable the FFT function in the millimeter-wave radar DSP so that the millimeter-wave radar can directly transmit the raw ADC data after collecting it.
[0039] As mentioned above, the communication process between the radar and the host computer follows the UART protocol. As an example, by sending a data packet in TLV format, without changing the program framework, a new TLV definition for sending ADC data is added and sent in this format. Figure 2 ) in the code section related to TLV definition and UART data transmission, add TLV definition for ADC data, and store the radar ADC data stored at the initial address of L3 (0x5100_0000) through the UART protocol. Then recompile the Master sub-system firmware.
[0040] After this, you can modify the rangeprogress function in the rangeprochwa.c file in the millimeter-wave radar SDK and set the enable (EN) flag to 0 to disable FFT operations. Then, recompile the library function and the radar firmware.
[0041] Based on this, 1Hz raw ADC data transmission was achieved by modifying the radar firmware without changing the radar hardware design. That is, the sleep monitoring device in this embodiment can receive raw analog-to-digital converter data via the Universal Asynchronous Receiver / Transmitter protocol. The millimeter-wave radar is configured to communicate with the sleep monitoring device via the Universal Asynchronous Receiver / Transmitter protocol, and the millimeter-wave radar's range fast Fourier transform function is disabled.
[0042] After acquiring the raw ADC data from the millimeter-wave radar in the above manner, in some embodiments, multiple frames of continuous initial point cloud data within a preset time length can be obtained based on the raw analog-to-digital converter data.
[0043] Since the millimeter-wave radar sends a data packet via UART, the next step is to parse the data packet to obtain the raw ADC data. According to the TLV format designed in the aforementioned firmware, the data packet can be unpacked and saved as a file in a specified format (for example, .bin).
[0044] Specifically, the operator can pre-determine the COM port and connect the data transmission link. Once the connection is successful, the corresponding parameter configuration file is selected and transferred to the corresponding millimeter-wave radar. Once the parameter configuration file is successfully transferred, the millimeter-wave radar will issue a command and begin operation, collecting data, packaging it, and sending it to the host computer. The program in the host computer will unpack the data and save it locally.
[0045] like Figure 4 As shown in the figure, when depacketizing a data packet, the magic word is first searched to determine the starting position of the packet. Then, the packet header information, including version, platform, timestamp, packet length, frame number, and number of TLVs, is parsed. The TLV blocks are traversed to obtain the total data packet. Then, according to the TLV format set in the radar firmware, the TLV block containing the ADC data is obtained and saved as a ".bin" file.
[0046] After obtaining the raw ADC data, in some embodiments, the raw ADC data may be further processed to make subsequent sleep monitoring more accurate: In some implementations, for each frame of the original analog-to-digital converter data, corresponding first point cloud data may be obtained by decoding.
[0047] Specifically, after obtaining the bin file containing ADC data, unpack the bin file and convert it into The first point cloud data, where Represents the number of points in each chirp, represents the number of chirps in each frame, Represents the number of channels, Represents the frame number.
[0048] In some implementations, a range fast Fourier transform may be performed on the first point cloud data to obtain the second point cloud data.
[0049] Specifically, for an echo time domain signal , the number of sampling points of this signal in one Chirp period is N. Use the Hamming window function to reduce frequency leakage and Perform fast Fourier transform to obtain frequency domain signal , where k represents the frequency.
[0050]
[0051] Thus, the final distance information is obtained , i.e., the second point cloud data; in some feasible implementations, is 128.
[0052] After obtaining the second point cloud data, since there are still some static objects in the sleeping scene that are also collected by the millimeter-wave radar, in order to eliminate most irrelevant objects, the distance range can be determined based on the sleeping position of the target user, thereby limiting the subsequent point cloud range to the actual sleeping area of the target user.
[0053] Specifically, according to the current target user's sleeping scene, the distance between the bed and the millimeter-wave radar can be selected as the radius to perform scene modeling, and the aforementioned distance range can be divided from the modeling.
[0054] In some embodiments, a Doppler fast Fourier transform can be performed on the second point cloud data within the distance range to obtain a third point cloud data. As mentioned above, by extracting part of the point cloud data by distance range, irrelevant point cloud data can be excluded, making the calculations in subsequent steps more accurate and precise.
[0055] After obtaining the third point cloud data, the Doppler sampling interval can be calculated based on the preset breathing parameters. It is understandable that when the human body breathes, the part that changes is mostly the chest. According to relevant data, the chest breathing speed of a person during sleep is about a few millimeters per second, the frequency is 0.2-0.33Hz, and the wavelength of a 60GHz radar is 5 mm. In extreme cases, assuming that the breathing speed is 1 mm / second, based on the Doppler frequency shift formula:
[0056] It can be seen that the Doppler bin range is ±0.4Hz. Since the Doppler resolution is related to the number of Doppler fast Fourier transform points, the Doppler resolution can be pre-set in this embodiment. , which can be substituted into the calculation to obtain:
[0057] Then we have:
[0058]
[0059] Right now,[ , ] is the Doppler sampling interval described in this embodiment.
[0060] After the Doppler sampling interval is obtained, the portion of the third point cloud data that is within the Doppler sampling interval can be determined as the fourth point cloud data. Based on the Doppler sampling interval, the effective frequency segment where the human body frequently moves can be effectively filtered out, thereby effectively eliminating the influence of environmental noise.
[0061] In some embodiments, the portion of the fourth point cloud data having energy higher than a preset energy threshold may be determined as the fifth point cloud data. It is understood that in order to further reduce background noise, the present embodiment may artificially set a preset energy threshold in advance. , for example, can be .in, After obtaining the preset energy threshold, for each point in the fourth point cloud data, if its energy value is higher than the preset energy threshold, , it is determined as the target point; if its energy value is less than or equal to the preset energy threshold , then it is not used as the target point. Finally, we get the set , n is the total number of points, and the fifth point cloud data is obtained ,in, .
[0062] In some embodiments, since the radar may have multiple receiving antennas and there are spatial differences between the receiving antennas, there will be a slight phase difference between the signals received by each antenna. In order to estimate the angle of arrival (AoA) of the target signal, in this embodiment, the fifth point cloud data can be calculated based on the antenna arrangement of the millimeter wave radar. Perform angle fast Fourier transform to obtain initial point cloud data.
[0063] Specifically, for a set of receiving antennas with a fixed spacing of M in the horizontal and vertical directions, there are K targets in the horizontal and vertical directions, so the signal received by the M antenna array can be expressed as,
[0064] Where amp is the amplitude, For signal arrival The steering vector of the phase difference between the receivers when The arrival angle estimation can be regarded as estimating the azimuth or elevation angle of each k object based on the M antenna array. .
[0065] Therefore, for a specific radar distribution, the channel dimension in the fifth point cloud data can be split into . For the azimuth receiving antenna, perform angle FFT and you can get Then perform angle FFT on the pitch angle receiving antenna to finally obtain the initial point cloud data .
[0066] Therefore, through step S110, after obtaining the original ADC data of the millimeter-wave radar in a convenient manner, the point cloud data generated in each step is filtered, denoised and converted through a series of transformations, thereby obtaining initial point cloud data containing distance, speed and angle information.
[0067] Step S120 , determining static clutter data and dynamic point cloud data of each frame of initial point cloud data.
[0068] Since the millimeter-wave radar continuously acquires and transmits data at regular intervals after commencing operation, the initial point cloud data can be acquired in real time through step S110. In this embodiment, considering that a single-frame point cloud image has limited information and is susceptible to interference from environmental noise, resulting in a relatively limited number of points containing valid information, the design fuses multiple frames of initial point cloud data to increase the number of points containing valid information, thereby further improving the accuracy of subsequent sleep monitoring.
[0069] In some embodiments, for each frame of initial point cloud data, the average value of multiple points in the initial point cloud data in the Doppler dimension is calculated, and based on the average value, a static clutter matrix is established as the static clutter data of the corresponding frame, wherein the value of each element in the static clutter matrix in the Doppler dimension is the average value.
[0070] Specifically. As an example, the number of frames to be processed simultaneously can be preset, for example, 10 frames of initial point cloud data can be processed each time (i.e. ). For each frame of initial point cloud data, its corresponding static clutter data can be calculated.
[0071] Specifically, for the matrix , we can find the average value in the second dimension (i.e. Doppler dimension) to get the static clutter matrix. For each frame of initial point cloud data, we can use the initial point cloud data Subtract the corresponding static clutter matrix S to obtain the dynamic point cloud data corresponding to the frame.
[0072] It can be understood that, compared with the initial point cloud data, the dynamic point cloud data has effectively removed some static noise points.
[0073] Step S130 , superimposing multiple frames of dynamic point cloud data to obtain fused point cloud data.
[0074] Step S140 : clustering the superimposed static clutter data to obtain at least one clutter cluster, wherein the at least one cluster includes a noise cluster.
[0075] In some embodiments, the dynamic clutter matrix of each frame can be superimposed to obtain fused point cloud data. For example, In some embodiments, the static clutter matrix of each frame can be superimposed to obtain a superimposed static clutter matrix .
[0076] During sleep, frequent movement is limited to the upper body, while the lower body can be considered static. However, the lower body is crucial for determining sleeping posture and therefore needs to be retained. To distinguish the primary source of noise and retain data corresponding to the target user's lower body, this embodiment clusters the aforementioned superimposed static clutter matrix and uses the clustering results for screening.
[0077] Specifically, the density-based spatial clustering of applications with noise (DBSCAN) algorithm can be used to perform clustering on the above-mentioned superimposed static clutter matrix. Clustering is performed to obtain a plurality of different clutter clusters. In some implementations, the staff can pre-set labels for different clusters, such as noise clusters, human body clusters, etc., which is not limited in this embodiment.
[0078] Step S150 : removing noise clusters from the fused point cloud data to obtain target point cloud data.
[0079] In this embodiment, clusters labeled as noise are removed from the fused point cloud data, thereby achieving further feature enhancement of the point cloud data.
[0080] Step S160: monitoring the sleeping posture of the target user based on the target point cloud data.
[0081] In some implementations, the target point cloud data may be converted into a three-dimensional point cloud image by performing operations based on the following rules:
[0082]
[0083]
[0084] in, is the azimuth, is the pitch angle, which is obtained by the angle FFT calculation in the aforementioned step S110 and will not be described in detail in this embodiment.
[0085] In some implementations, the three-dimensional point cloud image may be pre-processed first.
[0086] Among them, preprocessing can include the following steps: Unified data dimension: 3D point cloud can be abstracted into a ( ) dimension matrix, Represents the number of points, and 3 represents the three dimensions of x, y, and z. Since the number of inputs in the model input layer is fixed, we set It is a fixed value, and zero padding is performed when the number of point clouds is insufficient to ensure data dimension consistency.
[0087] Data standardization. All data are normalized by z-score (standardization), that is, to ensure that all data are normally distributed with a mean of 0 and a standard deviation of 1, thus finally obtaining (batch, ,3) A matrix of size ,serves as the input matrix of the subsequent deep learning model.
[0088] After obtaining the input matrix, the input matrix can be input into the pre-trained deep learning model, and the global features can be extracted by the multi-layer perception network layer of the deep learning model; then the global features are used as the input of the fully connected layer of the deep learning model, and the fully connected layer classifies the global features, thereby finally obtaining the sleeping posture prediction result of the target user.
[0089] It is understood that in this embodiment, the pre-selected deep learning model can be a Pointnet model. In the training phase of the Pointnet model, the problem that the model needs to handle is first defined. The problem to be handled in this embodiment is to output the predicted type of sleeping posture based on the input point cloud data. Therefore, for the input data, it satisfies ,in Then define the sleeping posture category C, which satisfies , where K is the number of categories.
[0090] In addition, the weight W of the deep learning model can be defined, so that the deep learning model can be defined as , the final model output It can be characterized as:
[0091] For classification problems, the cross entropy function can be used as the loss function ,
[0092] in is the true label, is the predicted label.
[0093] In this embodiment, after the input data is input into the deep learning model, the multi-layer perception network layer in the deep learning model resamples the standard point cloud data to obtain key points, groups all key points based on the neighborhood range of each key point, and obtains multiple grouping results. Maximum pooling is performed on each grouping result, and the first main feature is extracted. Maximum pooling is performed on each first main feature, and the second main feature is extracted. The first main feature and the second main feature are fused and maximum pooled in sequence to obtain global features.
[0094] Specifically, assuming that the input point cloud data is the aforementioned , then through the multi-layer perception network (MLP) layer, P can be resampled to obtain This step is to extract key points from the point cloud data. It can be understood that key points are also part of the points in the point cloud data.
[0095] After obtaining the key points, the multi-layer perception network layer can group the key points based on the pre-set neighborhood range (which can be pre-set by the staff), and divide the points in the same neighborhood range into the same group. The grouping result can be expressed as:
[0096] in, For each of the aforementioned key points.
[0097] For each group result , use MLP (multi-layer perception network) to extract local features and perform maximum pooling to obtain :
[0098] This completes the first round of feature abstraction, K1 (i.e., the first main feature). Repeating the above process yields the second dimension of feature abstraction, K2 (i.e., the second main feature). For the two dimensions, K1 and K2, feature fusion is performed through concat, and global features are calculated through maximum pooling:
[0099]
[0100] Finally, the global features are input into the fully connected layer and Softmax classification is performed to obtain the final prediction result:
[0101]
[0102]
[0103] In the actual application of deep learning models, Adam can be used as an optimizer for training, and an adaptive learning rate for each parameter can be used to dynamically adjust the learning rate during training, so that it can effectively control the gradient update.
[0104] This embodiment provides a sleep monitoring method, which obtains multiple continuous frames of initial point cloud data within a preset time period; determines the static clutter data and dynamic point cloud data of each frame of initial point cloud data by removing static clutter data; superimposes the multiple frames of dynamic point cloud data to obtain fused point cloud data, thereby performing feature fusion on the multiple frames of point cloud data to remove noise; further, by clustering the superimposed static clutter data, noise clusters are removed from the fused point cloud data, thereby further considering the importance of the static part of the human body for sleep posture judgment, ensuring higher posture recognition accuracy while removing noise points, thereby facilitating subsequent efficient and high-accuracy monitoring of the target user's sleep posture based on the target point cloud data.
[0105] After step S110 , another embodiment of the present application further provides a sleep monitoring method for monitoring the breathing frequency of a target user.
[0106] Specifically, such as Figure 5 As shown, the sleep monitoring method may include: Step S210: extracting a matrix of interest from the second point cloud data.
[0107] The distance parameter value of the matrix of interest is within a selected distance range. The selected distance range can be pre-selected by a staff member. For example, the selected distance range can be the same as the distance range in step S110. That is, the second point cloud data within the distance range is the matrix of interest.
[0108] It is understandable that the matrix of interest reflects the change of the echo signal intensity over time in the distance range. In some embodiments, to ensure the consistency of calculation, s, that is, each calculation processes ten seconds of data.
[0109] After obtaining the matrix of interest, the points in the matrix of interest can be accumulated and averaged in the distance dimension to extract the time series data. .
[0110] Step S220 , executing a constant false alarm rate algorithm on the time series data to extract a respiratory time series signal.
[0111] To reduce the impact of noise, the Constant False Alarm Rate (CFAR) algorithm can be used to process the time series data, filter out the static signal, and extract the dynamic signal, namely the respiratory time series signal.
[0112] As described in the aforementioned step S110, considering the actual breathing conditions of the human body, a 0.1-0.5 Hz bandpass filter can be designed to filter the respiratory timing signal, thereby further filtering out irrelevant data (such as data with excessive fluctuations that are obviously not chest fluctuations).
[0113] Step S230 , segmenting the respiratory timing signal using a Hamming window to obtain a plurality of windows.
[0114] In some embodiments, the respiratory timing signal can be first divided into windows using a Hamming window, where the length of each window can be defined as L, and different windows can overlap to avoid edge effects. The length of the overlapping part between windows can be defined as U, is the Hamming window function, then:
[0115]
[0116] After obtaining the signal corresponding to each window, a fast Fourier transform can be performed on each window to convert the time domain signal into a frequency signal, that is:
[0117] Then, the power spectral density of each window is:
[0118] Step S240 : calculating the average power density of the multiple windows based on the power spectrum density corresponding to each window.
[0119] In some embodiments, the average power density can be calculated by the following expression:
[0120] Step S250: Determine the frequency corresponding to the maximum value of the power spectrum density as the breathing frequency of the target user.
[0121] It is understandable that in the average power density, the power spectrum density corresponding to different frequencies may be different. The biggest one The value of is determined as the respiratory rate.
[0122] By extracting the matrix of interest in this embodiment, the influence of static objects is eliminated and the accuracy of respiratory frequency calculation is improved; by windowing the respiratory timing signal, the respiratory frequency of the target user can be quickly calculated, thereby improving the reliability of sleep monitoring of the target user.
[0123] The present application provides a sleep monitoring device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the sleep monitoring method of the above-mentioned embodiment 1.
[0124] Reference below Figure 6 , which shows a schematic structural diagram of a sleep monitoring device suitable for implementing embodiments of the present application. The sleep monitoring device in the embodiments of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 6 The sleep monitoring device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.
[0125] like Figure 6As shown, sleep monitoring device 200 may include a processing device 210 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 220 or programs loaded from storage device 230 into random access memory (RAM) 240. RAM 240 also stores various programs and data required for the operation of the sleep monitoring device. Processing device 210, ROM 220, and RAM 240 are interconnected via bus 250. An input / output (I / O) interface 260 is also connected to the bus. Typically, the following systems may be connected to I / O interface 260: input device 270, including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output device 280, including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage device 230, including, for example, a magnetic tape, hard disk, etc.; and communication device 290. Communication device 290 can allow the sleep monitoring device to communicate with other devices wirelessly or wired to exchange data. Although the figure shows a sleep monitoring device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or provided instead.
[0126] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device 230, or installed from a ROM 220. When the computer program is executed by the processing device 210, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.
[0127] The sleep monitoring device provided in this application, employing the sleep monitoring method described in the aforementioned embodiment, can solve the technical problem of efficiently and cost-effectively monitoring a person's sleep. Compared to the prior art, the beneficial effects of the sleep monitoring device provided in this application are the same as those of the sleep monitoring method described in the aforementioned embodiment. Other technical features of the sleep monitoring device are the same as those disclosed in the aforementioned embodiment and are not further elaborated here.
[0128] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any one or more embodiments or examples in a suitable manner.
[0129] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.
[0130] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, a computer program) stored thereon, wherein the computer-readable program instructions are used to execute the sleep monitoring method in the above-mentioned embodiment.
[0131] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.
[0132] The computer-readable storage medium may be included in the sleep monitoring device, or may exist independently without being incorporated into the sleep monitoring device.
[0133] The computer-readable storage medium carries one or more programs that, when executed by the sleep monitoring device, enable the sleep monitoring device to write computer program code for performing the operations of the present application in one or more programming languages, or a combination thereof. The programming languages include object-oriented programming languages such as Java, Smalltalk, and C++, as well as conventional procedural programming languages such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0134] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.
[0135] The modules described in the embodiments of the present application may be implemented in software or hardware, wherein the name of a module does not necessarily limit the unit itself.
[0136] The computer-readable storage medium provided in this application stores computer-readable program instructions (i.e., a computer program) for executing the sleep monitoring method described above. This computer-readable storage medium can address the technical problem of efficiently and cost-effectively monitoring human sleep. Compared to the prior art, the beneficial effects of the computer-readable storage medium provided in this application are similar to those of the sleep monitoring method provided in the aforementioned embodiments and are not further elaborated here.
[0137] The present application also provides a computer program product, comprising a computer program, which implements the steps of the sleep monitoring method as described above when executed by a processor.
[0138] The computer program product provided in this application can solve the technical problem of how to monitor a person's sleep status efficiently and cost-effectively. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the sleep monitoring method provided in the above embodiment, and will not be elaborated here.
[0139] The above description is only part of the embodiments of the present application and does not limit the patent scope of the present application. All equivalent structural transformations made by using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.
Claims
1. A sleep monitoring method, characterized in that: Applied to sleep monitoring equipment, the sleep monitoring method includes: Obtain multiple frames of initial point cloud data within a preset time period; Determining static clutter data and dynamic point cloud data for each frame of initial point cloud data; the dynamic point cloud data is obtained by removing the corresponding static clutter data from the initial point cloud data; Superimposing multiple frames of dynamic point cloud data to obtain fused point cloud data; In the fused point cloud data, noise points are removed to obtain target point cloud data; Based on the target point cloud data, the sleeping posture of the target user is monitored.
2. The sleep monitoring method according to claim 1, wherein: Before removing noise points from the fused point cloud data to obtain target point cloud data, the method further includes: Superimposing a plurality of the static clutter data to obtain a superimposed static clutter matrix; Clustering the superimposed static clutter matrix to obtain at least one clutter cluster, at least one of the clutter clusters including a noise cluster; The step of removing noise points from the fused point cloud data to obtain target point cloud data includes: In the fused point cloud data, points in the noise cluster are removed to obtain target point cloud data.
3. The sleep monitoring method according to claim 1, wherein: Determining the static clutter data of each frame of initial point cloud data includes: For each frame of initial point cloud data, calculating the average value of multiple points in the initial point cloud data in the Doppler dimension; Based on the average value, a static clutter matrix is established as static clutter data of the corresponding frame, wherein the value of each element in the static clutter matrix in the Doppler dimension is the average value.
4. The sleep monitoring method according to claim 1, wherein: The step of obtaining multiple frames of initial point cloud data within a preset time period includes: receiving raw analog-to-digital converter data via a universal asynchronous receiver-transmitter protocol; wherein the millimeter-wave radar is configured to communicate with the sleep monitoring device via the universal asynchronous receiver-transmitter protocol, and a distance fast Fourier transform function of the millimeter-wave radar is in a disabled state; Based on the original analog-to-digital converter data, a plurality of continuous frames of initial point cloud data within the preset time length are obtained.
5. The sleep monitoring method according to claim 4, wherein: The step of obtaining a plurality of frames of initial point cloud data continuously within a preset time period based on the original analog-to-digital converter data includes: For each frame of the raw analog-to-digital converter data, obtain corresponding first point cloud data by decoding; Performing a distance fast Fourier transform on the first point cloud data to obtain second point cloud data; determining a distance range based on a sleeping position of the target user; performing a Doppler fast Fourier transform on the second point cloud data within the distance range to obtain third point cloud data; Calculating a Doppler sampling interval based on a preset respiratory parameter, and determining a portion of the third point cloud data that is within the Doppler sampling interval as fourth point cloud data; determining a portion of the fourth point cloud data whose energy is higher than a preset energy threshold as fifth point cloud data; Based on the antenna arrangement of the millimeter-wave radar, an angular fast Fourier transform is performed on the fifth point cloud data to obtain the initial point cloud data.
6. The sleep monitoring method according to claim 5, wherein: The method further comprises: extracting an interest matrix from the second point cloud data; wherein a distance parameter value of the interest matrix is within a selected distance range; Executing a constant false alarm rate algorithm on the time series data to extract a respiratory time series signal; wherein the time series data is calculated by the average value of the matrix of interest in the distance dimension; Segmenting the respiratory timing signal using a Hamming window to obtain a plurality of windows; Calculating an average power density of the plurality of windows based on the power spectral density corresponding to each of the windows; The frequency corresponding to the maximum value of the power spectrum density is determined as the breathing frequency of the target user.
7. The sleep monitoring method according to claim 1, wherein: The monitoring of the sleeping posture of the target user based on the target point cloud data includes: Global features are extracted by a multi-layer perception network layer of a deep learning model; wherein the multi-layer perception network layer resamples the standard point cloud data to obtain key points, groups all the key points based on the neighborhood range of each key point to obtain multiple grouping results, performs maximum pooling on each of the grouping results, extracts a first main feature, performs maximum pooling on each of the first main features, extracts a second main feature, and sequentially fuses and performs maximum pooling on the first main features and the second main features to obtain the global feature; The global features are classified through the fully connected layer of the deep learning model to obtain the sleeping posture prediction result of the target user.
8. A sleep monitoring device, characterized in that: The sleep monitoring device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the sleep monitoring method according to any one of claims 1 to 7.
9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the sleep monitoring method according to any one of claims 1 to 7 are implemented.
10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the sleep monitoring method according to any one of claims 1 to 7 are implemented.