A method and apparatus for sleep analysis

Abstract

The application is suitable for the technical field of data recognition, and provides a sleep analysis method and device, the sleep analysis method comprises the following steps: synchronously collecting multi-modal original signals based on multi-modal sensors in an intelligent waistband; extracting a modal feature matrix composed of the multi-modal original signals; inputting the modal feature matrix into a cross-modal correlation feature extraction network for processing to obtain a cross-modal correlation feature vector corresponding to a comprehensive sleep state; inputting the cross-modal correlation feature vector into a sleep analysis model to obtain a sleep staging category and a sleep physiological event. The sleep health monitoring scheme based on the intelligent waistband not only improves the accuracy and reliability of sleep monitoring, but also enhances the use convenience of users in daily life.

Description

Technical Field

[0001] This invention belongs to the technical field of data recognition, and particularly relates to a method and apparatus for sleep analysis. Background Technology

[0002] With increasing emphasis on health management, sleep health has gradually become an important research area. Good sleep quality has a direct impact on an individual's physical and mental health. However, traditional sleep monitoring methods often rely on complex equipment and environments, making it difficult to achieve convenient and accurate real-time monitoring, and users find it difficult to use them consistently in daily life.

[0003] In recent years, the rapid development of smart wearable devices has provided new solutions for sleep monitoring. Smart belts, in particular, have gained widespread attention as an emerging wearable device due to their comfort and ease of use. Currently, there are some sleep monitoring products based on wearable devices on the market, but most still suffer from insufficient data processing capabilities and low monitoring accuracy. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a method and apparatus for sleep analysis to solve the technical problems of insufficient data processing capabilities and low monitoring accuracy of traditional methods.

[0005] A first aspect of this invention provides a method for sleep analysis, the method comprising: The smart belt uses a multi-modal sensor to synchronously acquire multi-modal raw signals; wherein the multi-modal raw signals include at least environmental audio signals, heart rate signals, body temperature signals, and body movement signals. Extract a modal feature matrix composed of multimodal raw signals; wherein, the horizontal elements of the modal feature matrix are multiple feature values ​​corresponding to the same modal raw signal, and the vertical elements of the modal feature matrix are feature values ​​corresponding to different modal raw signals; The modal feature matrix is ​​input into a cross-modal correlation feature extraction network for processing to obtain a cross-modal correlation feature vector corresponding to the comprehensive sleep state; The cross-modal correlation feature vector is input into the sleep analysis model to obtain sleep stage categories and sleep physiological events; wherein, the sleep analysis model includes at least a sleep stage classifier and a sleep event detector.

[0006] Furthermore, the step of extracting the modal feature matrix composed of the original multimodal signals includes: The original multimodal signals are timestamped and divided into analysis time windows of preset duration; For each modal signal in the multimodal raw signal, feature values ​​at different time scales or frequency scales are extracted within the analysis time window to form the feature vector corresponding to that modal signal; wherein, the feature values ​​corresponding to the environmental audio signal include at least the power spectral density, Mel frequency cepstral coefficients, spectral entropy, and / or zero-crossing rate of a specific frequency band; the feature values ​​corresponding to the heart rate signal include at least the average heart rate, time-domain index of heart rate variability, and / or frequency-domain index of heart rate variability; the feature values ​​corresponding to the body motion signal include at least the signal amplitude area, motion energy, and / or the proportion of rest time. The feature vectors corresponding to the original signals of multiple modes are concatenated to form a modal feature matrix; wherein, the multiple data values ​​in the horizontal direction of the modal feature matrix are feature vectors corresponding to the same modal signal.

[0007] Furthermore, the cross-modal correlation feature extraction network includes at least a modality-specific encoder layer, a cross-modal correlation attention layer, and a feature fusion layer connected in sequence; Accordingly, the step of inputting the modal feature matrix into a cross-modal correlation feature extraction network for processing to obtain the cross-modal correlation feature vector corresponding to the comprehensive sleep state includes: The modality-specific encoder layer maps the modality feature matrix of each modality to multiple high-level abstract feature vectors corresponding to each modality; wherein, the modality-specific encoder layer includes sub-encoders that correspond one-to-one with each modality; The cross-modal association attention layer is used to model the association between the high-level abstract feature vectors of all modalities; wherein, the cross-modal association attention layer is configured to use the high-level abstract feature vectors of all modalities as input nodes to construct a fully connected modal feature map; Calculate the association attention weights between any two feature vectors of different modalities using graph attention mechanism or Transformer multi-head attention mechanism; Based on the associated attention weights, the feature vectors of each modality are weighted and aggregated to generate a set of feature vectors modulated by information from other modalities; The feature fusion layer concatenates the set of feature vectors and performs dimensionality reduction and fusion through at least one fully connected layer, outputting the cross-modal correlation feature vector corresponding to the comprehensive sleep state within the current analysis time window.

[0008] Furthermore, the sleep analysis model includes a hierarchical attention feature allocation module, a sleep stage classification branch, and a sleep physiological event detection branch; the sleep physiological events include at least apnea events, hypopnea events, snoring events, limb movement events, and REM sleep behavioral disorder events; Accordingly, the step of inputting the cross-modal correlation feature vector into the sleep analysis model to obtain sleep stage categories and sleep physiological events includes: The hierarchical attention feature allocation module receives the cross-modal associated feature vector and generates sleep stage-specific feature vectors and sleep physiological event-specific feature vectors. Input the sleep stage-specific feature vector into the sleep stage classification branch to obtain the sleep stage category; The sleep physiological event-specific feature vector is input into the sleep physiological event detection branch to obtain the sleep physiological event; wherein, both the sleep stage classification branch and the sleep physiological event detection branch adopt the evidence depth network; the evidence depth network obtains the classification result by outputting the evidence vector and parameterizing the Dirichlet distribution.

[0009] Furthermore, the hierarchical attention feature allocation module includes a shared feature encoder, a sleep stage attention subnetwork, and a sleep event attention subnetwork; The sleep stage attention subnetwork and the sleep event attention subnetwork learn from the shared features output by the shared feature encoder and apply different attention weights to extract task-specific feature vectors.

[0010] Furthermore, the sleep analysis model also includes a multi-task consistency verification unit; The multi-task consistency verification unit evaluates whether sleep stage categories and sleep physiological events meet the constraints in the logical consistency rule base based on the logical consistency rule base. If the sleep stage category and the sleep physiological event meet the constraints, then the sleep stage category and the sleep physiological event are used as the sleep stage category and sleep physiological event of the current analysis time window. If the sleep stage category and sleep physiological event do not meet the constraints, then the sleep stage category and sleep physiological event corresponding to the previous analysis time window that met the constraints will be used as the sleep stage category and sleep physiological event for the current analysis time window.

[0011] Furthermore, prior to the step of extracting the modal feature matrix composed of the original multimodal signals, the method further includes: Calculate the signal-to-noise ratio of the heart rate signal within the characteristic frequency band; Calculate the periodicity index of the peak-to-peak interval sequence of the heart rate signal; Calculate the standard deviation of the vector amplitude of the body motion signal and convert it into a body motion interference index that reflects the risk of motion artifacts; The signal-to-noise ratio index, periodicity index, and motion interference index are weighted and fused to generate the final signal reliability index. When the signal confidence index is lower than a preset threshold, an adaptive filtering algorithm is triggered. The adaptive filtering algorithm uses the body motion signal as reference noise to perform dynamic noise reduction processing on the heart rate signal.

[0012] Furthermore, the step of triggering an adaptive filtering algorithm when the signal confidence index is lower than a preset threshold, wherein the adaptive filtering algorithm uses the body motion signal as reference noise to dynamically denoise the heart rate signal includes: The vector amplitude VM(n) is calculated based on the body motion signal, and the vector amplitude VM(n) is shaped by a linear time-varying filter to generate a reference noise signal r(n); The reference noise signal r(n) and the heart rate signal s_hr_raw(n) are input into an adaptive filter; wherein the weight vector W(n) of the adaptive filter is iteratively updated according to the following function: W(n+1)=W(n)+(μ(n) / (δ+R) T (n)R(n)))*e(n)*R(n); Where, e(n) = s_hr_raw(n) - W T (n)*R(n) represents the error signal at the current time, and R(n) is the input vector composed of r(n) at the current and past times. T (n) represents the transpose of R(n), W T (n) denotes the transpose of W(n), and δ is a constant to prevent division by zero; The step size factor μ(n) of the adaptive filter is dynamically adjusted according to the signal reliability index C(n); where μ(n) = μ_base*(1-α*C(n)), μ_base is the base step size, and α is the adjustment coefficient; The error signal e(n) is output as the heart rate signal after the dynamic noise reduction process.

[0013] A second aspect of the present invention provides a sleep analysis apparatus, comprising: The acquisition unit is used to synchronously acquire multimodal raw signals based on the multimodal sensors in the smart belt; wherein, the multimodal raw signals include at least environmental audio signals, heart rate signals, body temperature signals, and body movement signals; An extraction unit is used to extract a modal feature matrix composed of multimodal raw signals; wherein, the horizontal elements of the modal feature matrix are multiple feature values ​​corresponding to the same modal raw signal, and the vertical elements of the modal feature matrix are feature values ​​corresponding to different modal raw signals; The first identification unit is used to input the modal feature matrix into the cross-modal correlation feature extraction network for processing to obtain the cross-modal correlation feature vector corresponding to the comprehensive sleep state; The second identification unit is used to input the cross-modal correlation feature vector into the sleep analysis model to obtain sleep stage categories and sleep physiological events; wherein, the sleep analysis model includes at least a sleep stage classifier and a sleep event detector.

[0014] A third aspect of the present invention provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the sleep analysis method described in the first aspect above.

[0015] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the sleep analysis method described in the first aspect.

[0016] The beneficial effects of this invention compared to existing technologies are as follows: by integrating multimodal sensors to simultaneously collect multimodal raw signals such as environmental audio signals, heart rate signals, body temperature signals, and body movement signals, it overcomes the shortcomings of traditional monitoring methods. Modal feature matrices are extracted using these multimodal signals, and deep processing is performed through a cross-modal correlation feature extraction network, effectively extracting cross-modal correlation features related to sleep states. The sleep health monitoring solution based on a smart belt not only improves the accuracy and reliability of sleep monitoring but also enhances the convenience for users in daily life. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A schematic flowchart of a sleep analysis method provided by the present invention is shown; Figure 2 A schematic diagram of a sleep analysis device according to an embodiment of the present invention is shown; Figure 3 A schematic diagram of a terminal device provided in an embodiment of the present invention is shown. Detailed Implementation

[0019] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of the invention. However, those skilled in the art will understand that the invention can be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of the invention with unnecessary detail.

[0020] This invention provides a method and apparatus for sleep analysis to address the technical problems of insufficient data processing capabilities and low monitoring accuracy in traditional methods.

[0021] First, this invention provides a method for sleep analysis. Please see below. Figure 1 , Figure 1 A schematic flowchart of a sleep analysis method provided by the present invention is shown. Figure 1 As shown, the sleep analysis method may include the following steps: Step 101: Synchronously acquire multimodal raw signals based on the multimodal sensors in the smart belt; wherein, the multimodal raw signals include at least environmental audio signals, heart rate signals, body temperature signals, and body movement signals; Technical means: Based on the multi-mode sensors in the smart belt, multi-mode raw signals are collected synchronously.

[0022] Logical purpose: This is the data foundation of the entire method. It emphasizes the device carrier (smart belt) and data characteristics (multimodal, synchronous).

[0023] Smart belt: This limits the hardware scenario for the invention. It is worn on the waist and can stably collect physiological signals related to the torso, which distinguishes it from wristbands, bedside devices, etc.

[0024] Multimode sensor: refers to multiple sensors integrated into the belt (such as microphone, photoelectric heart rate sensor, temperature sensor, accelerometer / gyroscope).

[0025] Synchronous acquisition is crucial, meaning that different signals are aligned in time. This is essential for subsequent analysis of the temporal correlation between different physiological indicators (e.g., a single apnea event may simultaneously trigger changes in heart rate and body movement).

[0026] As an optional embodiment of this application, steps A1 to A5 are further included before step 102: Step A1: Calculate the signal-to-noise ratio of the heart rate signal within the characteristic frequency band; a. Signal-to-noise ratio of heart rate signal frequency band Technical means: Calculate the signal-to-noise ratio of the heart rate signal in the characteristic frequency band.

[0027] Logical purpose: To assess the purity and intensity of the heart rate signal.

[0028] "Characteristic frequency band": This refers to the frequency range where a normal heart rate (e.g., 0.5 Hz - 3 Hz, corresponding to 30-180 BPM) is located. Calculating the signal-to-noise ratio (SNR) within this band quantifies the relative strength of effective heart rate information power compared to broadband noise power (e.g., baseline drift, high-frequency interference). A low SNR indicates that the signal may be overwhelmed by noise.

[0029] Step A2: Calculate the periodicity index of the peak-to-peak interval sequence of the heart rate signal; b. Heart rate cycle regularity index Technical means: Calculate the periodicity index of the peak-to-peak interval sequence of the heart rate signal.

[0030] Logical purpose: To assess the physiological rationality and stability of heart rate signals.

[0031] "Peak-to-peak interval sequence": also known as RR interval sequence, is the basis for heart rate variability analysis.

[0032] "Period regularity index": This can be judged by calculating the standard deviation (SDNN) and coefficient of variation of the RR interval, or by checking whether it conforms to a physiologically possible range (e.g., not less than 300ms or greater than 2000ms). A healthy sleep heart rate sequence should have a certain degree of variability but should not exhibit extremely irregular beats. Poor regularity may mean a large number of missed or falsely detected R waves, i.e., severe signal distortion.

[0033] Step A3: Calculate the standard deviation of the vector amplitude of the body motion signal and convert it into a body motion interference index that reflects the risk of motion artifacts; Technical approach: Calculate the standard deviation of the vector amplitude of the body motion signal and convert it into a body motion interference index that reflects the risk of motion artifacts.

[0034] Logical objective: To directly quantify the intensity of motion artifacts, which are the primary source of noise in wearable optoelectronic heart rate signals.

[0035] "Vector amplitude standard deviation": It is usually calculated from the triaxial accelerometer signal (e.g., VM = sqrt(x^2 + y^2 + z^2)), and its standard deviation reflects the amplitude and fluctuation of body motion.

[0036] Converting to an interference index: This involves using a mapping function (which may be linear or non-linear) to transform the standard deviation of body movement amplitude into an index characterizing its potential interference risk to heart rate signals. The more intense the body movement, the higher this index.

[0037] Step A4: The signal-to-noise ratio index, periodicity index, and motion interference index are weighted and fused to generate the final signal reliability index; Technical approach: The signal-to-noise ratio, periodicity, and motion interference indicators are weighted and fused to generate the final signal reliability index.

[0038] Logical Purpose: Comprehensive Decision-Making. A single indicator may lead to misjudgment (e.g., minimal body movement during rest may result in a poor heart rate signal due to inadequate perfusion). Weighted fusion can generate a more comprehensive and robust overall quality score. Weights can be optimized based on experimental experience or machine learning.

[0039] Step A5: When the signal confidence index is lower than a preset threshold, an adaptive filtering algorithm is triggered. The adaptive filtering algorithm uses the body motion signal as reference noise to perform dynamic noise reduction processing on the heart rate signal.

[0040] Technical means: When the signal credibility index is lower than a preset threshold, an adaptive filtering algorithm is triggered. The adaptive filtering algorithm uses the body motion signal as reference noise to perform dynamic noise reduction processing on the heart rate signal.

[0041] Logical objective: To achieve "on-demand processing" and maximize efficiency while ensuring effectiveness.

[0042] Threshold triggering mechanism: The computationally intensive adaptive filtering is only activated when the signal quality is truly poor (low confidence index). For high-quality signal segments, the filtering is skipped directly, saving computational resources. This is crucial for low-power embedded devices (such as smart belts).

[0043] Adaptive filtering algorithms: This is a classic technique in signal processing, such as the minimum mean square error adaptive filter.

[0044] Using body motion signal as reference noise: This is the key and ingenious aspect of this step. It is based on a reasonable assumption: motion artifacts in the photoelectric heart rate signal are strongly correlated with the body motion signal measured by the accelerometer. The adaptive filter uses the body motion signal as "reference noise input," dynamically adjusts its coefficients, estimates and subtracts the noise components related to body motion from the mixed signal (heart rate + motion artifacts), thereby recovering a purer heart rate signal.

[0045] In the embodiments corresponding to steps A1 to A5, signal quality is evaluated by calculating the signal-to-noise ratio, periodicity regularity, and body motion interference index, and a signal reliability index is generated through weighted fusion. When the signal quality is substandard, an adaptive filtering algorithm is triggered for dynamic noise reduction. This series of steps ensures the accuracy and effectiveness of subsequent feature extraction, providing a reliable data foundation for sleep health monitoring.

[0046] Specifically, step A5 includes steps A51 to A54: Step A51: Calculate the vector amplitude VM(n) based on the body motion signal, and shape the vector amplitude VM(n) through a linear time-varying filter to generate a reference noise signal r(n); Technical means: Calculate the vector amplitude VM(n) based on the body motion signal, and shape the vector amplitude VM(n) through a linear time-varying filter to generate a reference noise signal r(n).

[0047] Logical objective: To prepare a reference input for the adaptive filter that is as relevant as possible to the noise components in the heart rate signal.

[0048] Calculate the vector magnitude VM(n): This is the standard method for extracting the overall motion intensity of the body from triaxial accelerometer data (VM = sqrt(x^2 + y^2 + z^2)).

[0049] Linear time-varying filter shaping: This is a crucial optimization step. The original body motion amplitude VM(n) is not perfectly identical in waveform to the motion artifacts in the heart rate signal; there may be differences in phase, frequency response, or amplitude. The role of this shaping filter (which can be an FIR or IIR filter, whose coefficients may change slowly according to the signal characteristics) is to preprocess VM(n) to make its morphology closer to the actual motion artifacts present in the heart rate channel, thereby improving the cancellation effect of the adaptive filter. For example, it can simulate the specific frequency response of a photoelectric sensor to mechanical motion.

[0050] Step A52: Input the reference noise signal r(n) and the heart rate signal s_hr_raw(n) into the adaptive filter; wherein, the weight vector W(n) of the adaptive filter is iteratively updated according to the following function: W(n+1)=W(n)+(μ(n) / (δ+R)) T (n)R(n)))*e(n)*R(n); Where, e(n) = s_hr_raw(n) - W T (n)*R(n) represents the error signal at the current time, and R(n) is the input vector composed of r(n) at the current and past times. T (n) represents the transpose of R(n), W T (n) denotes the transpose of W(n), and δ is a constant to prevent division by zero; Technical means: A clear definition of the weight update function and error signal is given.

[0051] Error signal: e(n) = s_hr_raw(n) - WT(n)*R(n) Logically: s_hr_raw(n) is the original noisy heart rate signal. WT(n)*R(n) is the weighted sum of the current reference noise R(n) (a vector containing the current and past times r(n)) by the adaptive filter, i.e., the noise component estimated by the filter. e(n) is the original signal minus the estimated noise, theoretically yielding a clean heart rate signal. e(n) is both the output and used for feedback adjustment of the filter weights.

[0052] Weight update function: W(n+1) = W(n) + (μ(n) / (δ+RT(n)R(n)))*e(n)*R(n) Logical: This is the standard form of the Normalized Least Mean Square (NLMS) adaptive algorithm, an improved variant of the LMS algorithm with better stability.

[0053] μ(n): Step size factor, which controls the speed and stability of weight updates (detailed below).

[0054] RT(n)R(n): The energy (autocorrelation) of the reference input vector R(n). The normalization operation (dividing by δ+RT(n)R(n)) makes the algorithm insensitive to changes in the input signal power, resulting in more robust convergence performance.

[0055] e(n)*R(n): This is the gradient estimation term, indicating in which direction the weights should be adjusted to reduce the error.

[0056] Step A53: The step size factor μ(n) of the adaptive filter is dynamically adjusted according to the signal confidence index C(n); where μ(n) = μ_base*(1-α*C(n)), μ_base is the base step size, and α is the adjustment coefficient; Technical means: The step size factor μ(n) of the adaptive filter is dynamically adjusted according to the signal reliability index C(n); where μ(n) = μ_base*(1-α*C(n)).

[0057] Logical objective: To achieve intelligent adaptive filtering behavior, which is the essence of this claim.

[0058] The significance of the step size factor μ: In the NLMS algorithm, μ determines the convergence speed and steady-state error. A large μ results in fast convergence, but a large steady-state error and potential instability; a small μ results in slow convergence, but high steady-state accuracy.

[0059] Dynamic adjustment mechanism: When the signal reliability is high (large C(n)), it means that the original heart rate signal itself is of good quality and the noise is likely small. In this case, we hope that the filter will not change the weights too aggressively to avoid introducing distortion. According to the formula μ(n)=μ_base*(1-α*C(n)), the larger C(n) is, the smaller μ(n) is, and the filter behavior tends to be conservative and fine-tuned.

[0060] When signal reliability is low (small C(n)): this means the original heart rate signal quality is poor and noise is strong. In this case, we need the filter to adjust its weights quickly and significantly to keep up with changes in noise and perform strong noise reduction. At this point, μ(n) will approach μ_base (or be larger, depending on the setting of α), and the filter behavior will become more aggressive and fast-tracking.

[0061] Logical Value: This dynamic adjustment transforms the filter from a fixed set of parameters into a system that intelligently balances tracking speed and steady-state accuracy based on real-time signal quality assessments. It achieves rapid convergence to suppress strong interference in poor signal conditions and fine-tunes to avoid overshoot in good signal conditions, resulting in superior overall noise reduction performance.

[0062] Among these factors, body motion interference is the most important. Motion artifacts are the primary noise source for wearable photoelectric heart rate monitors during sleep. Body motion amplitude directly and reliably reflects noise risk and should therefore have the highest weight (a weight of 0.5 is appropriate). Periodicity regularity is a core validity test. Even if the SNR is acceptable, if an error in detecting the R-peak leads to a disordered RR interval sequence, the signal becomes completely unusable. This indicator can effectively capture such fatal errors (a weight of 0.3 is appropriate). Bandwidth signal-to-noise ratio is a basic quality metric. It reflects the purity of the signal but may be affected by non-motion noise (such as electronic noise and perfusion variations) and has some correlation with body motion indicators; therefore, its weight is slightly lower (a weight of 0.2 is appropriate).

[0063] Step A54: Output the error signal e(n) as the heart rate signal after the dynamic noise reduction process.

[0064] As mentioned earlier, in the adaptive noise cancellation framework, the error signal e(n) is the expected signal (pure heart rate) estimated by the system.

[0065] In the embodiments corresponding to steps A51 to A54, the process of dynamically denoising the heart rate signal using an adaptive filtering algorithm is described. Through steps such as calculating vector amplitude, generating reference noise, inputting it into the adaptive filter, iteratively updating weights, calculating the error signal, and dynamically adjusting the step size factor, effective denoising of the heart rate signal is ultimately achieved, providing more accurate and reliable data for subsequent analysis.

[0066] Step 102: Extract the modal feature matrix composed of multimodal raw signals; wherein, the horizontal elements of the modal feature matrix are multiple feature values ​​corresponding to the same modal raw signal, and the vertical elements of the modal feature matrix are feature values ​​corresponding to different modal raw signals; Technical approach: Extract the modal feature matrix composed of multimodal raw signals and define the horizontal (multi-modal features) and vertical (different modal features) elements of the matrix.

[0067] Logical objective: To transform raw, high-dimensional, and time-series sensor data into a structured data representation suitable for model processing.

[0068] Feature extraction: Calculate representative indicators from each raw signal. For example, extract time-frequency domain features of mean heart rate and heart rate variability (HRV) from heart rate signals; extract activity intensity and posture change frequency from body movement signals.

[0069] Matrix-based construction: This is a core concept. It unifies multi-source heterogeneous features into a two-dimensional data structure.

[0070] The horizontal row represents the "feature depth" of a particular modality signal. For example, the heart rate row may contain multiple feature values ​​such as [mean HR, LF power, HF power, etc.].

[0071] Vertical column: Represents the "breadth of modalities" at the same point in time. For example, a column may contain [audio energy, heart rate, body temperature, and body movement amplitude] at time T.

[0072] Logical value: This matrix preserves the parallelism of multimodal information, providing structured input for the next step of cross-modal correlation analysis.

[0073] Specifically, step 102 includes steps 1021 to 1023: Step 1021: Timestamp-align the original multimodal signal and divide it into analysis time windows of preset duration; Technical means: The original multimodal signals are timestamped and divided into analysis time windows of preset duration.

[0074] Logical purpose: This is a prerequisite for constructing an effective feature matrix, and it solves two key problems in multi-source data fusion.

[0075] Timestamp alignment: This ensures that signals from different sensors are strictly synchronized on the timeline. For example, snoring recorded by a belt microphone and heart rate fluctuations recorded by a heart rate sensor must correspond to the same moment; otherwise, subsequent correlation analysis will be meaningless. This is typically achieved through hardware synchronization or software timestamp interpolation.

[0076] Dividing the signal into analysis time windows: The continuous signal stream is cut into segments of fixed length (e.g., 30 seconds, 1 minute) for processing. This is a key step in simulating standard sleep staging rules (PSG is usually interpreted in 30-second "frames") and adapting to the model's processing requirements. It realizes the conversion from continuous signals to discrete analysis units.

[0077] Step 1022: For each modal signal in the multimodal raw signal, within the analysis time window, extract feature values ​​at different time scales or frequency scales to construct the feature vector corresponding to that modal signal; wherein, the feature values ​​corresponding to the environmental audio signal include at least the power spectral density, Mel frequency cepstral coefficients, spectral entropy, and / or zero-crossing rate of a specific frequency band; the feature values ​​corresponding to the heart rate signal include at least the average heart rate, time-domain index of heart rate variability, and / or frequency-domain index of heart rate variability; the feature values ​​corresponding to the body motion signal include at least the signal amplitude area, motion energy, and / or the proportion of resting time; Technical means: For each modal signal, feature values ​​of different time scales or frequency scales are extracted within the window, and the core features are specifically listed.

[0078] Logical objective: To extract the most representative digital indicators of sleep physiological state from the raw waveforms, and suitable for computational analysis. This is a concentrated manifestation of the professionalism and scientific nature of the technical solution.

[0079] "Different time scales or frequency scales": This emphasizes the comprehensiveness of feature extraction. For example, heart rate variability (HRV) has both time-domain indicators (such as SDNN, which reflects overall changes) and frequency-domain indicators (such as LF / HF, which reflects the balance of sympathetic / parasympathetic nerves), which reveal the state of the autonomic nervous system from different dimensions.

[0080] Characteristics of ambient audio signals: Power spectral density in a specific frequency band: quantifies the intensity (energy) of snoring and its main frequency components.

[0081] Mel frequency cepstral coefficients: These are classic features in speech / sound recognition, effectively characterizing the timbre of snoring and helping to distinguish different types of breathing sounds.

[0082] Spectral entropy: Reflects the complexity or regularity of a sound signal. The spectral entropy value of a steady snoring sound is different from that of a sudden cough / sleep talking.

[0083] Zero-crossing rate: roughly reflects the frequency of the signal and can be used to detect respiratory rhythm.

[0084] Heart rate signal characteristics: Average heart rate: a basic physiological indicator. The average heart rate varies systematically depending on the sleep stage (lowest during deep sleep).

[0085] Heart rate variability (HRV) time-domain / frequency-domain metrics are the gold standard for assessing sleep quality and autonomic nervous activity. The HRV patterns of deep sleep and REM sleep are distinctly different, and apnea events are often accompanied by specific fluctuations in HRV.

[0086] Characteristics of body movement signals: Signal amplitude area / kinetic energy: quantifies the overall intensity of physical activity.

[0087] Percentage of resting time: This directly reflects the degree of rest during sleep and is an important characteristic that distinguishes between wakefulness and sleep.

[0088] Logical value: This part not only defines "what to do", but also clarifies "how to do it" through specific, medical and signal processing-based feature enumeration, making the technical solution fully public and implementable, and building a more robust patent protection barrier.

[0089] Step 1023: Concatenate the feature vectors corresponding to the original signals of multiple modes into a modal feature matrix; wherein, the multiple data values ​​in the horizontal direction of the modal feature matrix are feature vectors corresponding to the same modal signal.

[0090] Technical approach: The feature vectors corresponding to the original signals of multiple modes are concatenated into a modal feature matrix, and the horizontal structure of the matrix is ​​clarified again.

[0091] Logical objective: To organize the extracted, heterogeneous multimodal features into a unified data structure defined in the independent claims that can be processed by neural networks.

[0092] Concatenation operation: This typically refers to joining feature vectors from audio, heart rate, body temperature, and body movement within each time window, either row-wise or column-wise. For example, forming a matrix where each row represents all features of a modality, and each column represents a specific feature dimension.

[0093] To reiterate the definition of a horizontal matrix: multiple data values ​​in a horizontal matrix constitute an eigenvector corresponding to the same modal signal. This echoes the definition in the independent claim, ensuring logical consistency. It emphasizes that a row of the matrix is ​​an eigenvector composed of multiple eigenvalues ​​of the same modality.

[0094] In the embodiments corresponding to steps 1021 to 1023, by dividing and extracting the features of multimodal signals in detail, it is ensured that the characteristics of various signals can be fully utilized when analyzing sleep health, thereby improving the accuracy and effectiveness of monitoring.

[0095] Step 103: Input the modal feature matrix into the cross-modal correlation feature extraction network for processing to obtain the cross-modal correlation feature vector corresponding to the comprehensive sleep state; Technical approach: The modal feature matrix is ​​input into a cross-modal correlation feature extraction network for processing to obtain cross-modal correlation feature vectors.

[0096] Logical Purpose: This is a key step that could enable the inventive step of this claim. It aims to discover and learn the implicit, complex interrelationships between different modal features, rather than simply piecing them together.

[0097] “Cross-modal associations”: For example, “specific slight body movements accompanied by a brief increase in heart rate and subtle snoring” may be associated with “the transition from deep sleep to light sleep”; “a slow decrease in body temperature and a decrease in heart rate with no body movement” may be associated with “entering deep sleep”.

[0098] Network processing: This suggests the use of a deep learning model (such as CNN, Transformer, or a custom network) that can automatically learn the interaction patterns in the matrix both horizontally (between features) and vertically (between modalities).

[0099] Output vector: The processed high-dimensional information is compressed into a low-dimensional, dense feature vector. This vector is a higher-level abstract representation that incorporates multimodal information, making it more discriminative than the original feature matrix and more suitable for subsequent classification and detection tasks.

[0100] The cross-modal correlation feature extraction network includes at least a modality-specific encoder layer, a cross-modal correlation attention layer, and a feature fusion layer connected in sequence.

[0101] Specifically, step 103 includes steps 1031 to 1035: Step 1031: Through the modality-specific encoder layer, the modality feature matrix of each modality is mapped to multiple high-level abstract feature vectors corresponding to each modality; wherein, the modality-specific encoder layer includes sub-encoders that correspond one-to-one with each modality; Technical approach: Through a modality-specific encoder layer, the modal feature matrix of each modality is mapped to multiple high-level abstract feature vectors corresponding to each modality, and explicitly includes sub-encoders that correspond one-to-one with each modality.

[0102] Logical objective: To gain a deep understanding and refine the information of each modality before engaging in cross-modal interaction.

[0103] "Sub-encoder" means designing or selecting the most suitable neural network for different modalities such as audio, heart rate, and body movement. For example, one-dimensional convolutional neural networks (1D-CNN) or variants of recurrent neural networks (RNN) may be used for time-series physiological signals (heart rate, body movement); two-dimensional CNNs may be used for spectral features (audio). This reflects respect for modal heterogeneity and targeted processing.

[0104] "High-level abstract feature vectors": The output of each sub-encoder is no longer the original statistical features (such as average heart rate), but a dense vector learned through nonlinear transformation that better represents the intrinsic patterns and deep semantics of the modality. For example, the heart rate encoder may output a vector representing the "intensity of autonomic nervous activity", and the audio encoder may output a vector representing the "degree of respiratory sound disturbance".

[0105] 2. Cross-modal Relational Attention Layer: Core Innovation – Dynamic Relationship Modeling This is the most inventive part of the claim, as it specifically implements the mechanism of "cross-modal association".

[0106] Step 1032: Through the cross-modal association attention layer, perform association modeling on the high-level abstract feature vectors of all modalities; wherein, the cross-modal association attention layer is configured to construct a fully connected modal feature map using the high-level abstract feature vectors of all modalities as input nodes; Technical approach: Construct a fully connected modal feature map using high-level abstract feature vectors of all modalities as input nodes.

[0107] Logical objective: To formalize the multimodal feature interaction problem as a graph learning problem. Each modality is treated as a node, and the edges (fully connected) between nodes represent potential interaction relationships. This provides a structural basis for subsequently calculating the attention weights between any pairwise modalities.

[0108] Step 1033: Calculate the association attention weights between any two feature vectors of different modalities using graph attention mechanism or Transformer multi-head attention mechanism; Technical means: Calculate the correlation attention weight between any two feature vectors of different modalities through graph attention mechanism or Transformer multi-head attention mechanism.

[0109] Logical objective: To dynamically and adaptively learn the importance relationships between different modalities.

[0110] "Attention weight": This is not a fixed value, but is calculated based on the current input (i.e., the specific physiological signal in the current time window). For example, when detecting sleep apnea events, the attention weight of the "audio" modality over the "heart rate" modality may be very high because sleep apnea usually triggers changes in blood oxygen and heart rate first; while when determining sleep stages, the weight between the "body movement" and "heart rate variability" modalities may play a dominant role.

[0111] Mechanism Selection: Two cutting-edge and efficient implementation schemes are provided—Graph Attention Network (GAT) or Transformer multi-head attention. Both can capture complex nonlinear relationships and have good interpretability (visualized through weights).

[0112] Step 1034: Based on the associated attention weights, perform weighted aggregation on the feature vectors of each modality to generate a set of feature vectors modulated by information from other modalities; Technical approach: Based on the correlation attention weight, the feature vectors of each modality are weighted and aggregated to generate a set of feature vectors modulated by information from other modalities.

[0113] Logical objective: To enable the flow and enhancement of information between modalities.

[0114] For the feature vector of the "heart rate" modality, it now not only contains its own information, but also incorporates supplementary information from other modalities such as audio and body movement, which are weighted by the attention mechanism.

[0115] The term "modulation" is very precise; it means that the representation of each modality is adjusted and enriched according to the global context (the state of other modalities). This simulates the thought process of a doctor when comprehensively judging multiple physiological indicators.

[0116] Step 1035: The feature fusion layer concatenates the set of feature vectors and performs dimensionality reduction and fusion through at least one fully connected layer to output the cross-modal correlation feature vector corresponding to the comprehensive sleep state within the current analysis time window.

[0117] Technical approach: A set of feature vectors is concatenated and then subjected to dimensionality reduction and fusion through at least one fully connected layer to output... cross-modal correlated feature vectors.

[0118] Logical objective: To integrate fully interacted and modulated multimodal features into a compact and powerful single vector as input to the final sleep analysis model.

[0119] splicing: retain all information after modal modulation.

[0120] Fully connected layer dimensionality reduction and fusion: Through nonlinear transformation, information is further compressed, redundancy is eliminated, and the final joint representation best suited for downstream sleep staging and event detection tasks is learned. This output feature vector is the digital summary of the overall sleep state in the current time window learned by the entire network.

[0121] In the embodiments corresponding to steps 1031 to 1035, the modal feature matrix is ​​processed by a cross-modal correlation feature extraction network to obtain a cross-modal correlation feature vector of comprehensive sleep state. The joint design of a modality-specific encoder, a cross-modal correlation attention layer, and a feature fusion layer ensures the effective integration of multimodal data, thereby improving the accuracy and effectiveness of sleep analysis.

[0122] Step 104: Input the cross-modal correlation feature vector into the sleep analysis model to obtain sleep stage categories and sleep physiological events; wherein, the sleep analysis model includes at least a sleep stage classifier and a sleep event detector.

[0123] Technical approach: Input cross-modal correlation feature vectors into the sleep analysis model to obtain sleep stage categories and sleep physiological events.

[0124] Logical purpose: To achieve the ultimate application goal of the method—to produce specific sleep health indicators.

[0125] Sleep stages are typically categorized as N1, N2, N3 (deep sleep), REM (rapid eye movement), and Wake (awake). Sleep stage classifiers (such as temporal classification models like LSTM, GRU, or 1D-CNN) utilize the aforementioned fused features to classify each time segment (e.g., 30 seconds per frame).

[0126] Sleep physiological event detection refers to the identification of specific sleep disorders or events, such as snoring, sleep apnea, leg movements, and periodic limb movements. Sleep event detectors (which may be one or more binary classification / detection models) are also based on fused features to determine whether a specific event has occurred and when it occurred.

[0127] Integrated output: This step demonstrates that the model can output both phased and event results simultaneously or within a unified framework, reflecting the system's efficiency and comprehensiveness.

[0128] The sleep analysis model includes a hierarchical attention feature allocation module, a sleep stage classification branch, and a sleep physiological event detection branch; the sleep physiological events include at least apnea events, hypopnea events, snoring events, limb movement events, and REM behavioral disorder events.

[0129] Specifically, step 104 includes steps 1041 to 1043: Step 1041: Receive the cross-modal association feature vector through the hierarchical attention feature allocation module, and generate sleep stage-specific feature vectors and sleep physiological event-specific feature vectors; Technical means: Through a hierarchical attention feature allocation module, cross-modal associated feature vectors are received, and sleep stage-specific feature vectors and sleep physiological event-specific feature vectors are generated.

[0130] Logical objective: To resolve the issues of feature sharing and conflict in multi-task learning. While sleep staging (overall state) and specific event detection (local anomalies) are related, they focus on different information.

[0131] "Hierarchical attention": This suggests that the module may use an attention mechanism to reweight and select the fused features of the input at multiple levels or dimensions. For example, it might learn: For sleep staging, more attention needs to be paid to the slow variation trend of heart rate variability (HRV) and the macroscopic rhythm of body movement.

[0132] For the detection of sleep apnea events, more attention needs to be paid to short-term sudden patterns in audio signals and immediate heart rate responses.

[0133] Generate custom feature vectors: The output of this module is no longer the same feature vector, but two feature vectors that have been "customized" and recombined. This is more flexible and efficient than simple hard parameter sharing (sharing the bottom layer and the top layer of the branch) or soft parameter sharing (sharing all parameters), and can effectively prevent negative transfer between tasks (one task interferes with the learning of another task).

[0134] Step 1042: Input the sleep stage-specific feature vector into the sleep stage classification branch to obtain the sleep stage category; Technical approach: Two dedicated feature vectors are input into the sleep stage classification branch and the sleep physiological event detection branch, respectively.

[0135] Logical objective: To achieve parallel and specialized processing of tasks.

[0136] Sleep staging classification branch: This is a multi-classification model (such as N1, N2, N3, REM, Wake) that deals with mutually exclusive category labels that cover the entire time window.

[0137] The sleep physiological event detection branch is a multi-label detection model (which may simultaneously include multiple events). It processes binary labels (such as "presence / absence of apnea" or "presence / absence of leg movement") that may coexist and occur within a time window. The claims explicitly list five core categories of events: apnea, hypopnea, snoring, limb movement, and REM behavioral disorders, thus specifying the scope of protection and enhancing its practicality.

[0138] Parallel processing: allows two tasks to run simultaneously, improving system efficiency and conforming to physiological reality (the simultaneous existence of staged states and events).

[0139] Step 1043: Input the sleep physiological event-specific feature vector into the sleep physiological event detection branch to obtain the sleep physiological event; wherein, both the sleep stage classification branch and the sleep physiological event detection branch adopt the evidence deep network; the evidence deep network obtains the classification result by outputting the evidence vector and parameterizing the Dirichlet distribution.

[0140] Technical means: Both branches use evidence deep networks, and the principle is explained as obtaining classification results by outputting evidence vectors and parameterizing the Dirichlet distribution.

[0141] Logical Purpose: This is the major technical highlight and potential inventiveness of this claim. It addresses the key shortcomings of traditional deep learning models in healthcare applications, namely, their tendency to be "overconfident" or "unable to express uncertainty."

[0142] The limitation of the traditional Softmax classifier: even if the input data is fuzzy or noisy, Softmax will output a distribution with a total probability of 1, giving people the illusion that "the model is very deterministic".

[0143] How Evidential Deep Learning works: Output Evidence Vector: Instead of directly outputting class probabilities, the network outputs an evidence vector. Each evidence value corresponds to a class and is a non-negative number. The larger the evidence value, the more sufficient the observational evidence supporting that class.

[0144] Parameterized Dirichlet Distribution: A Dirichlet distribution is constructed by using the evidence vector as parameters. This distribution describes the model's subjective confidence (i.e., uncertainty) in the classification results.

[0145] Obtaining classification results and uncertainties: From this Dirichlet distribution, we can simultaneously obtain: Predicted probability: The expected value of the distribution, used as the final classification result (e.g., "N3 period, probability 80%").

[0146] Prediction uncertainty (total uncertainty): the reciprocal of the concentration parameter of the distribution (sum of evidence). The larger the sum of evidence, the more concentrated the distribution, and the lower the uncertainty; the smaller the sum of evidence, the flatter the distribution, and the higher the uncertainty.

[0147] Its immense value in sleep monitoring: Reliable decision support: When signal quality is poor (e.g., excessive body movement) or the pattern is atypical, the model can output high uncertainty, prompting the user or system that "this result has low confidence," thus avoiding misleading information. This demonstrates the responsible nature of a medical AI system.

[0148] Data quality self-check: High uncertainty may indicate poor sensor contact or severe environmental interference.

[0149] Active learning: Samples with high uncertainty can be labeled and prioritized for manual review or model iterative training.

[0150] Specifically, the hierarchical attention feature allocation module includes a shared feature encoder, a sleep stage attention subnetwork, and a sleep event attention subnetwork; The sleep stage attention subnetwork and the sleep event attention subnetwork learn from the shared features output by the shared feature encoder and apply different attention weights to extract task-specific feature vectors.

[0151] 1. Shared Feature Encoder: Constructing a Universal Feature Basis Technical means: The module contains a shared feature encoder.

[0152] Logical purpose: This is the foundational layer of the entire allocation module. It receives highly fused cross-modal correlation feature vectors from upstream (the cross-modal correlation feature extraction network).

[0153] "Shared": This means that the encoder's parameters serve both sleep staging and event detection tasks simultaneously during training and inference. Its optimization objective is to minimize the total loss for both tasks.

[0154] Function: This encoder typically consists of several layers of fully connected networks (MLP) or one-dimensional convolutions. Its role is to further abstract, compress, and nonlinearly transform the input features, learning a deeper level of shared feature representation that simultaneously includes both period-related and event-related information. It can be understood as a "common feature pool" of all sleep-related knowledge.

[0155] 2. Dual Attention Subnetworks: Enabling Task-Driven Feature Selection Technical approach: The module includes a sleep stage attention subnetwork and a sleep event attention subnetwork. They learn from the shared features output by the shared feature encoder and apply different attention weights to extract task-specific feature vectors.

[0156] Logical purpose: This is the core mechanism for achieving "hierarchical attention feature allocation". It acknowledges a key fact: although the two tasks share underlying information, their respective focuses are different.

[0157] "Attention Subnetwork": Each subnetwork is a small neural network (usually a shallow MLP) whose output dimension is the same as the dimension of the shared features. Its task is to learn an attention weight vector.

[0158] Working principle: Input: Both subnets receive the same input—that is, the shared features output by the shared feature encoder (assuming it is a feature vector of length D).

[0159] Computational task-specific attention: Sleep staging attention subnetwork: Analyzes shared features and calculates a weight vector. Each element of this vector represents the importance of the corresponding dimension of the shared features in determining the sleep stage (e.g., distinguishing between deep sleep and REM sleep). For example, it might assign a high weight to the feature dimension representing "heart rate trend stability".

[0160] Sleep Event Attention Subnetwork: Analyzes the same shared features, but computes a completely different weight vector. This vector represents the importance of detecting a specific event (such as sleep apnea). For example, it might assign high weights to feature dimensions representing "audio burst energy" and "instantaneous heart rate fluctuations".

[0161] Applying weights and extracting specific features: The learned attention weight vector is multiplied element-wise with the original shared feature vector (Hadamard product).

[0162] This operation is equivalent to applying a "feature filter" to the shared features. High-weighted feature dimensions are enhanced, while low-weighted ones are suppressed.

[0163] The weighted feature vectors are the sleep stage-specific feature vectors and the sleep physiological event-specific feature vectors. They are from the same source but different in nature, and are respectively "modulated" into forms best suited to their respective downstream tasks.

[0164] Shared features are generated by a shared feature encoder, and different attention weights are applied to two attention subnetworks to extract specific feature vectors related to sleep stages and physiological events. This design effectively enhances the model's ability to handle different tasks, thereby improving the accuracy and reliability of sleep analysis.

[0165] Specifically, the sleep analysis model also includes a multi-task consistency verification unit; The multi-task consistency verification unit evaluates whether sleep stage categories and sleep physiological events meet the constraints in the logical consistency rule base based on the logical consistency rule base. If the sleep stage category and the sleep physiological event meet the constraints, then the sleep stage category and the sleep physiological event are used as the sleep stage category and sleep physiological event of the current analysis time window. If the sleep stage category and sleep physiological event do not meet the constraints, then the sleep stage category and sleep physiological event corresponding to the previous analysis time window that met the constraints will be used as the sleep stage category and sleep physiological event for the current analysis time window.

[0166] 1. Multi-task consistency verification unit: Introducing a domain knowledge engine Technical approach: Add a multi-task consistency verification unit to the sleep analysis model and clarify that it is evaluated based on a logical consistency rule base.

[0167] Logical Purpose: To create an independent, rule-based logical verification module. This module does not participate in feature learning or prediction, but rather acts as a **"reasonableness checker"**, reviewing the preliminary prediction results of the two upstream branches (stage classifier and event detector).

[0168] "Multi-task consistency": This emphasizes that the core responsibility of this unit is to ensure that the outputs of the two tasks, sleep stage (macro state) and sleep physiological events (micro events), are logically consistent.

[0169] "Logical Consistency Rule Base": This is the core knowledge source of this unit. It encapsulates hard constraints or strong correlation rules about the relationship between stages and events, summarized from sleep medicine.

[0170] 2. Logical Consistency Rule Base: Concretizing Domain Knowledge Technical means: The rule base contains specific constraints.

[0171] Logical Purpose and Examples: These constraints are crucial for verifying their effectiveness. They are typically deterministic, generally accepted facts about sleep physiology. For example, a rule base might contain: The mutual exclusion rule of events and stages: "REM behavioral disorder events should not occur during the Wake period." This is because RBD events specifically refer to events that occur during REM sleep.

[0172] The stage-specific rule for the occurrence of events: "Snoring, sleep apnea, hypopnea and other respiratory events mainly occur in the N1, N2 and REM sleep stages, and rarely in the N3 (deep sleep) stage." Event timing and intensity constraints: "After a prolonged physical movement event (such as turning over), the sleep stage is more likely to be Wake or N1, and less likely to be N3." The rationale for physiological indicators: "If marked as N3 (deep sleep) stage, the average heart rate should be significantly lower than that during wakefulness, and body movement should be minimal." (This rule may need to be combined with original characteristics, but the underlying idea is consistent).

[0173] Logical value: By explicitly encoding these rules into the system, the AI ​​model not only relies on data-driven approaches but also follows expert knowledge, resulting in outputs that are more in line with medical consensus.

[0174] 3. Verification and Decision Logic: Two-Level Processing Flow Technical approach: Two processing paths are defined - if the constraints are met, the output is directly output; if not, the result of the previous window that meets the constraints is used for substitution or smoothing.

[0175] Logical objective: To provide a clear and actionable error correction strategy.

[0176] Path 1 (Meets the criteria): Use the sleep stage category and the sleep physiological event as the sleep stage category and sleep physiological event for the current analysis time window.

[0177] Note: This is the ideal situation, indicating that the predictions of the machine learning model have passed the "common sense" test and are considered reasonable and credible, and are therefore directly adopted as the final output.

[0178] Path 2 (Not meeting the conditions): Use the sleep stage category and sleep physiological event corresponding to the previous analysis time window that met the constraints as the sleep stage category and sleep physiological event for the current analysis time window.

[0179] Logical explanation: This is a conservative and robust correction strategy. When the model's prediction at the current moment yields a clearly "absurd" result (e.g., predicting snoring during deep sleep), the system does not trust this outlier. Instead, it assumes that sleep states are continuous and therefore replaces the current unreliable state with the most recent state that is considered reasonable (the previous window that conforms to the rules).

[0180] Technical effect: This is essentially a rule-based temporal smoothing or filling. It can effectively filter out "glitch" or "jump" caused by instantaneous signal interference and accidental model misjudgment, making the final output sleep stages and event sequences smoother and more coherent in time, and more in line with physiological reality.

[0181] By comparing the results with a logical consistency rule base, the logical consistency between sleep stage categories and physiological events is ensured. If the current analysis result does not conform to the rules, the model automatically reverts to the previous valid result. This design not only improves the accuracy of the analysis but also enhances the reliability and robustness of the model when handling complex physiological data.

[0182] This rule base contains strong association rules (based on explicit pathophysiological mechanisms) and weak association / statistical association rules (based on clinical epidemiological statistics). All rules are coded in the logical form of "IF-THEN" or "IF-THEN-NOT".

[0183] I. Strong Correlation Rules Based on Sleep Stages and Inherent Physiological Mechanisms of Events Rule 1 (REM phase-specific events): If a rapid eye movement behavior disorder (RBD) event is detected... Then the sleep stage category should be Rapid Eye Movement (REM). Impact of confidence level: If the periodization result is not a REM period, high random uncertainty will be generated.

[0184] Rule 2 (NREM-specific events): If night terrors or sleepwalking events are detected The sleep stage category should be N3 (deep sleep). Impact of confidence level: If the periodization result is REM or N1 / N2, high random uncertainty will be generated.

[0185] Rule 3 (Negative correlation between respiratory events and sleep depth): If an obstructive apnea or hypoventilation event is detected. AND event duration > 30 seconds or blood oxygen decrease > 4%. Then the sleep stage category should not be N3 (deep sleep) (because severe respiratory events often lead to micro-arousals and shallow sleep). Impact of confidence level: If the periodization result is N3, then moderate random uncertainty will be generated.

[0186] Rule 4 (Exclusionary Rule for the Healing Period): If loud, irregular snoring events or periodic leg movement events are detected. Then the sleep stage category should not be waking (these events are more typical in stable sleep). Impact of confidence level: If the staging result is a period of lucidity, then high random uncertainty will be generated.

[0187] II. Statistical Association Rules Based on Event Occurrence Probability Distribution Rule 5 (Period Distribution of Leg Movement Events): IF detected periodic leg movement (PLMS) events. Then, the probability of sleep stage N1 or N2 was significantly higher than that of N3 or REM sleep. Impact of confidence level: If the period is N3, the random uncertainty increases; if it is N1 / N2, the random uncertainty decreases.

[0188] Rule 6 (Micro-awakening stage tendency): IF detected cortical microarousal events (usually inferred from EEG changes following physical or respiratory events). Then, the probability of a sleep stage being N1 or N2 is higher than that of N3. Confidence impact: Used to adjust the confidence level when the period is ambiguous between N1 / N2 and N3.

[0189] Rule 7 (Relationship between heart rate variability and sleep stage): Based on heart rate signal analysis, IF detected events with significantly increased high-frequency heart rate variability (HF-HRV) (associated with parasympathetic activity / REM phase). AND No obvious physical activity was detected Then the probability of the sleep stage being REM increases. Impact of confidence level: If the period is REM, the random uncertainty is reduced; if it is N3, the random uncertainty is increased.

[0190] Rule 8 (Period-related correlation of physical events): If a large-scale physical movement event (such as rolling over) is detected. Then, the probability of the next sleep stage being N1 (the transition period between wakefulness and light sleep) increases. Then this stage should not be stage N3 (deep sleep stage). Confidence level impact: Used to verify the staging results of the first analysis window after body movement.

[0191] III. Composite Rules Based on Multimodal Signal Cross-Validation Rule 9 (Temporal correlation between snoring and respiratory events): IF snoring event detected AND apnea / hypopnea events were detected within the following 10–60 seconds. Then, the sleep stage category during these two events should not be REM (this type of respiratory event pattern is more common in NREM). Impact of confidence level: If the periodization result is a REM period, it will generate high random uncertainty.

[0192] Rule 10 (Elimination of Environmental Interference and False Events): A sudden high-decibel noise event was detected in the ambient audio signal. AND body movement signals synchronously detected large amplitude signals AND heart rate signals did not show characteristic changes associated with sleep events (such as heart rate recovery after apnea). Then any suspected sleep physiological events detected within this time window may be pseudo-events. THEN sleep stage category is likely to be waking or minimally waking. Impact on confidence level: Imposes extremely high random uncertainty on all event detection and staging results within this window, and manual review is recommended.

[0193] Rule 11 (The Relationship Between Body Temperature Trends and Deep Sleep): The IF body temperature signal indicates a continuous downward trend in core body temperature. AND body movement signals indicate low activity levels Then the probability of sleep stage N2 or N3 increases. THEN The probability of detecting frequent micro-awakening events decreases during this stage. Impact of confidence level: As an auxiliary verification, adjust the confidence level of staging and event detection.

[0194] In the embodiments corresponding to steps 1041 to 1043, through feature extraction by the hierarchical attention feature allocation module and the classification and detection capabilities of the evidence depth network, the model can effectively analyze complex sleep data and provide accurate sleep health monitoring results.

[0195] In the embodiments corresponding to steps 101 to 104, multimodal raw signals such as environmental audio signals, heart rate signals, body temperature signals, and body movement signals are synchronously collected by integrating multimodal sensors, overcoming the shortcomings of traditional monitoring methods. Modal feature matrices are extracted using these multimodal signals, and deep processing is performed through a cross-modal correlation feature extraction network, effectively extracting cross-modal correlation features related to sleep states. The sleep health monitoring solution based on a smart belt not only improves the accuracy and reliability of sleep monitoring but also enhances the convenience for users in daily life.

[0196] like Figure 2 This invention provides a sleep analysis device; please refer to [link / reference]. Figure 2 , Figure 2 A schematic diagram of a sleep analysis device provided by the present invention is shown, such as... Figure 2 The sleep analysis device shown includes: The acquisition unit 21 is used to synchronously acquire multimodal raw signals based on the multimodal sensors in the smart belt; wherein, the multimodal raw signals include at least environmental audio signals, heart rate signals, body temperature signals and body movement signals; Extraction unit 22 is used to extract a modal feature matrix composed of multimodal raw signals; wherein, the horizontal elements of the modal feature matrix are multiple feature values ​​corresponding to the same modal raw signal, and the vertical elements of the modal feature matrix are feature values ​​corresponding to different modal raw signals; The first identification unit 23 is used to input the modal feature matrix into the cross-modal correlation feature extraction network for processing to obtain the cross-modal correlation feature vector corresponding to the comprehensive sleep state; The second identification unit 24 is used to input the cross-modal correlation feature vector into the sleep analysis model to obtain sleep stage categories and sleep physiological events; wherein, the sleep analysis model includes at least a sleep stage classifier and a sleep event detector.

[0197] This invention provides a sleep analysis device that overcomes the shortcomings of traditional monitoring methods by integrating multimodal sensors to simultaneously collect multimodal raw signals such as environmental audio signals, heart rate signals, body temperature signals, and body movement signals. Modal feature matrices are extracted from these multimodal signals, and deep processing is performed through a cross-modal correlation feature extraction network to effectively extract cross-modal correlation features related to sleep states. The sleep health monitoring solution based on a smart belt not only improves the accuracy and reliability of sleep monitoring but also enhances the convenience for users in daily life.

[0198] Figure 3 This is a schematic diagram of a terminal device provided in an embodiment of the present invention. Figure 3 As shown, a terminal device 3 in this embodiment includes: a processor 30, a memory 31, and a computer program 32 stored in the memory 31 and executable on the processor 30, such as a sleep analysis program. When the processor 30 executes the computer program 32, it implements the steps in the various sleep analysis method embodiments described above, for example... Figure 1 Steps 101 to 103 are shown. Alternatively, when the processor 30 executes the computer program 32, it implements the functions of each unit in the above-described device embodiments, for example... Figure 2 The function of the unit shown.

[0199] For example, the computer program 32 can be divided into one or more units, which are stored in the memory 31 and executed by the processor 30 to complete the present invention. The one or more units can be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program 32 in the terminal device 3. For example, the specific functions of each unit of the computer program 32 can be divided as follows: The acquisition unit is used to synchronously acquire multimodal raw signals based on the multimodal sensors in the smart belt; wherein, the multimodal raw signals include at least environmental audio signals, heart rate signals, body temperature signals, and body movement signals; An extraction unit is used to extract a modal feature matrix composed of multimodal raw signals; wherein, the horizontal elements of the modal feature matrix are multiple feature values ​​corresponding to the same modal raw signal, and the vertical elements of the modal feature matrix are feature values ​​corresponding to different modal raw signals; The first identification unit is used to input the modal feature matrix into the cross-modal correlation feature extraction network for processing to obtain the cross-modal correlation feature vector corresponding to the comprehensive sleep state; The second identification unit is used to input the cross-modal correlation feature vector into the sleep analysis model to obtain sleep stage categories and sleep physiological events; wherein, the sleep analysis model includes at least a sleep stage classifier and a sleep event detector.

[0200] The terminal device includes, but is not limited to, a processor 30 and a memory 31. Those skilled in the art will understand that... Figure 3 This is merely an example of a terminal device 3 and does not constitute a limitation on a terminal device 3. It may include more or fewer components than shown, or combine certain components, or different components. For example, the terminal device may also include input / output devices, network access devices, buses, etc.

[0201] The processor 30 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0202] The memory 31 can be an internal storage unit of the terminal device 3, such as a hard disk or memory of the terminal device 3. The memory 31 can also be an external storage device of the terminal device 3, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the terminal device 3. Furthermore, the memory 31 can include both internal and external storage units of the terminal device 3. The memory 31 is used to store the computer program and other programs and data required by the roaming control device. The memory 31 can also be used to temporarily store data that has been output or will be output.

[0203] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0204] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of the present invention. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0205] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this invention. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0206] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0207] This invention provides a computer program product that, when run on a mobile terminal, enables the mobile terminal to implement the steps described in the above-described method embodiments.

[0208] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to a photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks.

[0209] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0210] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0211] In the embodiments provided by this invention, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0212] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they may be located in one place or distributed across multiple network units.

[0213] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0214] It should also be understood that the term “and / or” as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0215] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [the described condition or event] is detected" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once [the described condition or event] is detected," or "in response to detection of [the described condition or event]."

[0216] Furthermore, in the description of this invention and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0217] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of the invention include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0218] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for sleep analysis, characterized in that, The sleep analysis method includes: The smart belt uses a multi-modal sensor to synchronously acquire multi-modal raw signals; wherein the multi-modal raw signals include at least environmental audio signals, heart rate signals, body temperature signals, and body movement signals. Extract a modal feature matrix composed of multimodal raw signals; wherein, the horizontal elements of the modal feature matrix are multiple feature values ​​corresponding to the same modal raw signal, and the vertical elements of the modal feature matrix are feature values ​​corresponding to different modal raw signals; The modal feature matrix is ​​input into a cross-modal correlation feature extraction network for processing to obtain a cross-modal correlation feature vector corresponding to the comprehensive sleep state; The cross-modal correlation feature vector is input into the sleep analysis model to obtain sleep stage categories and sleep physiological events; wherein, the sleep analysis model includes at least a sleep stage classifier and a sleep event detector.

2. The sleep analysis method as described in claim 1, characterized in that, The step of extracting the modal feature matrix composed of the original multimodal signals includes: The original multimodal signals are timestamped and divided into analysis time windows of preset duration; For each modal signal in the multimodal raw signal, feature values ​​at different time scales or frequency scales are extracted within the analysis time window to form the feature vector corresponding to that modal signal; wherein, the feature values ​​corresponding to the environmental audio signal include at least the power spectral density, Mel frequency cepstral coefficients, spectral entropy, and / or zero-crossing rate of a specific frequency band; the feature values ​​corresponding to the heart rate signal include at least the average heart rate, time-domain index of heart rate variability, and / or frequency-domain index of heart rate variability; the feature values ​​corresponding to the body motion signal include at least the signal amplitude area, motion energy, and / or the proportion of rest time. The feature vectors corresponding to the original signals of multiple modes are concatenated to form a modal feature matrix; wherein, the multiple data values ​​in the horizontal direction of the modal feature matrix are feature vectors corresponding to the same modal signal.

3. The sleep analysis method as described in claim 1, characterized in that, The cross-modal correlation feature extraction network includes at least a modality-specific encoder layer, a cross-modal correlation attention layer, and a feature fusion layer connected in sequence; Accordingly, the step of inputting the modal feature matrix into a cross-modal correlation feature extraction network for processing to obtain the cross-modal correlation feature vector corresponding to the comprehensive sleep state includes: The modality-specific encoder layer maps the modality feature matrix of each modality to multiple high-level abstract feature vectors corresponding to each modality; wherein, the modality-specific encoder layer includes sub-encoders that correspond one-to-one with each modality; The cross-modal association attention layer is used to model the association between the high-level abstract feature vectors of all modalities; wherein, the cross-modal association attention layer is configured to use the high-level abstract feature vectors of all modalities as input nodes to construct a fully connected modal feature map; Calculate the association attention weights between any two feature vectors of different modalities using graph attention mechanism or Transformer multi-head attention mechanism; Based on the associated attention weights, the feature vectors of each modality are weighted and aggregated to generate a set of feature vectors modulated by information from other modalities; The feature fusion layer concatenates the set of feature vectors and performs dimensionality reduction and fusion through at least one fully connected layer, outputting the cross-modal correlation feature vector corresponding to the comprehensive sleep state within the current analysis time window.

4. The sleep analysis method as described in claim 1, characterized in that, The sleep analysis model includes a hierarchical attention feature allocation module, a sleep stage classification branch, and a sleep physiological event detection branch; the sleep physiological events include at least apnea events, hypopnea events, snoring events, limb movement events, and REM behavioral disorder events; Accordingly, the step of inputting the cross-modal correlation feature vector into the sleep analysis model to obtain sleep stage categories and sleep physiological events includes: The hierarchical attention feature allocation module receives the cross-modal associated feature vector and generates sleep stage-specific feature vectors and sleep physiological event-specific feature vectors. Input the sleep stage-specific feature vector into the sleep stage classification branch to obtain the sleep stage category; The sleep physiological event-specific feature vector is input into the sleep physiological event detection branch to obtain the sleep physiological event; wherein, both the sleep stage classification branch and the sleep physiological event detection branch adopt the evidence depth network; the evidence depth network obtains the classification result by outputting the evidence vector and parameterizing the Dirichlet distribution.

5. The sleep analysis method as described in claim 4, characterized in that, The hierarchical attention feature allocation module includes a shared feature encoder, a sleep stage attention subnetwork, and a sleep event attention subnetwork; The sleep stage attention subnetwork and the sleep event attention subnetwork learn from the shared features output by the shared feature encoder and apply different attention weights to extract task-specific feature vectors.

6. The sleep analysis method as described in claim 4, characterized in that, The sleep analysis model also includes a multi-task consistency verification unit; The multi-task consistency verification unit evaluates whether sleep stage categories and sleep physiological events meet the constraints in the logical consistency rule base based on the logical consistency rule base. If the sleep stage category and the sleep physiological event meet the constraints, then the sleep stage category and the sleep physiological event are used as the sleep stage category and sleep physiological event of the current analysis time window. If the sleep stage category and sleep physiological event do not meet the constraints, then the sleep stage category and sleep physiological event corresponding to the previous analysis time window that met the constraints will be used as the sleep stage category and sleep physiological event for the current analysis time window.

7. The sleep analysis method as described in claim 1, characterized in that, Before the step of extracting the modal feature matrix composed of the original multimodal signals, the method further includes: Calculate the signal-to-noise ratio of the heart rate signal within the characteristic frequency band; Calculate the periodicity index of the peak-to-peak interval sequence of the heart rate signal; Calculate the standard deviation of the vector amplitude of the body motion signal and convert it into a body motion interference index that reflects the risk of motion artifacts; The signal-to-noise ratio index, periodicity index, and motion interference index are weighted and fused to generate the final signal reliability index. When the signal confidence index is lower than a preset threshold, an adaptive filtering algorithm is triggered. The adaptive filtering algorithm uses the body motion signal as reference noise to perform dynamic noise reduction processing on the heart rate signal.

8. The sleep analysis method as described in claim 7, characterized in that, When the signal confidence index is lower than a preset threshold, an adaptive filtering algorithm is triggered. The adaptive filtering algorithm uses the body motion signal as reference noise to perform dynamic noise reduction processing on the heart rate signal, including the following steps: The vector amplitude VM(n) is calculated based on the body motion signal, and the vector amplitude VM(n) is shaped by a linear time-varying filter to generate a reference noise signal r(n); The reference noise signal r(n) and the heart rate signal s_hr_raw(n) are input into an adaptive filter; wherein the weight vector W(n) of the adaptive filter is iteratively updated according to the following function: W(n+1)=W(n)+(μ(n) / (δ+R) T (n)R(n)))*e(n)*R(n); Where, e(n) = s_hr_raw(n) - W T (n)*R(n) represents the error signal at the current time, and R(n) is the input vector composed of r(n) at the current and past times. T (n) represents the transpose of R(n), W T (n) denotes the transpose of W(n), and δ is a constant to prevent division by zero; The step size factor μ(n) of the adaptive filter is dynamically adjusted according to the signal reliability index C(n); where μ(n) = μ_base*(1-α*C(n)), μ_base is the base step size, and α is the adjustment coefficient; The error signal e(n) is output as the heart rate signal after the dynamic noise reduction process.

9. A sleep analysis device, characterized in that, The sleep analysis device includes: The acquisition unit is used to synchronously acquire multimodal raw signals based on the multimodal sensors in the smart belt; wherein, the multimodal raw signals include at least environmental audio signals, heart rate signals, body temperature signals, and body movement signals; An extraction unit is used to extract a modal feature matrix composed of multimodal raw signals; wherein, the horizontal elements of the modal feature matrix are multiple feature values ​​corresponding to the same modal raw signal, and the vertical elements of the modal feature matrix are feature values ​​corresponding to different modal raw signals; The first identification unit is used to input the modal feature matrix into the cross-modal correlation feature extraction network for processing to obtain the cross-modal correlation feature vector corresponding to the comprehensive sleep state; The second identification unit is used to input the cross-modal correlation feature vector into the sleep analysis model to obtain sleep stage categories and sleep physiological events; wherein, the sleep analysis model includes at least a sleep stage classifier and a sleep event detector.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the sleep analysis method as described in any one of claims 1 to 8.

Images