Sleep Disorder Assessment Methods Based on Basic Audio Models

Abstract

This invention discloses a sleep disorder assessment method based on a fundamental audio model. The steps include: using a self-supervised learning model as the backbone network of a sleep sound fundamental model, and adding a parameter-efficient fine-tuning module to the Transformer module of the sleep sound fundamental model; pre-training the sleep sound fundamental model using sleep sound data samples, adding a neural network module corresponding to the sleep disorder assessment task as the downstream task head to the pre-trained sleep sound fundamental model to obtain an assessment model for the sleep disorder assessment task; freezing the pre-trained part of the assessment model and using sleep sound data samples to train the assessment model for the downstream task; acquiring real-time sleep sound data and inputting it into the trained assessment model to obtain the corresponding sleep disorder assessment result. This invention improves the applicability of sound signals in sleep analysis tasks while reducing computational resource requirements.

Description

Technical Field

[0001] This invention relates to the field of audio analysis, and more specifically to a method for assessing sleep disorders based on a basic audio model. Background Technology

[0002] Sleep disorders (such as sleep apnea and insomnia) have become a significant health problem affecting millions of people worldwide, and in severe cases, can lead to health risks such as cardiovascular disease, hypertension, and memory decline. Therefore, accurate monitoring and assessment of sleep status is crucial for improving individual health. Currently, clinical sleep monitoring mainly relies on polysomnography (PSG). While this method is highly accurate, it requires hospital or sleep laboratory testing, is costly and complex, and is greatly affected by environmental factors, making it unsuitable for large-scale home sleep monitoring.

[0003] In recent years, sound-based sleep analysis methods have gradually gained attention. These methods infer sleep states by analyzing physiological sound signals such as snoring and breathing, offering a potential solution for non-invasive, low-cost home sleep monitoring. However, existing sound-based sleep analysis methods still face the following technical challenges: poor model generalization ability—most current models are task-specific and difficult to adapt to various sleep analysis tasks; significant differences in sound data across different environments also affect the model's applicability in new scenarios; high data annotation costs—deep learning models require a large amount of labeled data, while high-quality sleep sound data annotation relies on expert manual analysis, which is time-consuming and expensive; high computational resource requirements—existing deep learning models are computationally intensive and difficult to run efficiently on resource-constrained mobile devices, limiting their real-time monitoring capabilities; and noise interference affecting analysis results—background noise in the home environment (such as fan noise, partner snoring, etc.) affects the model's accuracy, making it difficult to stably apply to complex environments. Therefore, current sound-based sleep monitoring technology still needs improvement to enhance its accuracy, generalization ability, and applicability. Summary of the Invention

[0004] The technical problem to be solved by this invention is to provide a sleep disorder assessment method based on a basic audio model, which addresses the above-mentioned problems in the prior art. By constructing a sleep sound basic model based on self-supervised learning, the applicability of sound signals in sleep analysis tasks is improved, the generalization ability of the model is enhanced, and the parameters of the sleep sound basic model are adjusted using an efficient parameter fine-tuning strategy, thereby optimizing computational efficiency and model performance.

[0005] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0006] A sleep disorder assessment method based on a fundamental audio model includes the following steps:

[0007] A self-supervised learning model is used as the backbone network of the sleep sound base model, enabling the sleep sound base model to learn feature representations from unlabeled sleep audio. At the same time, an efficient parameter fine-tuning module is added to the Transformer module of the sleep sound base model to enhance the model's performance on specific tasks.

[0008] The sleep sound base model is pre-trained using sleep sound data samples. According to the type of sleep disorder assessment task, a corresponding neural network module is added to the pre-trained sleep sound base model as the head of the downstream task to obtain the assessment model corresponding to the sleep disorder assessment task. Then, the pre-trained part of the assessment model is frozen and the assessment model is trained for the downstream task using sleep sound data samples.

[0009] Real-time sleep sound data is acquired and input into a trained assessment model to obtain the corresponding sleep disorder assessment results.

[0010] Furthermore, the self-supervised learning model adopts the Wav2vec 2.0 model. When the sleep sound base model learns feature representations from unlabeled sleep audio, it includes the following steps:

[0011] A hidden representation is obtained by extracting temporal and local contextual features from the raw audio data using a feature extractor.

[0012] Randomly mask the hidden representation to generate masking blocks corresponding to the masked parts of the hidden representation.

[0013] The quantization module is used to generate corresponding discrete pseudo-labels for the masked parts in the hidden representation and these labels are used as the training targets.

[0014] Input all unmasked hidden representations and the masking blocks corresponding to the masked hidden representations into the Transformer module;

[0015] The Transformer module uses contextual information to predict the hidden representation content corresponding to the masked block, and trains the entire model by predicting discrete pseudo-labels, ultimately outputting the context-enhanced representation corresponding to the hidden representation.

[0016] Furthermore, when adding a parameter fine-tuning module to the Transformer module of the sleep sound base model, the following steps are included:

[0017] An adapter module is embedded before and after the feedforward neural network of each Transformer layer in the Transformer module. The adapter module consists of a lower projection layer, a non-linear activation function, and an upper projection layer, and has skip connections to retain key information, as shown in the following expression:

[0018] A(x) = x + U(f(D(x)))

[0019] Where A(x) represents the adapter function, x represents the input data, D(·) represents the downward projection, f(·) represents the nonlinear activation function, and U(·) represents the upward projection.

[0020] Furthermore, when adding an efficient parameter fine-tuning strategy module to the Transformer module of the sleep sound base model, the following steps are included:

[0021] In the multi-head self-attention layer of the Transformer module, a prefix tuning module is introduced into both the key and value tensors. This prefix tuning module generates a learnable prefix encoding matrix for the corresponding key or value through a linear layer, and then concatenates the learnable prefix encoding matrix with the corresponding key or value through a prefix multilayer perceptron layer to calculate the concatenated key or value matrix, as shown in the following expression:

[0022] K′=[P k ;K]

[0023] V′=[P v ;V]

[0024] Where K′ represents the concatenated key matrix, V′ represents the concatenated value matrix, K represents the key tensor of the multi-head self-attention layer, V represents the value tensor of the multi-head self-attention layer, and P k ∈R l×d and P v ∈R l×d Let l be the learnable prefix encoding matrix for the key and value, respectively, where l is the prefix length, d is the hidden dimension, and [;] denotes a connection along the sequence length dimension.

[0025] Furthermore, when adding an efficient parameter fine-tuning strategy module to the Transformer module of the sleep sound base model, the following steps are included:

[0026] In the multi-head self-attention layer of the Transformer module, corresponding low-rank adaptation modules are added to the query weight matrix and the key weight matrix, respectively. The low-rank adaptation module updates the corresponding query weight matrix or key weight matrix by parameterizing it into the product of two low-rank matrices using two trainable matrices, resulting in the updated query weight matrix or key weight matrix, as shown in the following expression:

[0027] W′=W+αAB

[0028] Where W′ represents the updated query weight matrix or key weight matrix, W represents the query weight matrix or key weight matrix, A and B represent low-rank matrices respectively, and α is the scaling factor.

[0029] Furthermore, when pre-training the sleep sound base model using sleep sound data samples, the following steps are included:

[0030] Freeze the feature extractor of the sleep sound base model, while keeping the quantization module and parameter fine-tuning module of the sleep sound base model trainable;

[0031] After randomly masking a portion of the features of the sleep sound data samples, they are input into the sleep sound base model for pre-training. During the pre-training process, the loss function between the predicted value of the masked portion and the original value output by the sleep sound base model is calculated. Based on the loss function, the parameters of the trainable part of the sleep sound base model are adjusted, and the parameters of all normalized layers of the sleep sound base model are updated until the loss function meets the requirements.

[0032] Furthermore, the expression for the loss function is as follows:

[0033]

[0034] Where M is the set of occluded portions in the sleep sound data samples, x'i is the input of the occluded portion, and f θ This represents the predicted output of the sleep sound baseline model. is the positive vectorized latent representation corresponding to the original value xi of the occluded part, q is an element in the set Q of all quantized representations of the original values ​​of the occluded parts, sim(·,·) represents the similarity function, and κ represents the hyperparameter.

[0035] Furthermore, when adding corresponding neural network modules to the pre-trained sleep sound baseline model as the head of downstream tasks based on the type of sleep disorder assessment task, this includes:

[0036] If the sleep disorder assessment task is a sleep apnea detection task, a neural network module with higher structural complexity should be selected as the head of the downstream task.

[0037] If the sleep disorder assessment task is snoring detection, sleep position recognition, or sleep stage segmentation, a neural network module with lower structural complexity should be selected as the head of the downstream task.

[0038] Furthermore, when freezing the pre-trained portion of the evaluation model and training the evaluation model for downstream tasks using sleep sound data samples, the following steps are included:

[0039] The sleep sound base model portion of the evaluation model is frozen, and the sleep sound data samples are sliced, labeled, and then input into the evaluation model for downstream task training. During the downstream task training, the sleep sound base model portion of the evaluation model extracts contextual features from the input audio and inputs the extracted features into the neural network module for classification or regression. The loss function between the classification or regression prediction value output by the neural network module and the original label value is calculated, and the parameters of the neural network module are adjusted according to the loss function until the loss function meets the requirements.

[0040] Furthermore, before acquiring real-time sleep sound data and inputting it into the trained evaluation model, the following steps are also included:

[0041] Model quantization technology is used to convert the weights and activations in the evaluation model from floating-point format to low-precision format, and the trained evaluation model is converted into a format supported by mobile devices, and adaptation tests are performed.

[0042] Integrate the inference framework into mobile device applications and load the converted model files;

[0043] When acquiring real-time sleep sound data and inputting it into a trained evaluation model, the process includes: using the audio acquisition module of a mobile device to acquire the sleeper's real-time audio signal, inputting the real-time audio signal into the evaluation model, the sleep sound basic model part of the evaluation model extracting features from the real-time audio signal and inputting the extracted features into the neural network module for classification or regression, and using the classification or regression result as the corresponding sleep disorder evaluation result.

[0044] Compared with the prior art, the advantages of the present invention are as follows:

[0045] This invention uses a self-supervised learning model as the backbone network of a sleep sound base model, enabling the sleep sound base model to capture rich semantic representations from unlabeled sleep audio using self-supervised learning methods. Furthermore, through pre-training, it can maintain high generalization ability in multiple sleep analysis tasks while reducing dependence on labeled data.

[0046] This invention employs a parameter-efficient fine-tuning strategy, adding a relevant parameter-efficient fine-tuning module to the Transformer module of the sleep sound basic model. This enhances the model's performance on specific tasks, enabling it to run efficiently even in environments with limited computing resources (such as mobile devices). Compared to traditional deep learning models, this significantly reduces computational costs while maintaining high performance. Attached Figure Description

[0047] Figure 1 This is a flowchart of a method according to an embodiment of the present invention.

[0048] Figure 2 This is a schematic diagram illustrating the working principle of the sleep sound basic model in an embodiment of the present invention.

[0049] Figure 3 This is a schematic diagram of the Transformer module improved by employing a parameter fine-tuning strategy in an embodiment of the present invention.

[0050] Figure 4 This is a schematic diagram illustrating the process of model training in stages according to an embodiment of the present invention. Detailed Implementation

[0051] The present invention will be further described below with reference to the accompanying drawings and specific preferred embodiments, but this does not limit the scope of protection of the present invention.

[0052] Before introducing specific embodiments of the present invention, relevant concepts will be explained.

[0053] Self-supervised learning: Self-supervised learning is an unsupervised learning method that learns the representation of data by utilizing the structure of the data itself to generate supervisory signals. This method does not require externally labeled data; instead, it allows the model to automatically learn the intrinsic features of the data by designing prediction tasks.

[0054] Parameter-Efficient Fine-Tuning Strategy: Parameter-Efficient Fine-Tuning (PEFT) is a fine-tuning technique for pre-trained models that aims to improve the performance of a model on a specific downstream task by adjusting only a small portion of the model's parameters.

[0055] Example

[0056] This embodiment proposes a sleep disorder assessment method based on a basic audio model. It constructs a basic sleep sound model based on self-supervised learning and a strategy for efficient parameter fine-tuning, which is referred to as S2FM in this embodiment. This model improves the adaptability and generalization ability of the model while reducing computational costs. It is suitable for mobile devices and provides an economical, efficient, and non-invasive solution for convenient and efficient sleep health monitoring.

[0057] like Figure 1 As shown, the method in this embodiment includes the following steps:

[0058] S101) Model building stage: The self-supervised learning model is used as the backbone network of the sleep sound base model, so that the sleep sound base model learns feature representations from unlabeled sleep audio. At the same time, an efficient parameter fine-tuning module is added to the Transformer module of the sleep sound base model to enhance the performance of the model on specific tasks.

[0059] S102) Model training stage: The sleep sound basic model is pre-trained using sleep sound data samples. According to the type of sleep disorder assessment task, a corresponding neural network module is added to the pre-trained sleep sound basic model as the head of the downstream task to obtain the assessment model corresponding to the sleep disorder assessment task. Then, the pre-trained part of the assessment model is frozen and the assessment model is trained for the downstream task using sleep sound data samples.

[0060] S103) Model application stage: Acquire real-time sleep sound data and input it into the trained evaluation model to obtain the corresponding sleep disorder evaluation results.

[0061] Through the above steps, this embodiment pre-trains the sleep sound basic model, enabling it to learn semantically rich feature representations from a large number of unlabeled sleep audio recordings and extract deep semantic information. Then, for actual sleep disorder assessment tasks, different downstream specific function heads are configured for the sleep sound basic model and downstream task training is performed, so that the finally trained assessment model can perform efficient assessments of various specific sleep disorder assessment tasks (such as sleep apnea, sleep staging, snoring detection, etc.), thus effectively applying it to multiple sleep-related tasks such as sleep apnea detection, sleep staging, and snoring detection.

[0062] The following provides a detailed explanation of each step.

[0063] Existing methods have significant limitations when processing sleep audio data. On the one hand, models relying on handcrafted features have weak generalization ability and struggle to adapt to different individuals and complex noise environments. On the other hand, deep learning-based supervised methods heavily rely on large amounts of labeled data, resulting in high acquisition costs. Furthermore, current models generally ignore the characteristics of sparse events and large signal differences in sleep audio, leading to poor stability and sensitivity to changes in data distribution in practical applications. In particular, regarding general feature learning, there is a lack of effective methods to extract high-quality representations from large-scale unlabeled data, limiting the model's applicability in different tasks and scenarios.

[0064] To address the aforementioned issues, in step S101 of this embodiment, a self-supervised learning (SSL) model is adopted as the basic architecture of the sleep sound base model. This allows the sleep sound base model to learn general features from a large amount of unlabeled sleep sound data using a mask prediction training method. Then, by optimizing the contrastive learning objective, the model's ability to distinguish between different types of sleep sounds is improved, thereby enhancing the model's generalization ability and robustness.

[0065] In this embodiment, the self-supervised learning model adopts the Wav2vec 2.0 model. When the sleep sound base model learns feature representations from unlabeled sleep audio, such as... Figure 2As shown, it includes the following steps:

[0066] S201) Original audio encoding: Temporal features and local contextual features are extracted from the original audio data X by a feature extractor to obtain the hidden representation Z;

[0067] In this embodiment, the feature extractor uses a 7-layer CNN network, such as... Figure 2 As shown, the original audio data X is encoded by a 7-layer CNN network using the function f(·), resulting in a hidden representation Z = {z1, z2, ..., z7} output by each layer. These representations preserve temporal features and local contextual features.

[0068] S202) Masking process: Random masking process is performed on the hidden representation Z to generate masking blocks corresponding to the masked parts in the hidden representation;

[0069] like Figure 2 As shown, in this embodiment, M consecutive frames are masked at selected p time steps to form a corresponding masked block MSK. The content of the masked position corresponding to the masked block MSK will not be input into the Transformer module, but will be predicted by the model.

[0070] S203) Generating pseudo-labels: Using the quantization module, generate corresponding discrete pseudo-labels for the masked parts in the hidden representation and use them as training targets;

[0071] like Figure 2 As shown, for the masked portions z2, z3, z6, a quantization module is used to generate discrete pseudo-labels Q = {q2, q3, q6} from the codebook. These pseudo-labels serve as the training targets. The quantization module can employ masking and quantization methods proposed in the Wav2vec2.0 model, or it can be replaced with a functionally equivalent structure according to actual needs.

[0072] S204) Context modeling: Input the masking blocks MSK corresponding to all unmasked and masked hidden representations of the hidden representation Z into the Transformer module. The Transformer module uses context information to predict the hidden representation content corresponding to the masking blocks MSK and trains the whole model by predicting discrete pseudo-labels.

[0073] S205) Output context representation: The Transformer module finally outputs the context augmentation representation C = {c1,…,c7} corresponding to all parts of the hidden representation Z. The context augmentation representation serves as the basic model for sleep sounds. It learns feature representations from unlabeled sleep audio and can be used for downstream tasks, such as sleep event classification.

[0074] Applying self-supervised learning models to sleep analysis typically requires substantial computational resources for training. Current sleep sound analysis techniques generally rely on Full Fine-Tuning (FFT), which involves adjusting all parameters of the pre-trained model for a specific sleep analysis task (such as sleep apnea detection or sleep staging). However, this approach suffers from several drawbacks: high computational resource consumption, poor task adaptability, and susceptibility to overfitting to small sample data.

[0075] To address the aforementioned issues, in step S101 of this embodiment, a parameter efficient fine-tuning (PEPT) strategy is introduced into the Transformer module of the sleep sound base model. While maintaining the main structure of the pre-trained sleep sound base model, only a small number of parameters need to be adjusted during the pre-training stage, without requiring extensive retraining. This reduces the demand for computational resources, improves computational efficiency, and enhances task adaptability, enabling it to be efficiently transferred to different sleep analysis tasks.

[0076] like Figure 3 As shown in this embodiment, when adding a parameter-efficient fine-tuning module to the Transformer module of the sleep sound base model, specifically, one or more of the following modules are added: Adapter, Low-Rank Adaptation (LoRA), and Prefix Tuning. These added modules enhance the performance of the sleep sound base model on specific tasks (such as sleep apnea analysis, sleep staging, and snoring detection) without requiring extensive retraining. This optimizes the pre-trained model, improves computational efficiency, and enhances task adaptability. Specifically, the following steps are included:

[0077] S301) When introducing an adapter module, such as Figure 3 As shown in section (a), in this embodiment, an adapter module is embedded before and after the feedforward neural network of each Transformer layer in the Transformer module. The adapter module in this embodiment is as follows: Figure 3 As shown in section (b), it consists of a lower projection layer, a nonlinear activation function, and an upper projection layer, with skip connections to preserve key information, as expressed below:

[0078] A(x)=x+U(f(D(x)))(1)

[0079] Where A(x) represents the adapter function, x represents the input data, D(·) represents the downward projection, f(·) represents the nonlinear activation function (e.g., ReLU), and U(·) represents the upward projection.

[0080] In the adapter module of this embodiment, the lower projection layer reduces the dimension of the input (R).d →R m ) to achieve efficient computation, while the up-projection layer restores the original dimension (R m →R d ), and m < d is the dimension after dimensionality reduction. This design enables the model to efficiently learn task-specific features while maintaining the general capabilities of the pre-trained model.

[0081] In step S301 of this embodiment, the introduced adapter module belongs to an improved version of the standard bottleneck structure, with multiple innovations in structure and deployment method: on the one hand, by embedding an adapter module at the front and back positions of the feed-forward neural network of the Transformer, more abundant context modeling is achieved (compared with the traditional method of only inserting after the feed-forward neural network); on the other hand, the module structure maintains an efficient down-projection-activation-up-projection path, but the insertion position is more flexible and the coverage is wider. At the same time, each adapter module adopts a residual connection method to fully integrate the adapter features with the main branch information.

[0082] S302) When introducing the prefix tuning module, as Figure 3 shown in part (a) of Figure 3 , in this embodiment, the prefix tuning module is introduced into both the key (Key) and value (Value) tensors of the multi-head self-attention (MHSA) layer of the Transformer module. The prefix tuning module of this embodiment is as

[0083] shown in part (c) of k . A learnable prefix encoding matrix corresponding to the key or value is generated through a linear layer, and then the learnable prefix encoding matrix is connected to the corresponding key or value through a prefix multi-layer perceptron (MLP) layer to calculate the connected key or value matrix. The expression is as follows:

[0084] K′ = [P v ; K](2)

[0085] V′ = [P k ; V](3)

[0085] where K′ represents the connected key matrix, V′ represents the connected value matrix, K represents the key tensor of the multi-head self-attention layer, V represents the value tensor of the multi-head self-attention layer, P k ∈R l×d and P v ∈R l×d are the learnable prefix encoding matrices for the key and value respectively, where l is the prefix length and d is the hidden dimension, and [;] represents concatenation along the sequence length dimension.

[0086] Compared to traditional prefix tuning methods, the prefix tuning module introduced in step S302 of this embodiment maintains the main model weights frozen by modifying only the prefix parameters, thereby achieving efficient task adaptation. Traditional prefix tuning methods typically learn a fixed, task-independent set of prefix vectors and concatenate them to the Key and Value of the multi-head attention mechanism in the Transformer to guide the model's attention distribution. While this approach has a small number of parameters, the prefixes are independent of the input content, limiting their expressive power and making it difficult to adapt to complex tasks.

[0087] When introducing a low-rank adapter module (S303), such as Figure 3 As shown in section (a), the weight matrix W of the query in the multi-head self-attention (MHSA) layer of the Transformer module in this embodiment is... q The weight matrix W of the key k Each branch is added as a corresponding low-rank adaptation module. In this embodiment, the low-rank adaptation module is as follows: Figure 3 As shown in section (d), a low-rank decomposition is introduced in the corresponding branch, consisting of two trainable matrices, to capture task-specific adaptability while keeping the original weight matrix unchanged. The low-rank adaptation module updates the corresponding query weight matrix or key weight matrix parameterized as the product of the two low-rank matrices using the two trainable matrices, obtaining the updated query weight matrix or key weight matrix, as expressed below:

[0088] W′=W+αAB (4)

[0089] Where W′ represents the updated query weight matrix or key weight matrix, W∈R d×k This represents the weight matrix of the query or the weight matrix of the key, A∈R d×r and B∈R r×k Let r and d represent low-rank matrices, respectively, r << min(d,k), and α be the scaling factor.

[0090] As can be seen, in step S101 of this embodiment, an efficient and adaptive sleep sound basic model is created based on the self-supervised learning model. By combining the above-mentioned parameter efficient fine-tuning (PEPT) strategy, only a small number of parameters are adjusted while keeping the main structure of the model unchanged, thereby achieving the following improvements: reducing the demand for computing resources and improving adaptability; optimizing the small sample learning ability and reducing the dependence on large-scale labeled data; balancing computing efficiency and inference accuracy and optimizing mobile deployment.

[0091] like Figure 4 As shown, step S102 in this embodiment consists of two stages: the pre-training stage and the downstream task training stage.

[0092] In this embodiment, during the pre-training phase, the parameters of the parameter efficient fine-tuning (PEPT) module, the quantization module, and the task-related layer normalization (LayerNorm) layer of the sleep sound base model are trained. This strategy significantly reduces the number of trainable parameters while maintaining the model's ability to capture specific sleep features.

[0093] In the downstream task training phase, the training objective of this embodiment is to train the model parameters of the downstream task head with the added specific functions. The backbone network of the pre-trained and optimized sleep sound base model has its parameters frozen during the downstream task training phase and is not updated. This allows the pre-trained sleep sound base model to extract deep features from sleep sound data and input them into the task-specific head network for further analysis.

[0094] Correspondingly, in step S102 of this embodiment, when pre-training the sleep sound base model using sleep sound data samples, the following steps are included:

[0095] S501) Freeze the feature extractor of the sleep sound base model, while keeping the quantization module and parameter fine-tuning module of the sleep sound base model trainable, specifically:

[0096] In this embodiment, it is assumed that the CNN feature extractor f(·) originally designed for automatic speech recognition (ASR) tasks in the Wav2vec 2.0 model is still applicable to sleep-related tasks. Therefore, the feature extractor is frozen to leverage transfer learning from the speech domain to sleep sounds and improve the applicability of the model.

[0097] Maintaining the trainability of the quantization module allows it to adapt to unique characteristics of sleep sounds, such as weak signals and sparse events, thereby enhancing the model's masking prediction capabilities. The quantization module performs vector quantization on unmasked audio segments to generate discrete pseudo-labels as prediction targets, and through joint training with the PEPT module, achieves more stable and effective context learning.

[0098] When the parameters of the parameter efficient fine-tuning (PEPT) module (including one or more of the adapter module, low-rank adaptation module, and prefix tuning module) are trainable, the parameter efficient fine-tuning module introduces a variety of parameter efficient fine-tuning mechanisms in the Transformer structure. It can effectively model the input sleep audio context and continuously adjust the attention distribution with the help of self-supervised signals, so that the model can better adapt to the sleep sound analysis task and improve the generalization ability in different groups, scenarios and noise environments.

[0099] S502) Preprocess the sleep sound data samples. In this embodiment, the sleep sound data samples are preprocessed by collecting various sleep sound datasets (such as PSG-Audio, MPSSC, Snoring, etc.), cleaning and denoising the data, and then unifying the audio sampling rate and converting it into a feature format suitable for deep learning models (such as Mel spectrum, MFCC, etc.).

[0100] S503) After randomly masking a portion of the features of the preprocessed sleep sound data samples, the data is input into the sleep sound base model for pre-training. In this implementation, the pre-training objective follows a similar masking prediction task to BERT, but the sleep sound data has been adjusted. Given an input sequence X = [x1, ..., x...] of a set of sleep sound features... T The task involves randomly masking a subset of features and training a model to predict the original values ​​of these masked features. The loss function for this task is:

[0101]

[0102] Where M is the set of occluded portions in the sleep sound data samples, x'i is the input of the occluded portion, and f θ This represents the predicted output of the sleep sound baseline model. is the positive vectorized latent representation corresponding to the original value xi of the occluded part, q is an element in the set Q of all quantized representations of the original values ​​of the occluded parts, sim(·,·) represents the similarity function, and κ represents the hyperparameter.

[0103] During the pre-training process, the loss function between the predicted value of the masked part of the sleep sound base model output and the original value is calculated according to formula (5). The parameters of the trainable part of the sleep sound base model are adjusted according to the loss function. At the same time, the parameters of all normalized layers of the sleep sound base model are updated to enhance the convergence and stability of the model when adapting to specific tasks. This allows the model structure to have a stronger task adaptability while maintaining consistency. When the loss function meets the requirements (such as the loss function converges), the pre-training ends and the pre-trained sleep sound base model is obtained.

[0104] By employing the pre-training steps described above, the computational overhead is reduced while maintaining the sleep sound base model's ability to effectively represent and learn various sleep-related tasks, making it an efficient sleep sound analysis solution for resource-constrained environments.

[0105] In step S102 of this embodiment, before downstream task training, when adding a corresponding neural network module to the pre-trained sleep sound base model as the head of the downstream task according to the type of sleep disorder assessment task, the following steps are included:

[0106] If the sleep disorder assessment task is a sleep apnea detection task, a neural network module with higher structural complexity should be selected as the head of the downstream task.

[0107] If the sleep disorder assessment task is snoring detection, sleep position recognition, or sleep stage segmentation, a neural network module with lower structural complexity should be selected as the head of the downstream task.

[0108] Specifically, in the sleep apnea detection task, to accurately capture the temporal dependence and complex change patterns of apnea events, this embodiment selects the relatively complex EBranchFormer as the task header. EBranchFormer has strong temporal modeling and feature aggregation capabilities, and can fully mine the contextual information and potential abnormal breathing features in the speech signal, thereby improving the detection accuracy and robustness of apnea events.

[0109] For tasks such as snoring detection, sleep position recognition, and sleep stage segmentation, the tasks themselves are relatively simple, the target space is small, and the data scale is limited. Continuing to use highly complex structures like EBranchFormer can easily lead to overfitting and decreased generalization performance. Therefore, this study employs a more streamlined RNN (Recurrent Neural Network) model as the functional head for these tasks. RNNs can not only process time-series data but also model key features with fewer parameters, thus effectively balancing model performance and training stability.

[0110] In step S102 of this embodiment, downstream task training is performed, that is, when the pre-trained part of the evaluation model is frozen and the evaluation model is trained on the downstream task using sleep sound data samples, the following steps are included:

[0111] S601) Freeze the part of the sleep sound base model in the evaluation model. Specifically, freeze the backbone feature extraction network (including CNN feature extractor, Transformer module and the parameter fine-tuning module introduced therein) in the sleep sound base model after pre-training to avoid repeated optimization of backbone parameters in the downstream task training stage, thereby reducing the number of deployed parameters and storage requirements, and improving the model inference efficiency.

[0112] (S602) The sleep sound data samples preprocessed in step S502 are segmented and labeled, and then input into the evaluation model for downstream task training. During the downstream task training, the sleep sound base model acts as a feature extractor. The input audio first undergoes context feature extraction through the frozen S2FM backbone network. The feature extractor consists of a pre-trained CNN feature extractor of the sleep sound base model and a Transformer module, based on the Wav2vec2 architecture. All parameters are frozen during the downstream task training phase, thus preserving its general speech representation capabilities learned in the pre-training phase.

[0113] The sleep sound base model of the evaluation model extracts contextual features from the input audio and inputs the extracted high-dimensional features into a neural network module designed specifically for each task for classification or regression. After obtaining the classification or regression prediction value output by the neural network module, the loss function between the classification or regression prediction value output by the neural network module and the original label value can be calculated. The loss function includes, but is not limited to, mean squared error, mean absolute error, cross-entropy loss, etc. The parameters of the neural network module are adjusted according to the calculation result of the loss function. When the loss function meets the requirements (i.e., the loss function converges), the training of the downstream task ends and the trained evaluation model is obtained.

[0114] In step S103 of this embodiment, when acquiring real-time sleep sound data and inputting it into the trained evaluation model, the evaluation model is specifically deployed on a mobile device. The audio acquisition module of the mobile device is used to acquire the sleeper's real-time audio signal, which is then input into the evaluation model. The sleep sound basic model part of the evaluation model extracts features from the real-time audio signal and inputs the extracted features into the neural network module for classification or regression. The classification or regression result is used as the corresponding sleep disorder evaluation result.

[0115] To achieve efficient operation of the S2FM model on mobile devices, this embodiment employs multiple model compression and optimization strategies, significantly reducing the model's parameter size and inference overhead, ensuring good sleep sound analysis performance even on resource-constrained devices. Specifically, QLoRA (Quantization Low-Rank Adaptation) technology is used to quantize the model, reducing storage and computational overhead, enabling it to run efficiently on mobile devices. Then, testing is conducted in conjunction with mobile applications to optimize real-time performance and improve user experience. Correspondingly, in step S103 of this embodiment, before acquiring real-time sleep sound data and inputting it into the trained evaluation model, the following steps are also included:

[0116] The S701 significantly reduces model size, accelerates inference, and reduces device power consumption by employing model quantization techniques (such as Post-training Quantization or QuantizationAware Training) to convert the weights and activations in the model from floating-point (FP32) to low-precision formats (such as INT8 or FP16).

[0117] S702) Convert the trained evaluation model into a format supported by mobile devices. For example, convert the PyTorch model to TorchScript (.pt) format for PyTorch Mobile; or export it to ONNX format and then convert it to TFLite (.tflite) or CoreML format for Android and iOS inference. Maintain model structure consistency during the conversion process and perform adaptation testing.

[0118] S703) integrates an inference framework into the application of the mobile device, loads the converted model file, and then combines it with the audio acquisition module of the mobile device to input real-time audio signals into the model and perform inference, outputting the prediction results of sleep events.

[0119] In summary, this invention provides a sleep disorder assessment method based on a fundamental audio model. It constructs a sleep sound fundamental model based on self-supervised learning and an efficient parameter fine-tuning strategy for the mobile assessment of sleep disorders. This model utilizes self-supervised learning to capture rich semantic representations and introduces an efficient parameter fine-tuning strategy to effectively reduce computational and parameter costs to address the high computational demands of pre-training. Compared with existing technologies, this invention uses unlabeled data for pre-training, improving the model's generalization ability, reducing dependence on large-scale labeled data, and enhancing the feasibility of practical applications. It has the following advantages:

[0120] 1. A sleep sound basic model based on self-supervised learning is proposed. By pre-training on unlabeled data, the generalization ability of the model is improved and the dependence on labeled data is reduced.

[0121] 2. Introducing the Parameter Efficient Fine-Tuning (PEPT) strategy, which combines adapter, low-rank adaptation (LoRA), and prefix tuning, allows for adaptation to different tasks with only a few parameters adjusted, significantly reducing computational resource requirements.

[0122] 3. Construct a multi-task adaptive framework applicable to various sleep analysis tasks, supporting sleep apnea detection, sleep staging, and snoring classification in home environments, improving the convenience and accessibility of sleep health monitoring, and enhancing the applicability and transferability of the model.

[0123] 4. Optimize model deployment for mobile devices, and use QLoRA technology for model quantization to reduce storage and computing costs and achieve efficient real-time inference.

[0124] 5. Enhance the model's noise resistance by combining data augmentation and preprocessing techniques to improve the accuracy of sleep sound analysis in complex environments and enhance the feasibility of practical applications.

[0125] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A sleep disorder assessment method based on a basic audio model, characterized in that, Includes the following steps: A self-supervised learning model is used as the backbone network of the sleep sound base model, enabling the sleep sound base model to learn feature representations from unlabeled sleep audio. At the same time, an efficient parameter fine-tuning module is added to the Transformer module of the sleep sound base model to enhance the model's performance on specific tasks. The sleep sound base model is pre-trained using sleep sound data samples. According to the type of sleep disorder assessment task, a corresponding neural network module is added to the pre-trained sleep sound base model as the head of the downstream task to obtain the assessment model corresponding to the sleep disorder assessment task. Then, the pre-trained part of the assessment model is frozen and the assessment model is trained for the downstream task using sleep sound data samples. Acquire real-time sleep sound data and input it into a trained assessment model to obtain the corresponding sleep disorder assessment results; The self-supervised learning model uses the Wav2vec 2.0 model. When the sleep sound base model learns feature representations from unlabeled sleep audio, it includes the following steps: A hidden representation is obtained by extracting temporal and local contextual features from the raw audio data using a feature extractor. Randomly mask the hidden representation to generate masking blocks corresponding to the masked parts of the hidden representation. The quantization module is used to generate corresponding discrete pseudo-labels for the masked parts in the hidden representation and these labels are used as the training targets. Input all unmasked hidden representations and the masking blocks corresponding to the masked hidden representations into the Transformer module; The Transformer module uses contextual information to predict the hidden representation content corresponding to the masked block, and trains the entire model by predicting discrete pseudo-labels, and finally outputs the context-enhanced representation corresponding to the hidden representation. When adding an efficient parameter fine-tuning module to the Transformer module of the sleep sound base model, the following steps are included: An adapter module is embedded before and after the feedforward neural network of each Transformer layer in the Transformer module. The adapter module consists of a lower projection layer, a non-linear activation function, and an upper projection layer, and has skip connections to retain key information, as shown in the following expression: in, Represents the adapter function. Indicates input data, Indicates the downward projection. Represents a non-linear activation function. Indicates upward projection; When pre-training the sleep sound base model using sleep sound data samples, the following steps are included: Freeze the feature extractor of the sleep sound base model, while keeping the quantization module and parameter fine-tuning module of the sleep sound base model trainable; After randomly masking a portion of the features of the sleep sound data samples, they are input into the sleep sound base model for pre-training. During the pre-training process, the loss function between the predicted value of the masked part output by the sleep sound base model and the original value is calculated. The parameters of the trainable part of the sleep sound base model are adjusted according to the loss function, and the parameters of all normalized layers of the sleep sound base model are updated until the loss function meets the requirements. When freezing the pre-trained portion of the evaluation model and training the evaluation model for downstream tasks using sleep sound data samples, the following steps are included: The sleep sound base model portion of the evaluation model is frozen, and the sleep sound data samples are sliced, labeled, and then input into the evaluation model for downstream task training. During the downstream task training, the sleep sound base model portion of the evaluation model extracts contextual features from the input audio and inputs the extracted features into the neural network module for classification or regression. The loss function between the classification or regression prediction value output by the neural network module and the original label value is calculated, and the parameters of the neural network module are adjusted according to the loss function until the loss function meets the requirements.

2. The sleep disorder assessment method based on a basic audio model according to claim 1, characterized in that, When adding an efficient parameter fine-tuning module to the Transformer module of the sleep sound base model, the following steps are included: In the multi-head self-attention layer of the Transformer module, a prefix tuning module is introduced into both the key and value tensors. This prefix tuning module generates a learnable prefix encoding matrix for the corresponding key or value through a linear layer, and then concatenates the learnable prefix encoding matrix with the corresponding key or value through a prefix multilayer perceptron layer to calculate the concatenated key or value matrix, as shown in the following expression: in, This represents the concatenated key matrix. This represents the concatenated value matrix. Represents the key tensor of a multi-head self-attention layer. The value tensor representing the multi-head self-attention layer. and Here are the learnable prefix codes for the keys and values, respectively, where... It is the prefix length. It is a hidden dimension. This indicates a connection along the length dimension of the sequence.

3. The sleep disorder assessment method based on a basic audio model according to claim 1, characterized in that, When adding an efficient parameter fine-tuning module to the Transformer module of the sleep sound base model, the following steps are included: In the multi-head self-attention layer of the Transformer module, corresponding low-rank adaptation modules are added to the query weight matrix and the key weight matrix, respectively. The low-rank adaptation module updates the corresponding query weight matrix or key weight matrix by parameterizing it into the product of two low-rank matrices using two trainable matrices, resulting in the updated query weight matrix or key weight matrix, as shown in the following expression: in, This represents the weight matrix of the updated query or the weight matrix of the keys. W This represents the weight matrix of the query or the weight matrix of the keys. A and B Let represent low-rank matrices, and α be the scaling factor.

4. The sleep disorder assessment method based on a basic audio model according to claim 1, characterized in that, The expression for the loss function is as follows: in, It is the set of masked portions in the sleep sound data samples, x' i It is the input for the occluded part. This represents the predicted output of the sleep sound baseline model. It is the original value x of the occluded part. i The corresponding positive vectorization latent representation, It is the set of elements Q, which represents the quantized representations of the original values ​​of all occluded parts, where sim(·, ·) denotes the similarity function. This represents hyperparameters.

5. The sleep disorder assessment method based on a basic audio model according to claim 1, characterized in that, When adding a corresponding neural network module to the pre-trained sleep sound baseline model as the head of a downstream task based on the type of sleep disorder assessment task, this includes: If the sleep disorder assessment task is a sleep apnea detection task, a neural network module with higher structural complexity should be selected as the head of the downstream task. If the sleep disorder assessment task is snoring detection, sleep position recognition, or sleep stage segmentation, a neural network module with lower structural complexity should be selected as the head of the downstream task.

6. The sleep disorder assessment method based on a basic audio model according to claim 1, characterized in that, Before acquiring real-time sleep sound data and inputting it into the trained evaluation model, the following steps are also included: Model quantization technology is used to convert the weights and activations in the evaluation model from floating-point format to low-precision format, and the trained evaluation model is converted into a format supported by mobile devices, and adaptation tests are performed. Integrate the inference framework into mobile device applications and load the converted model files; When acquiring real-time sleep sound data and inputting it into a trained evaluation model, the process includes: using the audio acquisition module of a mobile device to acquire the sleeper's real-time audio signal, inputting the real-time audio signal into the evaluation model, the sleep sound basic model part of the evaluation model extracting features from the real-time audio signal and inputting the extracted features into the neural network module for classification or regression, and using the classification or regression result as the corresponding sleep disorder evaluation result.

Images