A sleep monitoring method and system based on contactless video data sequences

Abstract

This invention discloses a sleep monitoring method and system based on contactless video data sequences. The method includes: S1, acquiring video data streams of human sleep through a camera system; obtaining facial and chest / abdomen video data using a YOLOv11 network; S2, using an RPPG signal extraction model to extract RPPG signals from the facial video data; S3, using a respiratory signal extraction model to extract chest / abdomen micro-motion change features from the chest / abdomen video data using optical flow, and performing noise filtering to obtain chest / abdomen motion signals as respiratory signals; S4, using a blood oxygen saturation extraction model to detect and output blood oxygen saturation signals using RPPG signals; S5, using a multi-physiological index fusion and recognition model to perform multi-modal fusion analysis according to time slice T1 to obtain long-term sleep stage segmentation results. This invention uses contactless video data sequences to extract and identify physiological index signals and obtain long-term sleep stage segmentation results, achieving high-precision sleep monitoring and sleep stage identification.

Description

Technical Field

[0001] This invention relates to the field of sleep monitoring and analysis, and in particular to a sleep monitoring method and system based on non-contact video data sequences. Background Technology

[0002] In the field of sleep science, polysomnography (PSG) has long held a central position in sleep staging assessment as the traditional gold standard. This technology monitors sleep by collecting various physiological parameters: recording bioelectrical signals such as brain electrical activity, eye movement potentials, electromuscular signals, and cardiac activity; and simultaneously acquiring physiological indicators such as breathing patterns, chest and abdominal movements, blood oxygen saturation, and snoring characteristics. However, PSG monitoring requires strict adherence to electrode placement guidelines. The use of these direct-contact sensors often causes significant discomfort to the subject, affecting their normal sleep state and ultimately leading to data that deviates from their true sleep state. Furthermore, in the data analysis phase, physicians must manually interpret PSG results according to the standardized interpretation manual developed by the American Academy of Sleep Medicine. Individual differences in interpretation among physicians of the same set of monitoring data inevitably introduce assessment bias due to human subjectivity. Summary of the Invention

[0003] The purpose of this invention is to provide a sleep monitoring method and system based on contactless video data sequences. It utilizes the YOLOv11 network to capture facial and / or chest / abdomen images from video data streams and obtains facial and chest / abdomen video data based on human anatomy. An RPPG signal extraction model, based on inter-frame difference processing and temporal physiological signal feature extraction, obtains remote photoplethysmography (RPPG) signals and separates blood oxygen saturation signals. A respiratory signal extraction model extracts chest and abdominal signals, weighted and fused to obtain chest and abdominal motion signals as respiratory signals. A multi-physiological index fusion and recognition model uses RPPG signals, respiratory signals, and blood oxygen saturation signals for multimodal fusion analysis to obtain long-term sleep stage segmentation results, providing technical support for human sleep quality detection and analysis.

[0004] The objective of this invention is achieved through the following technical solution:

[0005] A sleep monitoring method based on contactless video data sequences, the method comprising:

[0006] S1. Collect video data streams of human sleep through a camera system; use the YOLOv11 network to detect, segment, and track the face, chest, and abdomen of the video data stream, and obtain facial video data and chest and abdomen video data.

[0007] S2. Construct an RPPG signal extraction model. The RPPG signal extraction model extracts RPPG signals from facial video data.

[0008] S3. Construct a respiratory signal extraction model. The respiratory signal extraction model uses optical flow to extract the micro-motion change features of the chest and abdomen in the chest and abdomen video data and simultaneously uses frequency domain adaptive notch filtering to filter out noise. It uses sparse representation time domain filtering to perform sparse representation and remove non-respiratory signals to obtain the chest and abdomen motion signal as the respiratory signal.

[0009] S4. Construct a blood oxygen saturation extraction model. The blood oxygen saturation extraction model uses RPPG signals to extract blood oxygen signal features and detects and outputs blood oxygen saturation signals.

[0010] S5. Construct a multi-physiological index fusion identification model. The multi-physiological index fusion identification model includes four sleep stages: wakefulness (W), rapid eye movement (REM) sleep, light sleep (N1 / N2), and deep sleep (N3). The multi-physiological index fusion identification model uses RPPG signals, respiratory signals, and blood oxygen saturation signals to perform multimodal fusion analysis according to time slice T1 to obtain long-term sleep stage segmentation results.

[0011] To better implement this invention, the YOLOv11 network is trained using a face dataset. The YOLOv11 network includes a backbone network, a feature enhancement network, and a detection head. The backbone network includes a C3k2 module and is used to maintain the receptive field and extract basic features. The feature enhancement network includes an SPPF-C2PSA joint module and is used to acquire global information and perform feature enhancement using a channel-space dual attention mechanism. The detection head is constructed based on a decoupled dual-branch structure for classification and regression. The detection head uses an IoU-aware mechanism to predict the center point offset and width and height, and incorporates a dynamic label allocation strategy and dynamically adjusts the positive and negative sample thresholds through the IoU-aware mechanism. The detection head of the YOLOv11 network outputs the detection and segmentation results of the face and chest / abdomen and tracks them.

[0012] Preferably, the camera system includes at least one infrared camera, and the camera area of ​​the camera system covers the facial triangle region and the chest and abdominal movement region of the subject. If the camera system includes one infrared camera, the YOLOv11 network identifies and segments the video data stream of the infrared camera into facial video data and chest and abdominal video data. If the camera system includes multiple infrared cameras, the multiple infrared cameras are fused and combined in time sequence to form a video data stream covering the entire data stream, and the YOLOv11 network identifies and segments the video data stream covering the entire data stream into facial video data and chest and abdominal video data.

[0013] Preferably, the RPPG signal extraction model processing method includes:

[0014] S21. A difference layer and a batch normalization layer are used to process the facial video data for noise, including illumination noise and motion noise. The difference layer calculates the inter-frame difference signal of two consecutive video frames and determines the difference frame. The batch normalization layer normalizes the difference frame to the same scale and processes the noise.

[0015] S22. The self-attention mechanism transfer network first undergoes normalization processing by a custom normalization module and then several two-dimensional convolutional layers to gradually extract temporal physiological signal features from low to high levels. At the same time, the attention mechanism is used to enhance the weights of important temporal physiological signal features, and finally outputs the RPPG signal.

[0016] Preferably, the respiratory signal extraction model extracts respiratory signals using a method comprising:

[0017] S31. Utilize optical flow to extract micro-motion variation features of the chest and abdomen from chest and abdominal video data. These micro-motion variation features include chest signals. and abdominal signals ;

[0018] S32, Signal to the chest Frequency domain adaptive notch filter Suppressing frequency domain noise ; Abdominal signals Using sparse decomposition to remove motion artifacts, we obtain :

[0019] S33. The processed chest signal and abdominal signal are weighted and fused to generate a respiratory signal.

[0020] Preferably, the method for extracting the micro-movement changes in the chest and abdomen is as follows:

[0021] S311. Using polynomial estimation, estimate the neighborhood information of each pixel in the image, and convert the pixel grayscale value into coordinates, thus obtaining the neighborhood information of the pixel. The expression is as follows:

[0022] ,in This represents the coordinates corresponding to the transformed pixel grayscale value. This represents the approximation of the second derivative of image grayscale. This represents the approximation of the first derivative of the image grayscale. This represents the gray level of the corresponding neighborhood center, and T represents the transpose.

[0023] S312. Obtain the preliminary inter-frame displacement vector of pixels. The expression is as follows:

[0024] ,in To integrate the weight function of the pixel neighborhood using the weighted least squares method, It is the average of the first derivative differences of gray levels between adjacent frames;

[0025] S313. Construct an 8-parameter parameterized displacement model to describe the displacement motion, with the following expression:

[0026]

[0027]

[0028] Where d is the pixel displacement vector, P is the motion parameter vector, and S is the design matrix, the final pixel displacement is obtained by weighted least squares solution. The expression is as follows:

[0029] .

[0030] Preferably, the method for obtaining the blood oxygen saturation signal is as follows: a blood oxygen saturation extraction model extracts the temporal signal features of the RPPG signal, and the RPPG signal is dimensionality-reduced and mapped using a convolutional layer to obtain the features. ,feature After sequential processing using Gaussian error linear unit activation functions, batch normalization, and feature concatenation to enhance the feature propagation mechanism, features are then compressed using global pooling. Blood oxygen saturation signal predicted by mapping through a fully connected layer.

[0031] Preferably, the time slice T1 is 30 seconds. The multi-physiological index fusion recognition model includes a signal decoder, a period mixer, and a sequence mixer. The signal decoder downsamples the signal to half its original length and uses tensor reshaping operations and fully connected layers of time distribution to finally generate a feature vector sequence. The period mixer fuses the feature vectors of multiple modalities into a unified representation for each 30-second time segment. Features of different modalities are fused through a linear transformation and activation function. The sequence mixer... Sleep stage segmentation is performed using 30-second time slices to identify and classify sleep stages, and sleep stage segmentation labels are output to obtain long-time series sleep stage segmentation results.

[0032] A sleep monitoring system for implementing sleep monitoring methods includes a camera system, a YOLOv11 network, an RPPG signal extraction model, a respiratory signal extraction model, a blood oxygen saturation extraction model, and a multi-physiological index fusion and recognition model. The camera system is used to collect video data streams of human sleep and input them into the YOLOv11 network. The YOLOv11 network is used to detect, segment, and track the face and chest / abdomen areas of the video data stream, obtaining facial video data and chest / abdomen video data. The RPPG signal extraction model extracts RPPG signals from the facial video data. The respiratory signal extraction model uses optical flow to extract the micro-motion change features of the chest and abdomen from the chest / abdomen video data. Simultaneously, a frequency-domain adaptive notch filtering method is used for noise removal, and a sparse representation time-domain filtering method is used for sparse representation and removal of non-respiratory signals to obtain chest and abdominal motion signals as respiratory signals. The blood oxygen saturation extraction model uses RPPG signals to extract blood oxygen signal features and detects and outputs blood oxygen saturation signals. The multi-physiological index fusion recognition model includes four sleep stages: wakefulness (W), rapid eye movement (REM) sleep, light sleep (N1 / N2), and deep sleep (N3). The multi-physiological index fusion recognition model uses RPPG signals, respiratory signals, and blood oxygen saturation signals divided into time slices (T1) for multimodal fusion analysis to obtain long-term sleep stage segmentation results.

[0033] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0034] (1) This invention uses the YOLOv11 network to capture the face and / or chest and abdomen of the video data stream and obtains facial video data and chest and abdomen video data based on the human body structure relationship. The RPPG signal extraction model obtains the remote photoplethysmography (RPPG) signal and separates the blood oxygen saturation signal based on inter-frame difference processing and temporal physiological signal feature extraction. The respiratory signal extraction model extracts the chest signal and abdominal signal and performs weighted fusion processing to obtain the chest and abdominal motion signal as the respiratory signal. The multi-physiological index fusion recognition model uses the RPPG signal, respiratory signal and blood oxygen saturation signal to perform multimodal fusion analysis to obtain the long-term sleep stage segmentation results, realizing high-precision sleep monitoring and sleep stage recognition, and providing technical support for human sleep quality detection and analysis.

[0035] (2) This invention uses contactless video data sequences to extract RPPG signals, respiratory signals, and blood oxygen saturation signals respectively. The multi-physiological index fusion recognition model is used to process and obtain long-term sleep stage segmentation results. The sleep monitoring process reduces sleep interference, and the detection and evaluation are not affected by human subjective factors. The evaluation has low subjective bias and has broad clinical application prospects. It can be widely used in hospitals, elderly care institutions, and home sleep health monitoring. Attached Figure Description

[0036] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0037] The present invention will be further described in detail below with reference to embodiments:

[0038] Example

[0039] like Figure 1 As shown, a sleep monitoring method based on contactless video data sequences includes the following steps:

[0040] S1. Acquire video data streams of human sleep using a camera system. Utilize a YOLOv11 network to detect, segment, and track the face and chest / abdomen areas of the video data stream, obtaining facial and chest / abdomen video data. The camera system includes at least one infrared camera, with its camera area covering the subject's facial triangle and chest / abdomen movement area. If the camera system contains one infrared camera, the YOLOv11 network identifies and segments the infrared camera's video data stream into facial and chest / abdomen video data. If the camera system contains multiple infrared cameras, these cameras are sequentially fused and combined into a video data stream covering the entire subject. The YOLOv11 network then identifies and segments this comprehensive video data stream into facial and chest / abdomen video data. Several infrared cameras of the camera system are vertically fixed to the center of one side of the long axis of the bed via adjustable brackets. The installation should meet the following technical requirements: horizontally, the optical center of the infrared camera should be located 0.8–1.2m from the edge of the bed; vertically, the center point of the lens should be 1.0–1.3m above the mattress surface; and in terms of pitch, the lens optical axis should be at an angle of 10°–15° to the horizontal plane. The observed images should meet the following standards: firstly, ensure complete coverage of the subject's facial triangle area (the area connecting the center of the eyebrows to the tip of the nose); secondly, be able to simultaneously capture the chest and abdominal movement area from the sternal notch to the umbilicus; and thirdly, achieve a bed surface coverage rate of no less than 85% on a standard double bed (1.8 × 2.0 meters).

[0041] In some embodiments, the YOLOv11 network is trained using a face dataset. The YOLOv11 network includes a backbone network, a feature enhancement network, and a detection head. The backbone network contains a C3k2 module and is used to maintain the receptive field and extract basic features. The feature enhancement network includes an SPPF-C2PSA joint module (containing an SPPF layer and a C2PSA module) for acquiring global information (mainly handled by the SPPF layer, which acquires global information through multi-scale pooling) and employs a channel-space dual attention mechanism for feature enhancement (mainly by the C2PSA module). The detection head is constructed based on a decoupled dual-branch structure for classification and regression. The detection head uses an IoU-aware mechanism to predict the center point offset and width / height, while incorporating a dynamic label allocation strategy and dynamically adjusting the positive and negative sample thresholds through the IoU-aware mechanism. The YOLOv11 network's detection head outputs the detection and segmentation results for the face and chest / abdomen, and tracks them. This embodiment can use a transfer learning strategy to fine-tune the model, which can effectively improve the face detection performance. This embodiment can detect only the face and obtain the face detection box. Then, based on the proportional relationship between the face detection box and the human body structure, the chest and abdomen regions of interest can be quickly predicted and the chest and abdomen detection boxes can be obtained. Of course, the detection and segmentation of the face and chest and abdomen can also use the YOLOv11 network. Then, the proportional relationship of the human body structure is used for verification.

[0042] S2. Construct an RPPG signal extraction model. The RPPG signal extraction model extracts RPPG signals (also known as remote photoplethysmography signals) from facial video data. In some embodiments, the RPPG signal extraction model processing method includes:

[0043] S21. A difference layer and a batch normalization layer are used to process the facial video data for noise, including illumination noise and motion noise. The difference layer calculates the inter-frame difference signal between two consecutive video frames and determines the difference frames. The batch normalization layer normalizes the difference frames to the same scale and performs noise processing. In this embodiment, the RPPG signal extraction model can also use a time-domain shift module to enhance the sequence feature extraction capability. Preferably, the difference layer establishes an optical model to calculate the inter-frame difference signal between two consecutive video frames, as shown in the following expression:

[0044] ,in For difference signals, Light intensity, The intensity of specular reflection. Diffuse reflection intensity The sensor noise signal is modulated by specular reflection, diffuse reflection, and sensor noise to represent the light intensity. The batch normalization layer normalizes the difference frames to the same scale, significantly amplifying subtle changes in skin pixels, making it easier to capture minute changes in physiological signals more accurately.

[0045] S22. The self-attention transfer network first performs normalization processing through a custom normalization module, and then processes the data through several two-dimensional convolutional layers to progressively extract temporal physiological signal features from low to high levels. Simultaneously, an attention mechanism is used to enhance the weights of important temporal physiological signal features, ultimately outputting the RPPG signal. In this embodiment, the self-attention transfer network can also use average pooling layers to downsample the feature map, reducing dimensionality and computational cost, preventing overfitting, and improving the model's generalization ability.

[0046] S3. Construct a respiratory signal extraction model. This model uses optical flow to extract micro-motion variation features of the chest and abdomen from chest and abdominal video data, and simultaneously employs frequency-domain adaptive notch filtering for noise removal. In some embodiments, the method for extracting chest and abdominal micro-motion variation features is as follows:

[0047] S311. Using polynomial estimation, estimate the neighborhood information of each pixel in the image, and convert the pixel grayscale value into coordinates, thus obtaining the neighborhood information of the pixel. The expression is as follows:

[0048] ,in Represents the coordinates corresponding to the pixel grayscale value. , This represents the approximation of the second derivative of image grayscale. This represents the approximation of the first derivative of the image grayscale. This represents the gray level of the corresponding neighborhood center, and T represents the transpose.

[0049] S312. Obtain the preliminary inter-frame displacement vector of pixels. The expression is as follows:

[0050] ,in To integrate the weight function of the pixel neighborhood using the weighted least squares method, This represents the average of the first-order derivatives of the grayscale values ​​between adjacent frames. Within the neighborhood of each pixel, the image grayscale can be approximated by a local quadratic polynomial, and the fitting parameters between two consecutive frames satisfy the following relationship: Since the image contains noise and local variations, direct solution is unstable. Therefore, weighted least squares method is used to integrate neighborhood pixel information, and the following error term is constructed:

[0051] ,in Represents the initial inter-frame displacement vector of pixels. , This indicates that a weighted least squares method is used to integrate neighboring pixels. This indicates the change in pixel displacement between frames.

[0052] S313. Construct an 8-parameter parameterized displacement model to describe the displacement motion, with the following expression:

[0053]

[0054]

[0055] Where d is the pixel displacement vector, P is the motion parameter vector, and S is the design matrix. Neighboring pixels are integrated using the weighted least squares method. The error term expression for the weighted least squares method for pixel i is as follows: Then, weighted least squares are used to obtain the final pixel displacement. The expression is as follows:

[0056] .

[0057] A sparse representation time-domain filtering method is used to perform sparse representation and remove non-respiratory signals to obtain chest and abdominal motion signals as respiratory signals. In some embodiments, the respiratory signal extraction model extracts respiratory signals using a method that includes:

[0058] S31. Utilize optical flow to extract micro-motion variation features of the chest and abdomen from chest and abdominal video data. These micro-motion variation features include chest signals. and abdominal signals This optical flow signal set , .

[0059] S32, Signal to the chest Frequency domain adaptive notch filter Suppressing frequency domain noise Abdominal signals Using sparse decomposition to remove motion artifacts, we obtain The expression is as follows:

[0060] ,in For Fourier transform, This is the inverse Fourier transform. and These represent the dictionary and sparse coefficients in the sparse representation, respectively.

[0061] S33. The processed chest signal and abdominal signal are weighted and fused to generate a respiratory signal.

[0062] S4. Construct a blood oxygen saturation extraction model. This model utilizes the RPPG signal to extract blood oxygen signal features and detects and outputs the blood oxygen saturation signal. In some embodiments, the blood oxygen saturation signal is obtained as follows: The blood oxygen saturation extraction model extracts the temporal signal features of the RPPG signal. The model integrates a prediction network incorporating the design principles of DenseNet and ConvMixer, and adaptively reconstructs the one-dimensional temporal signal features. The network comprises three core processing stages: feature projection, dense convolution, and feature aggregation. The overall network is a regression function. , This indicates the predicted blood oxygen level. These are the network parameters. Features are obtained by dimensionality reduction mapping of the RPPG signal through convolutional layers. Preferably, the kernel size of the convolutional layer is Step size is The projection process uses a stride that is equal to the kernel size, i.e. Special configuration: , For the number of channels, .feature After sequential processing using the Gaussian error linear unit activation function, batch normalization, and feature concatenation to strengthen the feature propagation mechanism, the specific steps are as follows: After processing with the Gaussian Error Linear Unit (GELU) activation function, the following is obtained: : , Then perform batch normalization: By constructing low-dimensional representations, key fluctuation patterns in signals can be effectively captured.

[0063] Next, spatiotemporal features are compressed using global pooling operations. Specifically, spatiotemporal dimension features The method is as follows: A fully connected hierarchical structure is adopted, and the output of each convolutional layer is concatenated with the features of all subsequent layers to form an enhanced feature propagation mechanism. The output of each layer is as follows:

[0064] Each basic convolutional module contains a residual structure composed of depthwise separable convolutions, followed by activation layers and batch normalization layers. Internally, the input features and the batch-normalized output are concatenated channel by channel, as shown in the expression: Then, point-by-point convolution operations are performed to form a multi-level feature fusion path: This invention's design preserves the fine structure of local features while enhancing the interaction capabilities of cross-layer information. It compresses spatiotemporal features through global pooling operations. The expression is as follows:

[0065]

[0066] The blood oxygen saturation signal predicted by mapping through a fully connected layer is expressed as follows: The entire processing flow of the above method achieves a nonlinear mapping from the original waveform to physiological parameters through a step-by-step abstraction of signal representation, thus improving the robustness of feature representation while ensuring computational efficiency.

[0067] S5. Construct a multi-physiological index fusion recognition model. This model includes four sleep stages: wakefulness (W), rapid eye movement (REM) sleep, light sleep (N1 / N2), and deep sleep (N3). The model utilizes RPPG signals, respiratory signals, and blood oxygen saturation signals, divided into time slices T1 (30 seconds in this example), to perform multimodal fusion analysis and obtain long-term sleep stage segmentation results. In some embodiments, the multi-physiological index fusion recognition model includes a signal decoder, a period mixer, and a sequence mixer. The signal decoder downsamples the signal to half its original length. Each modality... The original signal is represented as: The number of modalities is M. For each input modality, an independent CNN encoder architecture is used, consisting of stacked residual layers (each residual layer contains three convolutional layers and a subsequent max-pooling layer, which can downsample the signal to half its original length, as shown in the following structure:

[0068] ,in, This module represents a non-linear activation function (such as ReLU or GELU) that can downsample a signal to half its original length.

[0069] By utilizing tensor reshaping operations and fully connected layers with temporal distributions, a feature vector sequence is ultimately generated. The expression for the feature vector sequence is: The period mixer fuses the feature vectors of multiple modalities into a unified representation for each 30-second time interval. The input expression for a given time window is: ,in This indicates a vector concatenation operation.

[0070] Features from different modalities are fused using a linear transformation and activation function: Sequence mixers for containing Sleep stage segmentation and classification are performed using 30-second time slices, and sleep stage segmentation labels are output to obtain long-time series sleep stage segmentation results. This embodiment uses fusion representation. Apply the Transformer or temporal convolution module for global modeling: ,in The system assigns the number of sleep stage categories (W, REM, N1 / N2, and N3) and outputs the predicted label for each 30-second segment of sleep:

[0071] ,in This indicates a sleep stage divided into four phases: wakefulness (W), rapid eye movement (REM) sleep, light sleep (N1 / N2), and deep sleep (N3). This indicates the total number of sleep stages. It realizes the conversion from multimodal long-time signals to classification results of each 30-second video segment, while ensuring the sufficiency of feature representation and improving cross-modal modeling and global reasoning capabilities.

[0072] A sleep monitoring system for implementing sleep monitoring methods includes a camera system, a YOLOv11 network, an RPPG signal extraction model, a respiratory signal extraction model, a blood oxygen saturation extraction model, and a multi-physiological index fusion and recognition model. The camera system collects video data streams of human sleep and inputs them into the YOLOv11 network. The YOLOv11 network is used to detect, segment, and track the face and chest / abdomen in the video data stream, obtaining facial video data and chest / abdomen video data. The RPPG signal extraction model extracts RPPG signals from the facial video data. The respiratory signal extraction model uses optical flow to extract micro-motion change features of the chest and abdomen from the chest / abdomen video data, and simultaneously uses frequency-domain adaptive notch filtering for noise filtering. A sparse representation temporal filtering method is used for sparse representation and removal of non-respiratory signals to obtain chest / abdomen motion signals as respiratory signals. The blood oxygen saturation extraction model uses the RPPG signal to extract blood oxygen signal features and detects and outputs a blood oxygen saturation signal. The multi-physiological index fusion identification model includes four sleep stages: wakefulness (W), rapid eye movement (REM) sleep, light sleep (N1 / N2), and deep sleep (N3). The multi-physiological index fusion identification model uses RPPG signals, respiratory signals, and blood oxygen saturation signals to perform multimodal fusion analysis according to time slice T1 to obtain long-term sleep stage segmentation results.

[0073] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A sleep monitoring method based on contactless video data sequences, characterized in that: The methods include: S1. Acquire video data streams of human sleep through a camera system; use a YOLOv11 network to detect, segment, and track faces and chest / abdomen in the video data stream, obtaining facial video data and chest / abdomen video data; the YOLOv11 network includes a backbone network, a feature enhancement network, and a detection head. The backbone network includes a C3k2 module to maintain the receptive field and extract basic features; the feature enhancement network includes an SPPF-C2PSA joint module to acquire global information and perform feature enhancement using a channel-space dual attention mechanism; the detection head is constructed based on a decoupled dual-branch structure of classification and regression. The detection head uses an IoU-aware mechanism to predict the center point offset and width / height, while incorporating a dynamic label allocation strategy and dynamically adjusting the positive and negative sample thresholds through the IoU-aware mechanism; the detection head of the YOLOv11 network outputs the detection and segmentation results of the face and chest / abdomen and tracks them; S2. Construct an RPPG signal extraction model, which extracts RPPG signals from facial video data; the RPPG signal extraction model processing method includes: S21. A difference layer and a batch normalization layer are used to process the facial video data for noise, including illumination noise and motion noise. The difference layer calculates the inter-frame difference signal of two consecutive video frames and determines the difference frame. The batch normalization layer normalizes the difference frame to the same scale and processes the noise. S22. The self-attention mechanism transfer network first undergoes normalization processing through a custom normalization module, and then several two-dimensional convolutional layers gradually extract temporal physiological signal features from low to high levels. At the same time, the attention mechanism is used to enhance the weights of important temporal physiological signal features, and finally outputs the RPPG signal. S3. Construct a respiratory signal extraction model. This model uses optical flow to extract micro-motion characteristics of the chest and abdomen from chest and abdominal video data, and simultaneously employs frequency-domain adaptive notch filtering for noise removal. A sparse representation time-domain filtering method is used to perform sparse representation and remove non-respiratory signals, obtaining the chest and abdominal motion signal as the respiratory signal. The respiratory signal extraction method of the model includes: S31. Utilize optical flow to extract micro-motion variation features of the chest and abdomen from chest and abdominal video data. These micro-motion variation features include chest signals. and abdominal signals The method for extracting the micro-movement features of the chest and abdomen is as follows: S311. Using polynomial estimation, estimate the neighborhood information of each pixel in the image, and convert the pixel grayscale value into coordinates, thus obtaining the neighborhood information of the pixel. The expression is as follows: ,in This represents the coordinates corresponding to the transformed pixel grayscale value. This represents the approximation of the second derivative of image grayscale. This represents the approximation of the first derivative of the image grayscale. This represents the gray level of the corresponding neighborhood center, and T represents the transpose. S312. Obtain the preliminary inter-frame displacement vector of pixels. The expression is as follows: ,in To integrate the weight function of the pixel neighborhood using the weighted least squares method, It is the average of the first derivative differences of gray levels between adjacent frames; S313. Construct an 8-parameter parameterized displacement model to describe the displacement motion, with the following expression: ； ； Where d is the pixel displacement vector, P is the motion parameter vector, and S is the design matrix, the final pixel displacement is obtained by weighted least squares solution. The expression is as follows: ； S32, Signal to the chest Frequency domain adaptive notch filter Suppressing frequency domain noise ; Abdominal signals Using sparse decomposition to remove motion artifacts, we obtain : S33. The processed chest signal and abdominal signal are weighted and fused to generate a respiratory signal; S4. Construct a blood oxygen saturation extraction model. The blood oxygen saturation extraction model uses RPPG signals to extract blood oxygen signal features and detects and outputs blood oxygen saturation signals. S5. Construct a multi-physiological index fusion identification model. The multi-physiological index fusion identification model includes four sleep stages: wakefulness (W), rapid eye movement (REM) sleep, light sleep (N1 / N2), and deep sleep (N3). The multi-physiological index fusion identification model uses RPPG signals, respiratory signals, and blood oxygen saturation signals to perform multimodal fusion analysis according to time slice T1 to obtain long-term sleep stage segmentation results.

2. The sleep monitoring method based on contactless video data sequence according to claim 1, characterized in that: The camera system includes at least one infrared camera. The camera area of ​​the camera system covers the facial triangle region and the chest and abdominal movement region of the subject. If the camera system includes one infrared camera, the YOLOv11 network identifies and segments the video data stream of the infrared camera into facial video data and chest and abdominal video data. If the camera system includes multiple infrared cameras, the multiple infrared cameras are fused and combined in time sequence to form a video data stream that covers the entire subject. The YOLOv11 network then identifies and segments the video data stream that covers the entire subject into facial video data and chest and abdominal video data.

3. The sleep monitoring method based on contactless video data sequence according to claim 1, characterized in that: The method for obtaining the blood oxygen saturation signal is as follows: the blood oxygen saturation extraction model extracts the temporal signal features of the RPPG signal, and the features are obtained by dimensionality reduction mapping of the RPPG signal through a convolutional layer. ,feature After sequential processing using Gaussian error linear unit activation functions, batch normalization, and feature concatenation to enhance the feature propagation mechanism, features are then compressed using global pooling. Blood oxygen saturation signal predicted by mapping through a fully connected layer.

4. The sleep monitoring method based on contactless video data sequence according to claim 1, characterized in that: The time slice T1 is 30 seconds. The multi-physiological index fusion recognition model includes a signal decoder, a period mixer, and a sequence mixer. The signal decoder downsamples the signal to half its original length and uses tensor reshaping operations and fully connected layers of time distribution to finally generate a feature vector sequence. The period mixer fuses the feature vectors of multiple modalities into a unified representation for each 30-second time segment. Features of different modalities are fused through a linear transformation and activation function. The sequence mixer... Sleep stage segmentation is performed using 30-second time slices to identify and classify sleep stages, and sleep stage segmentation labels are output to obtain long-time series sleep stage segmentation results.

5. A sleep monitoring system implementing the sleep monitoring method of claim 1, characterized in that: The system includes a camera system, a YOLOv11 network, an RPPG signal extraction model, a respiratory signal extraction model, a blood oxygen saturation extraction model, and a multi-physiological index fusion recognition model. The camera system collects video data streams of human sleep and inputs them into the YOLOv11 network. The YOLOv11 network detects, segments, and tracks the face and chest / abdomen areas of the video data stream, obtaining facial and chest / abdomen video data. The RPPG signal extraction model extracts RPPG signals from the facial video data. The respiratory signal extraction model uses optical flow to extract micro-motion characteristics of the chest and abdomen from the chest / abdomen video data and simultaneously employs frequency domain adaptive methods. Notch filtering is used for noise removal, and a sparse representation time-domain filtering method is used for sparse representation and removal of non-respiratory signals to obtain chest and abdominal motion signals as respiratory signals. The blood oxygen saturation extraction model uses RPPG signals to extract blood oxygen signal features and detects and outputs blood oxygen saturation signals. The multi-physiological index fusion recognition model includes four sleep stages: wakefulness (W), rapid eye movement (REM) sleep, light sleep (N1 / N2), and deep sleep (N3). The multi-physiological index fusion recognition model uses RPPG signals, respiratory signals, and blood oxygen saturation signals divided into time slices (T1) for multimodal fusion analysis to obtain long-term sleep stage segmentation results.

Images