A non-contact sleep thermal comfort identification method based on multi-source data fusion

Abstract

The application discloses a non-contact sleep thermal comfort recognition method based on multi-source data fusion, and belongs to the technical field of sleep monitoring and thermal comfort evaluation, and comprises the following steps: acquiring a sleep non-contact image sequence; processing the sleep non-contact image sequence by using a sleep thermal comfort recognition model; extracting multi-channel and multi-scale depth image features from the sleep non-contact image sequence by using an image coding module to obtain multi-modal image features; aligning and fusing the multi-modal image features and environmental modal features by using a modal fusion and time sequence modeling module, modeling time sequence features based on a bidirectional long short-term memory network enhanced by attention, and obtaining fused dynamic features; predicting sleep stage labels, sleep event labels and thermal comfort feeling labels by using a multi-task output branch module, and obtaining sleep thermal comfort recognition results. The application can deeply fuse multi-scale features, adaptively capture key time sequence frames, and realize non-contact sleep thermal comfort recognition with multi-task joint precise prediction.

Description

Technical Field

[0001] This invention relates to a non-contact sleep thermal comfort identification method based on multi-source data fusion, belonging to the field of sleep monitoring and thermal comfort assessment technology. Background Technology

[0002] Sleep is a vital physiological process for maintaining metabolic balance and cognitive recovery. Accurately identifying individual sleep states and analyzing changes in thermal comfort is of significant value for promoting healthy sleep, early warning of potential diseases, and optimizing indoor environmental control. Currently, sleep stage identification and monitoring of sleep events such as breathing and body movement primarily rely on polysomnography (PSG) systems. These systems monitor sleep data using contact devices such as electroencephalograms (EEG), electrooculograms (EOG), electrocardiograms (ECG), respiratory airflow, blood oxygen saturation, and body movement sensors. While these systems can achieve high-precision sleep monitoring, they have significant limitations in practical applications, such as: causing user discomfort and sleep disturbances; complex deployment and operation, making them difficult to promote in homes or everyday environments; and high equipment costs, hindering large-scale individualized monitoring and comfort adjustment.

[0003] In recent years, some studies have attempted to introduce non-contact methods such as infrared thermal imaging and RGB video monitoring. By extracting information such as thermal distribution maps, body movement amplitude, or sleep posture contours, and combining this with deep learning models to analyze sleep states and body surface heat dissipation characteristics, these methods aim to reduce interference with users. However, existing technologies mostly focus on offline analysis in laboratory environments. When applied to smart home or bedroom temperature control systems that require real-time feedback, these methods are largely limited to shallow image recognition or local thermal distribution estimation, and still face the following significant bottlenecks in practical engineering applications:

[0004] 1. Insufficient cross-modal feature mining: Sleep scenarios involve multi-scale behaviors such as large-scale turning over and subtle eye movements. Existing models, when fusing low-texture infrared thermal images and high-texture RGB images, often use shallow stitching or general convolutional structures, lacking targeted cross-scale feature interaction mechanisms. This makes it difficult to capture subtle eye movements or local thermal spot changes that are crucial for REM phase determination, resulting in limited recognition accuracy.

[0005] 2. Temporal modeling lacks keyframe adaptation capability: The sleep process has the characteristics of "long periods of stillness and short periods of sudden changes". Conventional temporal networks (such as standard LSTM) often apply equal effort to all time frames, which can easily be overwhelmed by a large number of still frames. They cannot adaptively focus on key "behavioral frames" that determine thermal comfort changes, such as apnea and the onset of body movement, resulting in a sluggish response to sudden events.

[0006] 3. Lack of joint constraints for multi-task collaboration: Existing solutions often treat sleep stage identification, abnormal event detection, and thermal comfort prediction as independent tasks, ignoring the strong physiological coupling between the three (e.g., body movement is often accompanied by changes in thermal sensation). This task fragmentation prevents the model from using the inherent correlation to correct the prediction results, causing the final output environmental regulation instructions to often lag behind the actual physiological needs of the human body.

[0007] Therefore, there is an urgent need for a non-contact sleep thermal comfort recognition method that can deeply integrate multi-scale features, adaptively capture key temporal frames, and achieve multi-task joint accurate prediction. Summary of the Invention

[0008] This invention aims to address the problems in existing technologies, such as reliance on contact-based devices for sleep state recognition, limited dimensions for thermal comfort assessment, and insufficient accuracy of non-contact monitoring. It proposes a non-contact sleep thermal comfort recognition method based on multi-source data fusion. This invention utilizes human body thermal images, posture images, and eye-tracking information acquired collaboratively by a non-contact infrared camera and an RGB camera. Furthermore, it employs a novel sleep thermal comfort recognition model to achieve sleep staging, key sleep event detection, and thermal comfort prediction.

[0009] To achieve the above objectives / to solve the above technical problems, the present invention is implemented using the following technical solution:

[0010] This invention provides a non-contact sleep thermal comfort recognition method based on multi-source data fusion, comprising:

[0011] Acquire non-contact sleep image sequences;

[0012] The trained sleep thermal comfort recognition model is used to process non-contact sleep image sequences to obtain sleep thermal comfort recognition results;

[0013] The sleep thermal comfort recognition model includes an image encoder module, a modality fusion and temporal modeling module, and a multi-task output branch module. The image encoder module extracts multi-channel, multi-scale features from non-contact sleep image sequences to identify an individual's body surface heat distribution, sleep posture changes, and eye movement patterns, thus obtaining image features. The modality fusion and temporal modeling module aligns and fuses image features with environmental features, and further models temporal features using an attention-enhanced bidirectional long short-term memory network to obtain fused dynamic features. The multi-task output branch module predicts sleep stage labels, sleep event labels, and thermal comfort labels based on the fused dynamic features, thus obtaining the sleep thermal comfort recognition result.

[0014] Furthermore, the sleep non-contact image sequence includes infrared thermal images acquired by an infrared camera, sleeping posture images acquired by an RGB camera, and eye movement images.

[0015] Furthermore, the image encoder module includes four feature extraction stages, with the input of each feature extraction stage being the output of the previous feature extraction stage; before the first feature extraction stage, there are sequentially connected input layers, downsampling convolutional layers, and layer normalization modules; after each feature extraction stage, there is a cross-stage guided enhancement module, and before the second to fourth feature extraction stages, there is an attention module.

[0016] Furthermore, the feature extraction process in the feature extraction stage includes:

[0017] Three parallel depthwise separable convolutional kernels are used to perform channel-level fusion on the image from the input feature extraction stage. The fused feature representation is as follows:

[0018] ;

[0019] in, This represents the output features after fusion. This indicates a splicing operation along the channel dimension. This indicates that the kernel size for the convolution operation on the input feature x is... Depth-separable convolution operations, k=3,5,7;

[0020] The output features are integrated and fused using a 1×1 multi-scale depthwise separable convolution, and then normalized using group normalization to obtain the group-normalized features, as shown below:

[0021] ;

[0022] in, This represents the normalized features within the g-th group. This represents the feature data within the g-th group. , Let be the mean and standard deviation of the feature group g, respectively. For numerically stable terms, , These represent the learnable scaling parameter and offset parameter of the g-th group, respectively;

[0023] Two 1×1 convolutional layers are used to increase and decrease the dimensionality of the group-normalized features, respectively, and the stage features are output through residual connections, as shown below:

[0024] ;

[0025] in, The output features are those from the feature extraction stage, and GELU is the activation function. This represents a 1×1 convolution operation.

[0026] Furthermore, the cross-stage guidance enhancement module includes a depthwise separable convolution module, a 1×1 convolution module, a layer normalization module, an attention CA module, and a residual guidance connection module;

[0027] The output features from the feature extraction stage are input into the cross-stage guided enhancement module, where they are fused using a depthwise separable convolution module, a 1×1 convolution module, and a layer normalization module to obtain the backbone fused features. :

[0028] ;

[0029] in, This represents the output features of the feature extraction stage. Presentation layer normalization operation, This represents a 1×1 convolution operation. This represents a depthwise separable convolution operation;

[0030] Main trunk fusion features Input attention CA module, for Average pooling along the spatial dimension yields channel description vectors in the height and width directions, respectively. and The expression is as follows:

[0031] ;

[0032] in, This represents the backbone fusion feature value at height h. H represents the backbone fusion feature value when the width is w, H is the height of the image in the sleep non-contact image sequence, and W is the width of the image in the sleep non-contact image sequence.

[0033] Will and After concatenation, spatial guided features are learned through a shared multilayer perceptron, and then attention weights are generated by separation, resulting in:

[0034] ;

[0035] in, and These represent the attention weight vectors in the height and width directions, respectively. This represents the Sigmoid activation function. , These represent convolution along the height and width directions, respectively.

[0036] according to and We perform a weighted fusion of the main path and the residual path to obtain:

[0037] ;

[0038] in, To enhance the output characteristics of the cross-stage guidance module, This represents the learnable weighted coefficient of the main path of the cross-stage guidance enhancement module. This indicates element-wise multiplication. This indicates the downsampling operation of the residual path of the cross-stage guidance enhancement module.

[0039] Furthermore, the attention module includes channel attention and spatial attention, which incorporates the output features of the cross-stage guidance enhancement module. Input the attention module and obtain its output features:

[0040] ;

[0041] in, For the output features of the attention module, and These represent the channel attention map and the spatial attention map, respectively.

[0042] Furthermore, the attention-enhanced bidirectional long short-term memory network includes a bidirectional long short-term memory network, frame-adaptive semantic attention, and a fully connected layer;

[0043] The temporal context of the input features is extracted using a bidirectional long short-term memory network to obtain the concatenated bidirectional temporal features;

[0044] Frame-adaptive semantic attention is used to calculate the frame importance score for each frame in the bidirectional temporal features;

[0045] In the fully connected layer, frame importance scores are used to weight the multi-task attention to obtain fused dynamic features;

[0046] The formula for calculating the frame importance score is as follows:

[0047] ;

[0048] in, This represents the importance score of frame t. and These are the learnable weights and biases for frame-adaptive semantic attention, respectively. This represents the bidirectional temporal characteristics after splicing. This represents the Sigmoid activation function.

[0049] Furthermore, the multi-task output branch module includes a sleep stage branch, a sleep event recognition branch, and a thermal comfort prediction branch;

[0050] The sleep staging branch uses a fully connected network to classify and predict the sleep state by fusing dynamic features in each frame, thus obtaining sleep stage labels. The expression is as follows:

[0051] ;

[0052] in, For normalized exponential functions, Indicates the fusion of dynamic features, and These represent the weights and biases of the sleep stage branch, respectively.

[0053] The sleep event branch uses a multi-label binary classification method to identify sleep events and obtain sleep event labels. The expression is as follows:

[0054] ;

[0055] in, This represents the Sigmoid activation function. and Weights and biases for the sleep event identification branch;

[0056] The thermal comfort prediction branch outputs thermal comfort labels based on preset parameters. The expression is as follows:

[0057] ;

[0058] in, and These represent the weights and biases of the thermal comfort prediction branch, respectively.

[0059] Furthermore, the training process of the sleep thermal comfort recognition model is as follows:

[0060] Acquire non-contact sleep image sequences, polysomnography data, and environmental data including temperature and humidity information, and synchronize all data in time;

[0061] The sleep non-contact image sequence and environmental data are fused together, real labels are generated based on polysomnography data, and the sleep non-contact image sequence is labeled based on the real labels to obtain a sleep thermal comfort dataset.

[0062] The sleep thermal comfort dataset is input into the sleep thermal comfort recognition model. Based on the sleep thermal comfort recognition results and real labels output by the sleep thermal comfort recognition model, the model is optimized using a multi-task joint loss function to obtain a trained sleep thermal comfort recognition model.

[0063] Furthermore, using the cross-entropy sub-loss function The expression for calculating the sleep stage branch loss is as follows:

[0064] ;

[0065] Where T is the sequence length. Labels for sleep stages, This represents the true label of the c-th sleep stage in frame t. This represents the probability that the sleep thermal comfort recognition model predicts the label of the c-th sleep stage for the t-th frame;

[0066] Using the BCE sub-loss function The loss value of the sleep event recognition branch is calculated using the following expression:

[0067] ;

[0068] in, Tags for sleep events, This represents the true label of the j-th type of sleep event in frame t. To predict the probability of a type j sleep event occurring in frame t for a sleep thermal comfort recognition model;

[0069] Using smooth L1 sub-loss function The loss value of the thermal comfort prediction branch is calculated using the following expression:

[0070] ;

[0071] in, For smoothing L1 functions, This indicates that the sleep thermal comfort recognition model predicts the thermal comfort label for frame t. This represents the actual thermal comfort label in frame t.

[0072] Multi-task joint loss function of sleep thermal comfort recognition model for:

[0073] ;

[0074] in, , and These are the balance weight coefficients for the cross-entropy sub-loss function, the BCE sub-loss function, and the smoothing L1 sub-loss function, respectively.

[0075] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:

[0076] This invention proposes a non-contact sleep thermal comfort recognition method based on multi-source data fusion. It improves the model structure and algorithm of existing deep learning models to obtain a novel sleep thermal comfort recognition model. The image encoding module enhances the deep perception capability of multimodal image features, capturing more stable thermal region distributions, posture changes, and eye dynamic features, providing a richer semantic foundation for subsequent tasks. The modality fusion and temporal modeling module aligns and fuses multimodal image features with environmental modal features, and further models temporal features through an attention-enhanced bidirectional long short-term memory network. This effectively improves the model's ability to recognize sudden behaviors (such as turning over and REM eye movements) while strengthening its modeling ability for stable phases, enabling the model to learn more targeted temporal representations. The multi-task output branch module uses multiple branches to predict different tasks, achieving non-contact, high-precision sleep stage recognition and joint prediction of sleep events and thermal comfort. The model can capture long-term dependence and short-term behavioral changes in sleep states, possessing higher reliability in physiological state recognition and identifying abnormal sleep behaviors at high temporal resolution. This is beneficial for assisting intelligent bedroom and other indoor environmental control systems in making precise temperature control responses.

[0077] This invention employs non-contact sleep data combined with model algorithms for sleep thermal comfort identification, improving user experience, reducing identification costs, and easily integrating into existing furniture environments. By improving the model, this invention addresses the shortcomings of existing technologies, such as insufficient feature mining, weak keyframe adaptation capabilities, and lack of multi-task collaboration. It effectively enhances the accuracy and timeliness of sleep thermal comfort identification, providing a reliable foundation for its application in areas such as healthy sleep, potential disease early warning, and indoor environment optimization and control. Attached Figure Description

[0078] Figure 1 The diagram shows the steps of the non-contact sleep thermal comfort recognition method provided in an embodiment of the present invention.

[0079] Figure 2 The diagram shown is a structural schematic of the sleep thermal comfort perception model in an embodiment of the present invention.

[0080] Figure 3 The diagram shown is a structural schematic of the image encoder module in an embodiment of the present invention;

[0081] Figure 4 The diagram shown is a structural schematic of the New ConvNeXt Block module in each stage of the image encoder module in an embodiment of the present invention.

[0082] Figure 5 The diagram shown is a schematic representation of the bidirectional long short-term memory network based on attention enhancement in an embodiment of the present invention.

[0083] Figure 6 The diagram shown is a complete flowchart of the sleep thermal comfort perception model during the offline training phase in an embodiment of the present invention. Detailed Implementation

[0084] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations thereof. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0085] This embodiment introduces a non-contact sleep thermal comfort recognition method based on multi-source data fusion, such as... Figure 1 As shown, the specific steps include the following:

[0086] Step A: Obtain a non-contact sleep image sequence.

[0087] This invention captures user sleep videos without contact using a camera, and then generates a non-contact sleep image sequence by extracting video frames. To ensure the integrity of the non-contact sleep data acquisition, this invention uses at least one infrared camera and two RGB cameras. The infrared camera captures infrared thermal images and visible light images, while the two RGB cameras are fixed at the front and side of the user's sleep area (e.g., at a 45-degree angle downwards from the head of the bed and on the left side of the bed), respectively, to capture the subject's posture changes, limb movements, and facial eye movements during sleep. The sleep posture image and eye movement image are then obtained by extracting the video frames. In practical use, existing home security cameras can be used, meaning this method can be directly integrated into the existing home environment, reducing costs to some extent.

[0088] In addition to image data, this invention can also measure environmental data in real time using devices such as temperature and humidity sensors. In subsequent sleep thermal comfort recognition, environmental data can be fused with image data to improve recognition accuracy.

[0089] To achieve synchronous processing of multimodal data throughout the sleep process, this invention proposes a structured data organization and alignment method based on a fixed time window: Frames are extracted from continuous video streams captured by infrared and RGB cameras at set time intervals to generate time-series images, constructing an image dataset for subsequent visual feature extraction. Timestamps are recorded synchronously during frame extraction to ensure accurate alignment of the image sequence with other modal data in the time dimension. Next, deep learning and computer vision algorithms are used to process the extracted image sequence: skin surface thermal distribution information (thermograph) is extracted from infrared thermal images, and individual sleep postures, body turning movements, and eye movement features are identified from RGB images. A complete visual behavioral feature sequence is constructed through methods such as thermal region segmentation and local action recognition.

[0090] Step B: Use the trained sleep thermal comfort recognition model to process the sleep non-contact image sequence to obtain sleep thermal comfort recognition results including key sleep events such as sleep state, body movement, and apnea, as well as thermal comfort scores.

[0091] To address the problems in existing technologies, such as reliance on contact-based devices for sleep state recognition, limited dimensions for thermal comfort assessment, and insufficient accuracy of non-contact monitoring, this invention proposes a novel sleep thermal comfort perception model (Sleep Transformer for Heat and Sleep, or SleepFormer-HS for short), as follows: Figure 2 As shown, it mainly includes an image encoder module, a modal fusion and temporal modeling module, and a multi-task output branch module.

[0092] Step B1: Input the preprocessed sleep non-contact image sequence into the image encoder module. The image encoding module extracts multi-channel and multi-scale depth image features from the infrared thermogram, sleeping posture image and eye movement image respectively, and identifies the individual's body surface heat distribution, sleeping posture changes and eye movement patterns to obtain image features.

[0093] Step B2: Use the modal fusion and temporal modeling module to align and fuse image features with environmental features including environmental parameters such as temperature and humidity, and use an attention-enhanced bidirectional long short-term memory network to model the sleep state transitions, periodic rhythms and key sleep event occurrence patterns in the fused time series to obtain fused dynamic features.

[0094] Step B3: Based on the fused dynamic features, output the sleep thermal comfort recognition results using the multi-task output branch module. The multi-task output branch module further includes a sleep stage branch, a sleep event recognition branch, and a thermal comfort prediction branch. The sleep stage branch is used to predict and output sleep stage labels based on the fused dynamic features, which is equivalent to the output of a traditional multi-channel system. The sleep event recognition branch is used to predict key sleep events and output sleep event labels, such as apnea, leg movement, and eye movement. The thermal comfort prediction branch is used to regress or classify and predict the subjective thermal comfort level corresponding to the time period, and to model it by combining heat map distribution, body movement frequency, and temperature and humidity information.

[0095] In this embodiment of the invention, to achieve efficient extraction of temporal information and spatial semantics from infrared thermal images, sleeping posture images, and eye-tracking images, the image encoder module adopts an improved image coding structure, namely, a frame adaptive feature extraction network (AF-ConvNeXt) constructed based on multi-channel independent sub-networks, such as... Figure 3 As shown, AF-ConvNeXt mainly includes four feature extraction stages (referred to as stages 1 to 4). To enhance the model's ability to express complex sleep features in non-contact images such as multi-scale sleeping postures, thermal zone changes, and eye movement regions, this invention introduces a structurally improved NewConvNeXt Block module in each stage. The structure of the NewConvNeXt Block module is as follows: Figure 4 As shown, this invention introduces a Channel-Aware Fusion Bridge (CA-Bridge) and an Attention Module (CBAM) between every two stages, which can improve the responsiveness to cross-layer semantics and key sleep event regions while preserving local details.

[0096] The processing procedure of the image encoder module in step B1 is as follows:

[0097] Step B101: The preprocessed sleep non-contact image sequence Input image encoder module, wherein, Indicates an infrared thermal image. Images representing sleeping positions Let T represent the time series length, H represent the height of the input image, and W represent the width of the input image. In this embodiment of the invention, after normalization and size adjustment, the infrared thermal image, the sleeping posture image, and the eye movement image are uniformly standardized to 256×256 pixels, so H=W=256.

[0098] First, a downsampling convolutional layer with a stride of 4 is used to extract primary spatial features, resulting in:

[0099] ;

[0100] in, This is the output feature map of the downsampled convolution; This represents the input sequence of non-contact sleep images; it represents a convolution operation with a kernel size of 4 and a stride of 4. This refers to the bias term of the downsampling convolutional layer; The symbol represents the Sigmoid activation function; * represents the convolution operation.

[0101] Step B102: The output feature map of the downsampling convolution is fed into the Layer Normalization module to standardize the channel dimension of each spatial location, thereby improving the plasticity and stability of the features in subsequent stages.

[0102] For each downsampled feature map t, at spatial location (h, w), the channel dimension vector is defined as... And perform the following standardization process:

[0103] Mean calculation: ;

[0104] Standard deviation calculation: ;

[0105] Normalization and affine transformation: ;

[0106] in, Represents the feature map of frame t; represents the value of the i-th channel in feature map x; C represents the total number of channels; and These represent the mean and standard deviation of all channel values ​​at spatial locations (h, w), respectively. To prevent numerical stability constants with a denominator of zero, it is usually set to: ; and These represent the learnable channel orientation scaling parameter and offset parameter in the layer normalization module, respectively. This represents the output features after normalization.

[0107] Step B103: Each stage includes a multi-layered improved New ConvNeXt Block module. The four stages of the New ConvNeXt Block module extract features from shallow to deep. To strengthen the connection between the four stages, this invention fuses the shallow features extracted in the previous stage into the deep features of the next stage, enhancing multi-scale feature extraction capabilities and optimizing the model's global context modeling capabilities. A cross-stage guided enhancement module (CA-Bridge) is introduced after each stage to replace the original single downsampling. To improve the perception of key regions (such as eye movements, pose joints, and hotspots), an attention module (CBAM) is inserted before stages 2 to 4. Feature extraction is performed through the four stages and the cross-stage guided enhancement module and attention module, ultimately outputting image features.

[0108] Combined with appendix Figure 4 The feature extraction process at each stage is as follows:

[0109] (1) In order to enhance the model's ability to model sleeping postures and heat distribution at different spatial scales, this invention introduces three parallel depthwise separable convolutional kernels at each stage, with kernel sizes of 3×3, 5×5, and 7×7, respectively, and performs channel-level fusion. The fused feature representation is as follows:

[0110] ;

[0111] in, This indicates the characteristics of the input New ConvNeXt Block module; This indicates that the kernel size for the convolution of the input feature x is... For depthwise separable convolution operations, in this embodiment of the invention, k=3,5,7; This indicates a splicing operation along the channel dimension; This represents the output feature after multi-scale feature fusion.

[0112] (2) A 1×1 multi-scale depthwise separable convolution is used to integrate the channels, and group normalization (GN) is introduced to normalize the channels. The size of each group normalization is set to G (e.g., 16 or 32). The feature representation after group normalization is as follows:

[0113] ;

[0114] in, This represents the normalized features within the g-th group; This represents the feature data within the g-th group; and Let be the mean and standard deviation of the g-th feature group; It is a numerically stable term, usually... ; and represents the learnable scaling parameter and offset parameter of the g-th group, respectively.

[0115] (3) Two 1×1 convolutional layers are used for dimensionality increase and dimensionality reduction respectively to introduce non-linear expressive power. Finally, the stage features are output through residual connections, as shown in the following expression:

[0116] ;

[0117] in, The input features for the stage are denoted as GELU; GELU is the activation function. This represents a 1×1 convolution operation; This is the final output feature of the stage.

[0118] In this embodiment of the invention, the structure of the cross-stage guidance enhancement module is as follows: Figure 3 As shown, it mainly includes a depthwise separable convolution module, a 1×1 convolution module, a layer normalization module, an attention CA module, and residual guided connections. The depthwise separable convolution module achieves spatial size compression, and the 1×1 convolution module achieves channel alignment. The cross-stage guided enhancement module can structurally enhance the cross-level modeling capability for behaviors such as rolling over, eye movement, and posture switching.

[0119] For the nth stage (n=2, 3, 4), its input depends on the output features of the previous stage. ,Will The input cross-stage guided enhancement module, through a feature fusion path consisting of a depthwise separable convolutional module, a 1×1 convolutional module, and a layer normalization module, yields the backbone fused features. The expression is as follows:

[0120] ;

[0121] in, This represents the output characteristics of the previous stage; This indicates a depthwise separable convolution operation with a stride of 2, used to upsample the input feature map to half its size in the spatial dimension, thereby matching the spatial resolution of the next stage; This represents a 1×1 convolution operation, used for adjusting the channel dimension. Presentation layer normalization operation.

[0122] Main trunk fusion features Input attention CA module, for Average pooling along the spatial dimension yields channel description vectors in the height (h) and width (w) directions, respectively. and The expression is as follows:

[0123] ;

[0124] in, This represents the backbone fusion feature value at height h. This represents the backbone fusion feature value when the width is w.

[0125] After concatenating the two, the learned spatial guided features are processed through a shared multilayer perceptron (MLP), and then attention weights are generated by separation, resulting in:

[0126] ;

[0127] in, and These represent the attention weight vectors in the height and width directions, respectively; This represents the Sigmoid activation function. , These represent convolution along the height and width directions, respectively.

[0128] In addition to the main path, the cross-stage bootstrapping enhancement module also contains residual paths. A weighted fusion of the main path and residual paths yields:

[0129] ;

[0130] in, This indicates element-wise multiplication; This indicates the downsampling operation of the residual path; Represents the learnable weighting coefficients of the main path; The output features of the cross-stage guided enhancement module are used as input to the next stage pre-convolutional block attention module (CBAM).

[0131] In this embodiment of the invention, the structure of the attention module CBAM is as follows: Figure 3 As shown, the attention module CBAM integrates channel attention and spatial attention. Channel attention can identify key modal features (such as eye movement signals); spatial attention can focus on local hotspot areas (such as high body temperature points and posture inflection points), thereby significantly improving the ability to recognize events such as REM, rolling over, and thermal changes.

[0132] Output characteristics of the cross-stage guidance enhancement module The input to the attention module is used to obtain its output features, which are then used as input for the next stage. The output of the attention module is calculated using the following formula:

[0133] ;

[0134] in, The output features of the attention module; and These represent the channel attention map and the spatial attention map, respectively.

[0135] The processing procedure of the modal fusion and temporal modeling module in step B2 is as follows:

[0136] Step B201: In order to effectively integrate multi-source non-contact sensing data, after the image modal features are encoded by AF-ConvNeXt, they are first aligned and fused with environmental modal features (such as room temperature, humidity, etc.) at the feature level.

[0137] Let the image features output by the image encoder module be... Environmental characteristics (aligned sampling rate) are: To achieve modal dimension consistency and scale uniformity, the two features are first projected onto the same dimensional space. The length of the aligned consistent frame sequence is T, and the dimension is... , means as follows:

[0138] ;

[0139] ;

[0140] in, and These represent the aligned image features and environmental features, respectively; GeLU is a non-linear activation function. and These are the fully connected layer parameters for image features and environmental features, respectively; and These are the bias terms corresponding to image features and environmental features, respectively.

[0141] Next, the two features will be concatenated by channel:

[0142] ;

[0143] in, The spliced ​​multimodal features; This is a channel-based splicing operation.

[0144] Will Inputting the data into a multilayer perceptron (MLP) yields aligned and fused multimodal features:

[0145] ;

[0146] in, and These are the weight matrix and bias term of the fully connected network in the fusion layer, respectively; This is for aligning and fusing the multimodal feature sequences.

[0147] To further capture the temporal dynamic evolution characteristics during sleep, this invention proposes a structure based on a Frame-Adaptive Semantic Attention enhanced BiLSTM (FASA-BiSleepNet) network to model state transitions, cyclical rhythms, and key event patterns in time series data. FASA-BiSleepNet comprises a BiLSTM network, a Frame-Adaptive Semantic Attention (FASA) network, and fully connected layers (FC), as shown in the diagram. Figure 5 As shown, BiLSTM has frame-level bidirectional modeling capabilities, which more fully models the temporal dependencies of sleep states (such as REM cycles and N3 durations). FASA can integrate frame weight estimation, task-aware attention mechanisms, and semantic fusion gating structures to adapt to multi-task prediction needs. Figure 5 In the diagram, the letter h represents the hidden layer of the LSTM network, and Xt-1, Xt, and Xt+1 represent the outputs of different LSTM networks.

[0148] FASA-BiSleepNet first utilizes the bidirectional long short-term memory network BiLSTM to extract temporal context:

[0149] ;

[0150] in, and These represent the forward and backward LSTM hidden state vectors at time t, respectively; This represents the bidirectional temporal characteristics after splicing.

[0151] Based on the output features of the Bidirectional Long Short-Term Memory (BiLSTM) network, the frame importance score of each frame feature map is calculated using the Frame Adaptive Semantic Attention (FASA) mechanism. Then, this score is used to weight the multi-task attention in the fully connected (FC) layer to obtain the final frame-level semantic representation. Frame importance scoring can assign higher weight to key frames, enhancing the ability to perceive key frames (such as breathing pauses and movement initiation frames).

[0152] The formula for calculating frame importance score is as follows:

[0153] ;

[0154] in, This represents the importance score of the t-th frame. and These are the learnable weights and biases of the frame-adaptive semantic attention network, respectively. This represents the Sigmoid activation function.

[0155] This invention will The input temporal modeling network FASA-BiSleepNet is used to jointly model the temporal evolution of images using the improved FASA-BiSleepNet structure, extract the dynamic feature representation of sleep state under different time windows, and finally obtain the fused dynamic features.

[0156] In step B3, prediction is performed using the three multi-task branches of the multi-task output branch module, including:

[0157] (1) Sleep stage prediction (multi-class classification): For each frame of fused dynamic feature map output by the modality fusion and temporal modeling module, a fully connected network is used in the sleep stage branch to perform multi-class classification prediction of sleep state to obtain sleep stage labels, as shown in the following expression:

[0158] ;

[0159] in, This represents the probability distribution of sleep stages, i.e., sleep stage labels; Indicates the fusion of dynamic features; and These represent the weights and biases of the classification layer, respectively. It is a normalized exponential function.

[0160] In this embodiment of the invention, sleep stage labels are divided into: wakefulness (W), light sleep (N1, N2), deep sleep (N3) and rapid eye movement (REM).

[0161] (2) Sleep event prediction (multi-label classification): In the sleep event recognition branch, a multi-label binary classification method is used to identify sudden events (such as body movement, eye movement, and apnea) in each frame of fused dynamic feature map, and the sleep event labels are obtained as follows:

[0162] ;

[0163] in, This represents the probability of various sleep events (body movement, eye movement, etc.) occurring independently, i.e., sleep event labels; and The weights and biases for the event recognition layer.

[0164] (3) Thermal comfort prediction (regression): The thermal comfort prediction branch can output the subjective thermal comfort score (ASHRAE -3 to +3 interval) corresponding to the current frame, as shown in the following expression:

[0165] ;

[0166] in, This represents the predicted thermal comfort score (a continuous variable), i.e., the thermal comfort label; and These represent the weights and biases of the regression layer, respectively.

[0167] In addition to the above description, the sleep thermal comfort perception model of the present invention also includes an individualized correction and subjective alignment module, which is used to integrate the user's resting physiological characteristics and past subjective rating records to dynamically adjust the thermal comfort prediction results output by the three task branches, improve the consistency between the model output and the individual's real feelings, and realize personalized thermal comfort perception.

[0168] In this embodiment of the invention, the sleep thermal comfort recognition model needs to be trained before application, such as... Figure 6 As shown, the training process is as follows:

[0169] (1) Obtain non-contact image sequences of the entire sleep process, polysomnography data (EEG, ECG, blood oxygen, nasal airflow, body movement, etc.) and environmental data such as temperature and humidity information.

[0170] To achieve non-contact intelligent identification and analysis of the human body's thermal comfort state during sleep, this embodiment constructs an experimental environment that integrates multimodal data acquisition and verification. It combines multiple data sources, such as infrared thermal imaging, visible light images, environmental perception, and physiological parameter acquisition from a polysomnography system, and can acquire multidimensional information about the subject's sleep process under interference-free conditions for subsequent thermal comfort modeling and identification tasks.

[0171] In addition to infrared and RGB cameras, this embodiment of the invention also records the temperature and humidity changes of the indoor environment in real time using a temperature and humidity recorder as an external thermal environment influencing factor, and records sleep monitoring data (physiological parameters such as EEG, ECG, blood oxygen, nasal airflow, and body movement) using a PSG polysomnography device as an auxiliary indicator and reference standard for thermal comfort. Furthermore, the subjects are given a morning or midday thermal comfort questionnaire score survey.

[0172] This experiment collected complete nighttime and midday sleep data from 24 subjects. The nighttime testing period was approximately 6–7 hours, and the midday testing period was approximately 1 hour. Data for each subject was stored separately according to the testing time period (nighttime or midday), creating a structured directory. Each subject's data contained multiple subdirectories, each containing the following data: video frames (thermal and visible light images) captured by an infrared camera; video frames (eye-tracking images and sleeping posture images) captured by two RGB cameras; temperature and humidity values ​​collected every 5 minutes by a temperature and humidity recorder; multimodal physiological signals output by the polysomnography system (ECG, EEG, EEG, blood oxygenation, nasal airflow, leg movements, etc.); gold standard sleep stage labels and sleep event labels (apnea, REM sleep, body movement) analyzed by the professional polysomnography system data processing software SleepFairy-AS; and the subject's subjective thermal comfort rating questionnaire (TSV or a custom rating scale).

[0173] All sensing devices undergo time calibration through a unified data synchronization control module to ensure that multi-source data are aligned in the time dimension, improving the accuracy of subsequent data fusion processing. The collected data is stored and preliminarily processed in real time, providing reliable data support for subsequent thermal comfort recognition using deep learning models. The non-contact multi-source experimental platform built in this embodiment achieves efficient and accurate collection of physiological and environmental information closely related to thermal comfort during sleep, overcoming the interference of traditional contact measurement methods on the sleep state of subjects, and laying a solid foundation for the training and validation of thermal comfort recognition algorithms.

[0174] (2) The collected data are preprocessed and a structured sleep thermal comfort dataset is obtained through data fusion and data annotation. The preprocessing includes time synchronization, image cropping, and image enhancement.

[0175] 201) Time window synchronization of the acquired data: Frame extraction is performed on the acquired continuous video stream to generate a complete image sequence. Simultaneously with image preprocessing, physiological data corresponding to the time period is collected using a polysomnography system, including electrocardiogram (ECG), electroencephalogram (EEG), electrooculogram (EOG), blood oxygen saturation, heart rate, nasal airflow, and chest and abdominal movements. All physiological parameters are normalized before fusion to improve model training stability. Furthermore, environmental parameters (such as indoor temperature and humidity) and body surface temperature are simultaneously introduced to record the thermal load within the time window. Image features, physiological parameters, and environmental data are fused and aligned along the time dimension.

[0176] 202) Image data preprocessing and enhancement are performed through operations such as image cropping and normalization.

[0177] Infrared thermal images captured by infrared cameras primarily reflect the distribution of thermal radiation in subjects, serving as crucial evidence for analyzing changes in body surface temperature and thermal comfort. Since raw infrared images often suffer from low contrast and unclear details, contrast enhancement and pseudo-color processing are applied.

[0178] Contrast Enhancement: Histogram Equalization is used to improve image contrast. This method adjusts the image's grayscale distribution to create a stronger contrast between hot areas (high-radiation areas) and the background, thereby highlighting the body's thermal features. After enhancement, image details are more apparent, and the thermal distribution is clearer, which is beneficial for the model to recognize thermal patterns.

[0179] Pseudo-color processing: The grayscale infrared image is mapped to a pseudo-color space, and the Hot color mapping scheme is used to map different grayscale values ​​to red-yellow-blue color gradients, making the thermal distribution more intuitive to visualize. This process enhances the model's ability to perceive temperature gradients and improves the efficiency of feature representation.

[0180] After contrast enhancement and pseudo-color processing, the infrared thermal image can be represented as a three-channel pseudo-color image. To facilitate subsequent model training, this invention uniformly crops all infrared thermal images to the bed area and adjusts them to a uniform size (256×256 pixels), while also performing normalization processing.

[0181] Images of sleeping postures captured by RGB cameras reflect the body's pose information during sleep and are crucial for predicting sleep posture categories and judging turning-over behavior. Due to the diversity of postures and directional changes during actual sleep, data augmentation strategies are needed to improve the model's generalization ability. These strategies include: Horizontal Flip: simulating left and right turning-over behavior to allow the model to learn symmetrical posture features and avoid bias towards specific directions; Image Rotation: randomly rotating the image within a small angle range of [-15°, +15°] to enhance the model's robustness to slight directional changes and prevent overfitting to posture samples with fixed angles; Brightness / Contrast Jitter: appropriately adjusting brightness and contrast to simulate image changes under lighting disturbances, making the model more adaptable.

[0182] All enhanced sleeping posture images were cropped and scaled to a uniform size (256×256 pixels) and then normalized.

[0183] Eye-tracking images captured by RGB cameras are primarily used to detect the Rapid Eye Movement (REM) phase and awake states, requiring clear facial features. Due to significant interference outside the face during capture, eye-tracking image processing focuses on Region of Interest (ROI) extraction and blur enhancement. Specifically, this includes: ROI Cropping: Using the Haar face detection algorithm to extract facial regions from the image and cropping them to a fixed size to remove non-facial background interference and increase the proportion of the facial region in the image; and Gaussian Blur: Applying a slight Gaussian blur to the cropped facial image to enhance the model's robustness to edges and contours and reduce the interference of capture noise on model training.

[0184] All enhanced eye-tracking images were cropped and scaled to a uniform size (256×256 pixels) and then normalized.

[0185] 203) The sleep thermal comfort dataset was created through data fusion, data annotation, and dataset partitioning.

[0186] Based on preprocessed multi-channel image frames (including infrared thermograms, sleep posture images, and eye-tracking images), and combined with the reference standards output by a polysomnography (PSG) system, the images are precisely labeled using the Labelme annotation tool. Specifically, this includes:

[0187] By analyzing data such as EEG, heart rate, blood oxygenation, and EEG collected by a polysomnography (PSG) device using specialized software, sleep stage labels and sleep event labels are obtained as ground truth labels. For each image frame at any given time, the ground truth label at that time point is used as a supervisory signal. Using a sliding time window as the unit, image sequences of fixed time lengths are used as basic sample units, and their corresponding sleep stages (W / N1 / N2 / N3 / REM) are labeled. For observable events such as body movements and eye movements in the images, the start and end times of the corresponding sleep events are labeled in the image sequence through video review or automatic assisted recognition.

[0188] In addition, the present invention also assigns thermal comfort labels (such as "cool", "comfortable", "warm" or -3 to +3 points) to video frames in the corresponding time period based on the thermal comfort subjective rating questionnaire (TSV thermal comfort rating) filled out by the subjects before going to sleep and after waking up.

[0189] The annotation process strictly adhered to sleep medicine and thermal comfort evaluation standards to ensure that each data label was highly consistent with the actual physiological state. For sleep stage annotation, all labels were derived from the SleepFairy-AS output and used as the "gold standard" for monitoring. For sleep event annotation (such as apnea or body movement), the SleepFairy-AS output combined with image representation was manually reviewed and confirmed. For thermal comfort labels, a standardized subjective thermal comfort rating questionnaire was used to ensure a one-to-one correspondence between subjective perception and objective data.

[0190] Through the above annotation process, a multi-label synchronized structured sleep thermal comfort dataset was finally constructed, providing a high-quality and highly reliable training foundation for supervised learning of the model.

[0191] Training set partitioning strategy: To avoid leakage of individual information, the dataset is partitioned according to the subjects: training set: 18 people, validation set: 3 people, test set: 3 people. Each subset covers data from midday and nighttime to ensure the model's generalization ability to different sleep stages and environmental heat loads.

[0192] (3) Use the sleep thermal comfort dataset to train the sleep thermal comfort recognition model. During the training process, use the multi-task joint loss function to optimize the model based on the sleep thermal comfort recognition results and real labels output by the sleep thermal comfort recognition model, and obtain the trained sleep thermal comfort recognition model.

[0193] In this embodiment of the invention, cross-entropy loss is used as the sub-loss function of the sleep staging branch, label-by-label BCE loss is used as the sub-loss function of the sleep event recognition branch, and smooth L1 (Huber) loss is used as the sub-loss function of the thermal comfort prediction branch. The sub-loss functions corresponding to the three tasks need to be jointly optimized to construct a multi-task joint loss function, improve the performance of each sub-task, and achieve inter-task synergistic enhancement.

[0194] Cross-entropy sub-loss function The expression is:

[0195] ;

[0196] BCE sub-loss function The expression is:

[0197] ;

[0198] Smoothing L1 sub-loss function The expression is:

[0199] ;

[0200] Multi-task joint loss function The expression is:

[0201] ;

[0202] Where T is the sequence length (number of frames). Labeling for sleep stages; The true label (one-hot encoded) represents the c-th sleep stage in frame t. This represents the probability that the model predicts the t-th frame to be a sleep stage label of type c. Label sleep events; This represents the true label of the j-th type of sleep event in frame t; To predict the probability of the j-th type of sleep event occurring in the t-th frame for the model; For smoothing L1 functions; This represents the thermal comfort label predicted by the model for frame t. This represents the actual thermal comfort label in frame t. , and These are the balancing weights for the cross-entropy sub-loss function, the BCE sub-loss function, and the smoothed L1 sub-loss function, respectively, used to adjust the importance of different tasks during training.

[0203] This invention employs three sub-loss functions to optimize three types of tasks respectively, and achieves multi-task collaborative learning through a weighted strategy. This enhances the model's generalization ability to different sleep state features, improves the robustness of thermal perception prediction and event detection, and adapts to high-dimensional temporal feature representation after multimodal fusion. It has good engineering feasibility and clinical application potential.

[0204] (4) Combine the subjects’ morning or midday thermal comfort questionnaire scores with the analysis of polysomnography data by professional analysis software, calibrate the model output, and achieve modeling of consistency between subjective and objective perception.

[0205] Compared with the prior art, the method of the present invention has the following advantages:

[0206] 1. Deep Perception Capability of Multimodal Image Features: The image encoder module AF-ConvNeXt designed in this invention introduces an improved CA-Bridge connection structure and a convolutional attention mechanism CBAM, achieving cross-scale information guidance and spatial saliency enhancement across four stages. This improves the model's ability to represent non-contact images such as infrared thermal images, pose maps, and eye-tracking maps. Compared to the traditional next-generation convolutional network ConvNeXt model, AF-ConvNeXt can extract more stable thermal region distributions, pose changes, and eye dynamic features, providing a richer semantic foundation for subsequent tasks.

[0207] 2. Task-Aware Temporal Modeling Capability: In FASA-BiSleepNet, the frame-adaptive semantic attention FASA proposed in this invention integrates frame importance estimation, task-aware attention, and a multi-branch fusion gating mechanism, effectively improving the model's ability to recognize sudden behaviors (such as turning over and REM eye movements) while strengthening its modeling ability for stable phases. Unlike the traditional unified self-attention structure, FASA can be adapted to three types of tasks: sleep staging, event recognition, and hot rating prediction, enabling the model to learn more targeted temporal representations.

[0208] 3. Non-contact, high-precision sleep stage identification: This invention utilizes visible light and infrared images to accurately predict sleep stages (such as N1, N2, N3, REM, etc.) consistent with professional polysomnography (PSG) systems without physical contact, effectively replacing some functions of traditional polysomnography devices. Through joint modeling using a bidirectional long short-term memory (BiLSTM) network and a frame-level attention mechanism, the model can capture long-term dependencies and short-term behavioral changes in sleep states, achieving higher reliability in physiological state identification.

[0209] 4. Joint Prediction Capability of Sleep Events and Thermal Comfort: This invention is the first to achieve a unified framework modeling for sleep event (turning over, apnea, etc.) recognition and subjective thermal score prediction, proposing a multi-task head network structure that integrates image features, posture changes, and indoor environmental data. Compared to traditional systems that can only perform single-stage or temperature control, this invention can identify abnormal sleep behaviors at high temporal resolution and assist intelligent bedroom systems in making precise temperature control responses.

[0210] 5. End-to-End Deployability and Lightweight Structure: The sleep thermal comfort recognition model of this invention is designed as an end-to-end spatiotemporal multimodal perception framework, supporting input formats such as infrared video, RGB pose, and eye-tracking video. Multi-scale spatial features are extracted through the front-end encoder AF-ConvNeXt, and multi-task time-series modeling and accurate prediction are performed by the back-end FASA-BiSleepNet, forming a deployable, scalable deep learning model structure that adapts to the needs of non-contact sleep perception. AF-ConvNeXt adopts a residual hierarchical structure, combined with attention modules and lightweight gating structures, controlling the number of parameters while ensuring model accuracy, giving the entire model excellent edge deployment capabilities.

[0211] The method of this invention can be deployed on edge computing terminals or high-performance computing platforms, and enables one-click data input, status display, and result output through a visual interface (such as an interactive operating system based on Qt). Users only need to input image sequences such as real-time infrared thermal images, RGB sleeping posture images, and RGB eye-tracking images into the system. After processing by the deep learning model proposed in this invention, sleep thermal comfort recognition can be achieved. The model can be deployed on industrial-grade servers, smart mattress chips, or mobile edge devices through a deep learning framework, and provides real-time video input, status monitoring, and visual output through a QT interface.

[0212] 6. Combining technological innovation with practical feasibility: This invention proposes network structures with clear innovations in four dimensions: non-contact monitoring, modal fusion, attention mechanism, and multi-task modeling (CA-Bridge cross-stage guidance enhancement module, AF-ConvNeXt image encoder module, FASA-BiSleepNet bidirectional long short-term memory network based on attention enhancement, and task head fusion structure). Each structure has a theoretical basis and experimental verification, and meets the feasibility requirements for engineering deployment.

[0213] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0214] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0215] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0216] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0217] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. A non-contact sleep thermal comfort recognition method based on multi-source data fusion, characterized in that, include: Acquire non-contact sleep image sequences; The trained sleep thermal comfort recognition model is used to process non-contact sleep image sequences to obtain sleep thermal comfort recognition results; The sleep thermal comfort recognition model includes an image encoder module, a modality fusion and temporal modeling module, and a multi-task output branch module; The image encoding module extracts multi-channel, multi-scale features from non-contact sleep image sequences to identify an individual's surface heat distribution, sleep posture changes, and eye movement patterns, thus obtaining image features. The modality fusion and temporal modeling module aligns and fuses image features with environmental features, and further models temporal features through an attention-enhanced bidirectional long short-term memory network to obtain fused dynamic features. The multi-task output branch module predicts sleep stage labels, sleep event labels, and thermal comfort labels based on the fused dynamic features, thus obtaining sleep thermal comfort recognition results. The image encoder module includes four feature extraction stages, with the input of each feature extraction stage being the output of the previous feature extraction stage; before the first feature extraction stage, there are sequentially connected input layers, downsampling convolutional layers, and layer normalization modules; after each feature extraction stage, there is a cross-stage guided enhancement module; and before the second to fourth feature extraction stages, there are attention modules. The feature extraction process in the feature extraction stage includes: Three parallel depthwise separable convolutional kernels are used to perform channel-level fusion on the image from the input feature extraction stage. The fused feature representation is as follows: ； in, This represents the output features after fusion. This indicates a splicing operation along the channel dimension. This indicates that the kernel size for the convolution operation on the input feature x is... Depth-separable convolution operations, k=3,5,7; The output features are integrated and fused using a 1×1 multi-scale depthwise separable convolution, and then normalized using group normalization to obtain the group-normalized features, as shown below: ； in, This represents the normalized features within the g-th group. This represents the feature data within the g-th group. , Let be the mean and standard deviation of the feature group g, respectively. For numerically stable terms, , These represent the learnable scaling parameter and offset parameter of the g-th group, respectively; Two 1×1 convolutional layers are used to increase and decrease the dimensionality of the group-normalized features, respectively, and the stage features are output through residual connections, as shown below: ； in, The output features are those from the feature extraction stage, and GELU is the activation function. This represents a 1×1 convolution operation.

2. The non-contact sleep thermal comfort recognition method according to claim 1, characterized in that, The non-contact sleep image sequence includes infrared thermal images acquired by an infrared camera, sleeping posture images acquired by an RGB camera, and eye movement images.

3. The non-contact sleep thermal comfort recognition method according to claim 1, characterized in that, The cross-stage guidance enhancement module includes a depthwise separable convolution module, a 1×1 convolution module, a layer normalization module, an attention CA module, and a residual guidance connection module; The output features from the feature extraction stage are input into the cross-stage guided enhancement module, where they are fused using a depthwise separable convolution module, a 1×1 convolution module, and a layer normalization module to obtain the backbone fused features. : ； in, This represents the output features of the feature extraction stage. Presentation layer normalization operation, This represents a 1×1 convolution operation. This represents a depthwise separable convolution operation; Main trunk fusion features Input attention CA module, for Average pooling along the spatial dimension yields channel description vectors in the height and width directions, respectively. and The expression is as follows: ； in, This represents the backbone fusion feature value at height h. H represents the backbone fusion feature value when the width is w, H is the height of the image in the sleep non-contact image sequence, and W is the width of the image in the sleep non-contact image sequence. Will and After concatenation, spatial guided features are learned through a shared multilayer perceptron, and then attention weights are generated by separation, resulting in: ； in, and These represent the attention weight vectors in the height and width directions, respectively. This represents the Sigmoid activation function. , These represent convolution along the height and width directions, respectively. according to and We perform a weighted fusion of the main path and the residual path to obtain: ； in, To enhance the output characteristics of the cross-stage guidance module, This represents the learnable weighted coefficient of the main path of the cross-stage guidance enhancement module. This indicates element-wise multiplication. This indicates the downsampling operation of the residual path of the cross-stage guidance enhancement module.

4. The non-contact sleep thermal comfort recognition method according to claim 1, characterized in that, The attention module includes channel attention and spatial attention, and incorporates the output features of the cross-stage guidance enhancement module. Input the attention module and obtain its output features: ； in, For the output features of the attention module, and These represent the channel attention map and the spatial attention map, respectively.

5. The non-contact sleep thermal comfort recognition method according to claim 1, characterized in that, The attention-enhanced bidirectional long short-term memory network includes a bidirectional long short-term memory network, frame-adaptive semantic attention, and a fully connected layer; The temporal context of the input features is extracted using a bidirectional long short-term memory network to obtain the concatenated bidirectional temporal features; Frame-adaptive semantic attention is used to calculate the frame importance score for each frame in the bidirectional temporal features; In the fully connected layer, frame importance scores are used to weight the multi-task attention to obtain fused dynamic features; The formula for calculating the frame importance score is as follows: ； in, This represents the importance score of frame t. This represents the bidirectional temporal characteristics after splicing. and These are the learnable weights and biases for frame-adaptive semantic attention, respectively. This represents the Sigmoid activation function.

6. The non-contact sleep thermal comfort recognition method according to claim 1, characterized in that, The multi-task output branch module includes a sleep stage branch, a sleep event recognition branch, and a thermal comfort prediction branch. The sleep staging branch uses a fully connected network to classify and predict the sleep state by fusing dynamic features in each frame, thus obtaining sleep stage labels. The expression is as follows: ； in, For normalized exponential functions, Indicates the fusion of dynamic features, and These represent the weights and biases of the sleep stage branch, respectively. The sleep event branch uses a multi-label binary classification method to identify sleep events and obtain sleep event labels. The expression is as follows: ； in, This represents the Sigmoid activation function. and Weights and biases for the sleep event identification branch; The thermal comfort prediction branch outputs thermal comfort labels based on preset parameters. The expression is as follows: ； in, and These represent the weights and biases of the thermal comfort prediction branch, respectively.

7. The non-contact sleep thermal comfort recognition method according to claim 1, characterized in that, The training process of the sleep thermal comfort recognition model is as follows: Acquire non-contact sleep image sequences, polysomnography data, and environmental data including temperature and humidity information, and synchronize all data in time; The sleep non-contact image sequence and environmental data are fused together, real labels are generated based on polysomnography data, and the sleep non-contact image sequence is labeled based on the real labels to obtain a sleep thermal comfort dataset. The sleep thermal comfort dataset is input into the sleep thermal comfort recognition model. Based on the sleep thermal comfort recognition results and real labels output by the sleep thermal comfort recognition model, the model is optimized using a multi-task joint loss function to obtain a trained sleep thermal comfort recognition model.

8. The non-contact sleep thermal comfort recognition method according to claim 6, characterized in that, Using the cross-entropy sub-loss function The expression for calculating the sleep stage branch loss is as follows: ； Where T is the sequence length. Labels for sleep stages, This represents the true label of the c-th sleep stage in frame t. This represents the probability that the sleep thermal comfort recognition model predicts the label of the c-th sleep stage for the t-th frame; Using the BCE sub-loss function The loss value of the sleep event recognition branch is calculated using the following expression: ； in, Tags for sleep events, This represents the true label of the j-th type of sleep event in frame t. To predict the probability of a type j sleep event occurring in frame t for a sleep thermal comfort recognition model; Using the smoothed L1 sub-loss function The loss value of the thermal comfort prediction branch is calculated using the following expression: ； in, For smoothing L1 functions, This indicates that the sleep thermal comfort recognition model predicts the thermal comfort label for frame t. This represents the actual thermal comfort label in frame t. Multi-task joint loss function of sleep thermal comfort recognition model for: ； in, , and These are the balance weight coefficients for the cross-entropy sub-loss function, the BCE sub-loss function, and the smoothing L1 sub-loss function, respectively.

Images