Vibration photography sleep quality long-term tracking modeling system based on time sequence convolutional network

Abstract

The invention discloses a vibration photography sleep quality long-term tracking modeling system based on a time sequence convolutional network, and the system comprises a video collection module which is used for collecting a sleep video frame sequence of a subject; the vibration image construction module is used for generating a vibration image frame sequence; the statistical feature extraction module is used for performing frequency domain statistics and spatial-temporal feature extraction on the vibration image frame sequence; the sleep behavior modeling module is used for performing sleep behavior modeling and multi-task identification through the improved residual time sequence convolutional network; the cross-night drift correction module is used for carrying out distribution drift analysis on a vibration image frame sequence of continuous nights through Gaussian kernel density estimation and Jensen-Shannon divergence; the sleep index calculation module is used for calculating a sleep quality index set of the current night; and the trend modeling and visualization module is used for generating and visualizing a sleep quality trend curve. According to the method, the sleep behavior recognition precision and the continuity and stability of sleep quality trend modeling are improved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and physiological signal data analysis technology, and in particular to a long-term tracking and modeling system for sleep quality based on temporal convolutional networks using vibration photography. Background Technology

[0002] With the increasing demand for non-contact health monitoring and intelligent sleep management, the application of vibration photography technology based on visible light image sequences in remote sleep behavior assessment and quality tracking has attracted widespread attention. Existing sleep monitoring methods mainly rely on wearable heart rate belts, wristbands, or mattress-type pressure sensing systems to collect body movement and physiological signals, but these methods generally suffer from the following problems in practical applications:

[0003] The data collection process requires close contact with the human body or embedding into the bed structure, resulting in poor user compliance and susceptibility to changes in wearing posture or pressure contact points. This leads to distortion of body movement and respiratory waveform signals, affecting the recognition of micro-motion patterns and the modeling of respiratory rhythms. Traditional video analysis methods, which rely on background modeling, optical flow analysis, or human key point extraction, are not robust under low light, occlusion, or bedding conditions, making it difficult to extract subtle vibration information. Existing sleep quality assessments mostly rely on single-night static modeling or manual rating labels, lacking data structure design and modeling mechanisms for long-term tracking, making it difficult to depict the dynamic evolution trend of sleep quality over consecutive nights. For high-dimensional time-series data converted from vibration images, traditional recurrent neural network structures suffer from gradient degradation and historical memory decay, making it difficult to model long-term dependencies while maintaining computational efficiency, thus affecting the fitting accuracy and interpretability of sleep trends.

[0004] Therefore, how to provide a long-term tracking and modeling system for sleep quality based on temporal convolutional networks is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] One objective of this invention is to propose a long-term tracking and modeling system for sleep quality based on vibration photography using temporal convolutional networks. This invention fully utilizes non-contact vibration image modeling, inter-frame differential accumulation enhancement, and an improved residual temporal convolutional network. It describes in detail the process from sleep video frame acquisition, vibration image frame construction, frequency domain feature extraction to sleep quality trend analysis, and has the advantages of non-invasive monitoring, strong micro-motion recognition capability, and high accuracy in long-term tracking and modeling.

[0006] The long-term tracking and modeling system for sleep quality based on temporal convolutional networks according to an embodiment of the present invention includes the following steps: The video acquisition module is used to acquire a sequence of sleep video frames from the subject. The vibration image construction module is used to perform inter-frame pixel difference on the sleep video frame sequence and accumulate it frame by frame within a set window length to generate a vibration image frame sequence. The statistical feature extraction module is used to perform frequency domain statistics and spatiotemporal feature extraction on the vibration image frame sequence to generate a vibration image feature vector sequence. The sleep behavior modeling module is used to model sleep behavior and perform multi-task recognition by passing the vibration image feature vector sequence through an improved residual temporal convolutional network, and output a set of sleep behavior recognition sequences. The cross-night drift correction module is used to perform distribution drift analysis on a sequence of vibration images from consecutive nights using Gaussian kernel density estimation and Jensen-Shannon divergence, and to generate a drift correction feature vector sequence. The sleep index calculation module is used to calculate the set of sleep quality indexes for the current night. The trend modeling and visualization module is used to perform second-order regression modeling on a set of sleep quality indicators for consecutive nights, generate sleep quality trend curves, and visualize them.

[0007] Optionally, modules can be integrated using the following methods: Step 1: Collect sleep video frame sequences from the subjects; Step 2: Perform inter-frame difference on the sleep video frame sequence and accumulate frame by frame within a set window length to generate a vibration image frame sequence; Step 3: Perform frequency domain statistics and spatiotemporal feature extraction operations on the vibration image frame sequence to construct a vibration image feature vector sequence; Step 4: Input the vibration image feature vector sequence into the improved residual temporal convolutional network to perform sleep behavior feature modeling and generate a set of sleep behavior recognition sequences. The improved residual temporal convolutional network introduces spectral domain guided residual, drift-aware residual, variable temporal interval convolution and attention-guided residual bridge. Step 5: Perform distribution drift analysis and cross-night normalization on the vibration image frame sequence of consecutive nights using Gaussian kernel density estimation and Jensen-Shannon divergence to generate a drift correction feature vector sequence; Step 6: Calculate the set of sleep quality indicators for the current night based on the drift correction feature vector sequence and the sleep behavior recognition sequence set; Step 7: Model the trend changes of sleep quality index sequences for consecutive nights using a second-order regression method, generate sleep quality trend curves, and perform visualization analysis.

[0008] Optionally, the acquisition period of the sleep video frame sequence covers the subject's entire nighttime sleep period, and the subject maintains a static, supported posture during the acquisition process, and the subject's body does not contact any rigid structure other than the mattress.

[0009] Optionally, step two specifically includes: The sleep video frame sequence is arranged in chronological order to form a sleep image frame set, which includes several sleep images, each frame corresponding to a time step. Based on the set of sleep image frames, the sleep images of adjacent frames are subjected to inter-frame pixel difference to obtain a difference sleep image sequence, which is composed of difference sleep images; Set the cumulative window length, and within each time step, accumulate the difference sleep images of all frames within the cumulative window length frame by frame at the pixel position to generate the vibration image frame of the corresponding time step. The vibration image frames are arranged into a vibration image frame sequence according to time steps.

[0010] Optionally, step three specifically includes: Perform a two-dimensional Fourier transform on the image space coordinates of each vibration image frame to obtain a two-dimensional complex spectrum. The complex modulus of each two-dimensional complex spectrum is taken to obtain the spectrum amplitude distribution; The frequency mean of each frame of the vibration image is obtained by weighting and summing all frequency points in the two-dimensional complex spectrum according to the spectral amplitude distribution and dividing by the total number of frequency points. Based on the spectral amplitude, the squared deviation of each frequency point relative to the frequency mean distribution is calculated, and the root mean square value of the weighted squared deviation is used as the frequency dispersion. Based on the spectral amplitude distribution, the frequency point with the largest spectral amplitude is taken as the dominant frequency; Based on the spectral amplitude distribution, the frequency range of the corresponding breathing frequency band is set, and the sum of the spectral amplitudes of all frequency points within the frequency range is taken as the spectral energy of the breathing band. Divide the respiratory band spectral energy by the sum of the spectral amplitudes of all frequency points in the current two-dimensional complex spectrum to obtain the respiratory band power ratio of each frame of the vibration image. The body energy index of each vibration image frame is obtained by summing all pixel values ​​pixel by pixel and dividing by the total number of pixels. The frequency mean, frequency dispersion, dominant frequency, respiratory zone power ratio, and body energy index of each vibration image frame are used to form a vibration image feature vector. The vibration image feature vectors are then arranged in chronological order to form a vibration image feature vector sequence.

[0011] Optionally, step four specifically includes: The sleep behavior recognition sequence set includes sleep stage label sequences, micro-awakening event label sequences, body movement event label sequences, sleep confidence sequences, micro-awakening confidence sequences, and body movement confidence sequences; The vibration image feature vector sequence is input into an improved residual temporal convolutional network, which is composed of several stacked residual blocks. The input of the first residual block is the vibration image feature vector at each time step, and the input of the remaining residual blocks is the updated feature vector output by the previous residual block. Use the updated feature vector of the last residual block at the current time step as the sleep prediction feature vector; The sleep prediction feature vector is input into the sleep stage classification branch, the micro-arousal event discrimination branch, and the body movement event discrimination branch, respectively; Sleep stage labels and classification confidence scores are obtained through sleep stage classification branches; the sleep stage labels include 0: wakefulness, 1: non-rapid eye movement (NREM) stage 1, 2: NREM stage 2, 3: NREM stage 3, and 4: rapid eye movement (REM) sleep. Micro-awakening event labels and their classification confidence scores are obtained through the micro-awakening event discrimination branch; body movement event labels and their classification confidence scores are obtained through the body movement event discrimination branch. The sleep stage labels, micro-awakening event labels, body movement event labels, classification confidence scores of sleep stage labels, classification confidence scores of micro-awakening event labels, and classification confidence scores of body movement event labels for all time steps are respectively used to form sleep stage label sequences, micro-awakening event label sequences, body movement event label sequences, sleep confidence sequences, micro-awakening confidence sequences, and body movement confidence sequences.

[0012] Optionally, the step of generating an updated feature vector for each residual block in the improved residual temporal convolutional network specifically includes: Based on the feature vector sequence of vibration images, the difference in the dominant frequency between two adjacent time steps is calculated to obtain the rate of change of the dominant frequency. Set an adjustment coefficient, multiply the rate of change of the main frequency by the adjustment coefficient, add it to 1, and then use the Round function to generate the hole rate of the current time step; Based on the void ratio, the vibration image feature vector at the current time step is subjected to one-dimensional void convolution to obtain the main path feature vector. Extract the ratio of dominant frequency to respiratory band power from the feature vector of vibration images to form a frequency domain gating factor vector; Perform a one-dimensional convolution on the vibration image feature vector to obtain the intermediate residual feature vector; The gate factor vector is input into the MLP structure to generate the gate weight vector, and then normalized using the Sigmoid activation function to generate the normalized gate weight vector. The intermediate residual feature vector is multiplied element-wise with the normalized gate weight vector to obtain the gated response feature vector. The Gaussian kernel density estimation method is used to calculate the probability distribution of the vibration image feature vector at the current time step and the probability distribution of the vibration image feature vector at the historical time steps within a set window length, so as to obtain the feature distribution and baseline distribution at the current time step. Calculate the Kullback-Leibler divergence between the feature distribution and the baseline distribution at the current time step to obtain the drift divergence value. Input the drift divergence value into the MLP structure to generate the residual adjustment factor, and normalize it through the Sigmoid activation function to obtain the normalized residual adjustment factor. The drift residual feature vector is obtained by multiplying the gated response feature vector element by element with the normalized residual adjustment factor. The drift residual feature vector is globally averaged in the channel dimension, and channel attention is mapped through a single MLP structure to obtain the channel attention weight vector. The drift residual feature vectors of the current time step and all time steps before the current time step are combined into a time block. The time dimension average pooling operation is performed on the time block, and the time attention is mapped through a layer MLP structure to generate a time attention weight vector. The drift residual feature vector and the channel attention weight vector are multiplied element-wise to obtain the channel adjustment feature vector; the channel adjustment feature vector and the temporal attention weight vector are multiplied element-wise to obtain the attention adjustment residual vector. The attention-adjusted residual vector is added to and fused with the main path feature vector to obtain the updated feature vector of the current residual block at the current time step.

[0013] Optionally, step five specifically includes: Set the sliding window length, extract the feature vector sequence of vibration images from historical nights, and construct the historical window feature sequence; The Gaussian kernel density estimation method is used to estimate the current night feature distribution of the current night vibration image feature vector sequence and the historical baseline distribution of the historical window feature sequence; Obtain the average distribution of the current night feature distribution and the historical baseline distribution, and calculate the Kullback-Leibler divergence between the current night feature distribution and the historical baseline distribution and the average distribution, respectively, to generate the current night divergence index and the historical baseline divergence index; The Jensen-Shannon divergence value between the current night divergence index and the historical baseline divergence index is obtained by weighted averaging. The Jensen-Shannon divergence value is compared with a set drift threshold. If the Jensen-Shannon divergence value is greater than the drift threshold, cross-night normalization is performed, specifically as follows: The mean and standard deviation of the historical baseline distribution are calculated, and the vibration image feature vector of each time step of the current night is normalized by the Z-score normalization method, which introduces a regularization constant in the denominator. Set an update rate, and weight and fuse the current night feature distribution and the historical baseline distribution to obtain the historical baseline distribution for the next night; The normalized vibration image feature vector sequence for the current nighttime is used as the drift correction feature vector sequence.

[0014] Optionally, step six specifically includes: The set of sleep quality indicators includes sleep efficiency, duration of wakefulness after falling asleep, number of awakenings, spectral fluctuation, number of micro-awakening events, and body movement ratio. The drift correction feature vector sequence is used to generate a confidence correction factor sequence through a two-layer MLP structure; The confidence correction factor sequence was weighted and corrected step-by-step with the sleep confidence sequence, micro-arousal confidence sequence, and body movement confidence sequence to obtain the corrected sleep confidence sequence, corrected micro-arousal confidence sequence, and corrected body movement confidence sequence. Calculate the cumulative value of the corrected sleep confidence corresponding to the time step with label 0 in the sleep stage label sequence, and divide it by the cumulative value of the corrected sleep confidence sequence to obtain the sleep efficiency; The position where the first non-zero label appears in the sleep stage label sequence is taken as the sleep onset time point, and the weighted sum of the corrected sleep confidence corresponding to the time steps with zero labels after the sleep onset time point is calculated as the wakefulness time after falling asleep. A transition point in the sleep stage label sequence where the state changes from non-zero to zero and the corrected sleep confidence of the transition point is greater than a set threshold is recorded as a valid event, and the total number of valid events is taken as the number of awakenings. Extract the dominant frequency value sequence from the drift correction feature vector sequence and calculate the standard deviation of the dominant frequency value sequence as an index of spectral volatility; A time step in the micro-awakening event label sequence where the label is 1 and the corresponding corrected micro-awakening confidence is greater than the set micro-awakening confidence threshold is counted as a micro-awakening event. The total number of micro-awakening events is counted as the number of micro-awakening events. The ratio of the cumulative value of the corrected body movement confidence score corresponding to label 1 in the body movement event label sequence to the cumulative value of the corrected body movement confidence sequence is used as the body movement ratio index.

[0015] Optionally, the sleep quality trend curve includes a sleep efficiency trend curve, a wakefulness duration trend curve after falling asleep, a wakefulness frequency trend curve, a spectrum fluctuation trend curve, a micro-awakening event frequency trend curve, and a body movement ratio trend curve.

[0016] The beneficial effects of this invention are: First, this invention achieves non-contact, unconstrained micro-mechanical motion modeling by continuously acquiring sleep video frame sequences in the subject's natural sleep environment and constructing vibration image frame sequences based on inter-frame pixel differences and cumulative processing with a set window. This improves the visualization and resolution of key sleep behaviors such as respiratory fluctuations, body movement segments, and arousal events. Compared with traditional wearable sleep monitoring devices, this invention can obtain stable and continuous high temporal resolution sleep image sequences without interfering with the subject's natural sleep state, and maintains robustness to environmental disturbances.

[0017] Secondly, this invention introduces an improved residual temporal convolutional network to model vibration image frame sequences. This improved residual temporal convolutional network employs dilated convolution operations dynamically set based on the rate of change of the dominant frequency, effectively expanding the receptive field and improving the modeling ability for vibration image features of different rhythms. By introducing Kullback-Leibler divergence for distribution drift measurement, robust perception and dynamic adjustment of feature changes across time steps are achieved. Combining channel average pooling and temporal average pooling, channel attention and temporal attention are introduced to jointly regulate the residual path information flow, improving the network's ability to identify key behavioral states and its modeling accuracy. The improved residual temporal convolutional network exhibits stronger stability and recognition resolution in non-contact sleep behavior recognition scenarios.

[0018] Furthermore, this invention significantly improves the robustness and consistency of long-term vibration imaging sleep quality tracking modeling through distribution drift analysis and cross-night normalization. By constructing historical window feature sequences and current night feature distributions, and calculating Kullback-Leibler divergence and Jensen-Shannon divergence, sensitive detection of feature distribution changes is achieved, effectively identifying cross-night distribution drift phenomena. When significant drift is detected, the current night features are normalized using the Z-score normalization method based on the historical baseline distribution mean and standard deviation, and a regularization constant is introduced into the denominator to suppress model bias caused by abnormal fluctuations, improving the robustness of feature contrast and confidence inference, and enhancing the stable modeling capability of the system for long-term, multi-night vibration image behavior data.

[0019] In summary, this invention, by integrating vibration imaging and depth modeling technology, overcomes the limitations of traditional methods, such as insufficient dimensions of sleep behavior perception, poor model generalization, and lack of unified dimensions for trend analysis. It possesses non-invasive, high-precision, and sustainable sleep quality modeling capabilities. Attached Figure Description

[0020] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0021] Figure 1 This is a schematic diagram of the long-term tracking and modeling system for vibration photography sleep quality based on temporal convolutional networks proposed in this invention. Figure 2 This is a flowchart of the improved residual temporal convolutional network structure in the vibration photography sleep quality long-term tracking modeling system based on temporal convolutional networks proposed in this invention; Figure 3 This is a flowchart of drift correction and sleep index calculation in the long-term tracking modeling system for vibration photography sleep quality based on temporal convolutional networks proposed in this invention. Detailed Implementation

[0022] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0023] refer to Figures 1-3 A long-term tracking and modeling system for sleep quality based on temporal convolutional networks using vibration photography includes the following steps: The video acquisition module is used to acquire a sequence of sleep video frames from the subject. The vibration image construction module is used to perform inter-frame pixel difference on the sleep video frame sequence and accumulate it frame by frame within a set window length to generate a vibration image frame sequence. The statistical feature extraction module is used to perform frequency domain statistics and spatiotemporal feature extraction on the vibration image frame sequence to generate a vibration image feature vector sequence. The sleep behavior modeling module is used to model sleep behavior and perform multi-task recognition by passing the vibration image feature vector sequence through an improved residual temporal convolutional network, and output a set of sleep behavior recognition sequences. The cross-night drift correction module is used to perform distribution drift analysis on a sequence of vibration images from consecutive nights using Gaussian kernel density estimation and Jensen-Shannon divergence, and to generate a drift correction feature vector sequence. The sleep index calculation module is used to calculate the set of sleep quality indexes for the current night. The trend modeling and visualization module is used to perform second-order regression modeling on a set of sleep quality indicators for consecutive nights, generate sleep quality trend curves, and visualize them.

[0024] In this embodiment, the modules are connected through the following method: Step 1: Collect sleep video frame sequences from the subjects; Step 2: Perform inter-frame difference on the sleep video frame sequence and accumulate frame by frame within a set window length to generate a vibration image frame sequence; Step 3: Perform frequency domain statistics and spatiotemporal feature extraction operations on the vibration image frame sequence to construct a vibration image feature vector sequence; Step 4: Input the vibration image feature vector sequence into the improved residual temporal convolutional network to perform sleep behavior feature modeling and generate a set of sleep behavior recognition sequences. The improved residual temporal convolutional network introduces spectral domain guided residual, drift-aware residual, variable temporal interval convolution and attention-guided residual bridge. Step 5: Perform distribution drift analysis and cross-night normalization on the vibration image frame sequence of consecutive nights using Gaussian kernel density estimation and Jensen-Shannon divergence to generate a drift correction feature vector sequence; Step 6: Calculate the set of sleep quality indicators for the current night based on the drift correction feature vector sequence and the sleep behavior recognition sequence set; Step 7: Model the trend changes of sleep quality index sequences for consecutive nights using a second-order regression method, generate sleep quality trend curves, and perform visualization analysis.

[0025] In this embodiment, step one specifically includes: Video acquisition devices were deployed in the subjects' natural sleep environment to continuously acquire sleep video frame sequences of the subjects at a frame rate of 12 to 15 Hz and a spatial resolution of greater than or equal to 640×480 pixels, with the acquisition period covering the subjects' entire nighttime sleep period. During the collection process, the subject maintained a static support posture, and the subject's body did not come into contact with any rigid structure other than the mattress. The rigid structure included walls, furniture sides, and hard edges of the bed frame. The sleep video frame sequence is input into the cache module in chronological order.

[0026] In this embodiment, step two specifically includes: The sleep video frame sequence is arranged in chronological order to form a sleep image frame set, which includes several sleep images. Based on the set of sleep image frames, the sleep images of adjacent frames are subjected to inter-frame pixel difference to obtain a difference sleep image sequence, which is composed of difference sleep images; Set the cumulative window length, and within each time step, accumulate the difference sleep images of all frames within the cumulative window length frame by frame at the pixel position to generate the vibration image frame of the corresponding time step. The vibration image frames are arranged into a vibration image frame sequence according to time steps.

[0027] In this embodiment, step three specifically includes: Perform a two-dimensional Fourier transform on the image space coordinates of each vibration image frame to obtain a two-dimensional complex spectrum. The complex modulus of each two-dimensional complex spectrum is taken to obtain the spectrum amplitude distribution; The frequency mean of each frame of the vibration image is obtained by weighting and summing all frequency points in the two-dimensional complex spectrum according to the spectral amplitude distribution and dividing by the total number of frequency points. Based on the spectral amplitude, the squared deviation of each frequency point relative to the frequency mean distribution is calculated, and the root mean square value of the weighted squared deviation is used as the frequency dispersion. Based on the spectral amplitude distribution, the frequency point with the largest spectral amplitude is taken as the dominant frequency; Based on the spectral amplitude distribution, the frequency range of the corresponding breathing frequency band is set, and the sum of the spectral amplitudes of all frequency points within the frequency range is taken as the spectral energy of the breathing band. Divide the respiratory band spectral energy by the sum of the spectral amplitudes of all frequency points in the current two-dimensional complex spectrum to obtain the respiratory band power ratio of each frame of the vibration image. The body energy index of each vibration image frame is obtained by summing all pixel values ​​pixel by pixel and dividing by the total number of pixels. The frequency mean, frequency dispersion, dominant frequency, respiratory zone power ratio, and body energy index of each vibration image frame are used to form a vibration image feature vector. The vibration image feature vectors are then arranged in chronological order to form a vibration image feature vector sequence.

[0028] In this embodiment, step four specifically includes: The sleep behavior recognition sequence set includes sleep stage label sequences, micro-awakening event label sequences, body movement event label sequences, sleep confidence sequences, micro-awakening confidence sequences, and body movement confidence sequences; The vibration image feature vector sequence is input into an improved residual temporal convolutional network, which is composed of several stacked residual blocks. The input of the first residual block is the vibration image feature vector at each time step, and the input of the remaining residual blocks is the updated feature vector output by the previous residual block. Use the updated feature vector of the last residual block at the current time step as the sleep prediction feature vector; The sleep prediction feature vector is input into the sleep stage classification branch, the micro-arousal event discrimination branch, and the body movement event discrimination branch, respectively. The sleep stage classification branch includes a multilayer perceptron mapping structure and a Softmax classifier. The micro-arousal event discrimination branch and the body movement event discrimination branch both include a multilayer MLP structure and a Sigmoid activation function. In the sleep stage classification branch, the sleep prediction feature vector is input into the MLP structure to generate a class mapping vector. The class mapping vector is then used by a Softmax classifier to calculate the sleep stage probability distribution vector. The index of the maximum probability value in the sleep stage probability distribution vector is taken as the sleep stage label for the current time step, and the maximum probability value is used as the classification confidence of the sleep stage label. The sleep stage labels include 0: wakefulness, 1: non-rapid eye movement (NREM) stage 1, 2: NREM stage 2, 3: NREM stage 3, and 4: rapid eye movement (REM) sleep. In the micro-awake event discrimination branch, the sleep prediction feature vector is input into the MLP structure to generate micro-awake mapping values. The micro-awake mapping values ​​are then used to generate micro-awake event probability values ​​through the Sigmoid activation function. When the micro-awake event probability value is greater than the set micro-awake threshold, micro-awake event label 1 is generated: micro-awake state; otherwise, micro-awake event label 0 is generated: non-micro-awake state. The micro-awake event probability value is used as the classification confidence of the micro-awake event label. In the body movement event discrimination branch, the sleep prediction feature vector is input into the MLP structure to generate body movement mapping values. The body movement mapping values ​​are then used to generate body movement event probability values ​​through the Sigmoid activation function. When the body movement event probability value is greater than the set body movement threshold, body movement event label 1 is generated; otherwise, body movement event label 0 is generated. The body movement event probability value is used as the classification confidence of the body movement event label. The sleep stage labels, micro-awakening event labels, body movement event labels, classification confidence scores of sleep stage labels, classification confidence scores of micro-awakening event labels, and classification confidence scores of body movement event labels for all time steps are respectively composed into sleep stage label sequences, micro-awakening event label sequences, body movement event label sequences, sleep confidence sequences, micro-awakening confidence sequences, and body movement confidence sequences.

[0029] In this embodiment, the step of generating an updated feature vector for each residual block in the improved residual temporal convolutional network specifically includes: Based on the feature vector sequence of vibration images, the difference in the dominant frequency between two adjacent time steps is calculated to obtain the rate of change of the dominant frequency. Set an adjustment coefficient, multiply the rate of change of the main frequency by the adjustment coefficient, add it to 1, and then use the Round function to generate the hole rate of the current time step; Based on the void ratio, the vibration image feature vector at the current time step is subjected to one-dimensional void convolution to obtain the main path feature vector. Extract the ratio of dominant frequency to respiratory band power from the feature vector of vibration images to form a frequency domain gating factor vector; Perform a one-dimensional convolution on the vibration image feature vector to obtain the intermediate residual feature vector; The gate factor vector is input into the MLP structure to generate the gate weight vector, and then normalized using the Sigmoid activation function to generate the normalized gate weight vector. The intermediate residual feature vector is multiplied element-wise with the normalized gate weight vector to obtain the gated response feature vector. The Gaussian kernel density estimation method is used to calculate the probability distribution of the vibration image feature vector at the current time step and the probability distribution of the vibration image feature vector at the historical time steps within a set window length, so as to obtain the feature distribution and baseline distribution at the current time step. Calculate the Kullback-Leibler divergence between the feature distribution and the baseline distribution at the current time step to obtain the drift divergence value. Input the drift divergence value into the MLP structure to generate the residual adjustment factor, and normalize it through the Sigmoid activation function to obtain the normalized residual adjustment factor. The drift residual feature vector is obtained by multiplying the gated response feature vector element by element with the normalized residual adjustment factor. The drift residual feature vector is globally averaged in the channel dimension, and channel attention is mapped through a single MLP structure to obtain the channel attention weight vector. The drift residual feature vectors of the current time step and all time steps before the current time step are combined into a time block. The time dimension average pooling operation is performed on the time block, and the time attention is mapped through a layer MLP structure to generate a time attention weight vector. The drift residual feature vector and the channel attention weight vector are multiplied element-wise to obtain the channel adjustment feature vector; the channel adjustment feature vector and the temporal attention weight vector are multiplied element-wise to obtain the attention adjustment residual vector. The attention-adjusted residual vector is added to and fused with the main path feature vector to obtain the updated feature vector of the current residual block at the current time step.

[0030] In this embodiment, step five specifically includes: Set the sliding window length, extract the feature vector sequence of vibration images from historical nights, and construct the historical window feature sequence; The Gaussian kernel density estimation method is used to estimate the current night feature distribution of the current night vibration image feature vector sequence and the historical baseline distribution of the historical window feature sequence; Obtain the average distribution of the current night feature distribution and the historical baseline distribution, and calculate the Kullback-Leibler divergence between the current night feature distribution and the historical baseline distribution and the average distribution, respectively, to generate the current night divergence index and the historical baseline divergence index; The Jensen-Shannon divergence value between the current night divergence index and the historical baseline divergence index is obtained by weighted averaging. The Jensen-Shannon divergence value is compared with a set drift threshold. If the Jensen-Shannon divergence value is greater than the drift threshold, cross-night normalization is performed, specifically as follows: The mean and standard deviation of the historical baseline distribution are calculated, and the vibration image feature vector at each time step of the current night is normalized using the Z-score normalization method, which introduces a regularization constant into the denominator. ; in, This represents the normalized vibration image feature vector. Indicates the current nighttime time step Vibration image feature vector, Represents the mean of the historical baseline distribution. The standard deviation of the historical baseline distribution is represented by... Represents the regularization constant; Set an update rate, and weight and fuse the current night feature distribution and the historical baseline distribution to obtain the historical baseline distribution for the next night; The normalized vibration image feature vector sequence for the current nighttime is used as the drift correction feature vector sequence.

[0031] In this embodiment, step six specifically includes: The set of sleep quality indicators includes sleep efficiency, duration of wakefulness after falling asleep, number of awakenings, spectral fluctuation, number of micro-awakening events, and body movement ratio. The drift correction feature vector sequence is used to generate a confidence correction factor sequence through a two-layer MLP structure; The confidence correction factor sequence was weighted and corrected step-by-step with the sleep confidence sequence, micro-arousal confidence sequence, and body movement confidence sequence to obtain the corrected sleep confidence sequence, corrected micro-arousal confidence sequence, and corrected body movement confidence sequence. Calculate the cumulative value of the corrected sleep confidence corresponding to the time step with label 0 in the sleep stage label sequence, and divide it by the cumulative value of the corrected sleep confidence sequence to obtain the sleep efficiency; The position where the first non-zero label appears in the sleep stage label sequence is taken as the sleep onset time point, and the weighted sum of the corrected sleep confidence corresponding to the time steps with zero labels after the sleep onset time point is calculated as the wakefulness time after falling asleep. A transition point in the sleep stage label sequence where the state changes from non-zero to zero and the corrected sleep confidence of the transition point is greater than a set threshold is recorded as a valid event, and the total number of valid events is taken as the number of awakenings. Extract the dominant frequency value sequence from the drift correction feature vector sequence and calculate the standard deviation of the dominant frequency value sequence as an index of spectral volatility; A time step in the micro-awakening event label sequence where the label is 1 and the corresponding corrected micro-awakening confidence is greater than the set micro-awakening confidence threshold is counted as a micro-awakening event. The total number of micro-awakening events is counted as the number of micro-awakening events. The ratio of the cumulative value of the corrected body movement confidence score corresponding to label 1 in the body movement event label sequence to the cumulative value of the corrected body movement confidence sequence is used as the body movement ratio index.

[0032] In this embodiment, step seven specifically includes: The sleep quality trend curves include the sleep efficiency trend curve, the wakefulness duration after falling asleep trend curve, the number of awakenings trend curve, the spectrum fluctuation trend curve, the number of micro-awakening events trend curve, and the body movement ratio trend curve. Based on the fitting slope, extreme points, and fluctuation amplitude of each sleep quality trend curve, a rule-based classification method is used to obtain the trend classification label of each sleep quality trend curve. The trend classification label includes stable, improving, deteriorating, and fluctuating. The sleep quality trend curve and its corresponding trend category label are displayed graphically.

[0033] Example 1 To verify the feasibility of this invention in practice, it was applied to a real sleep monitoring scenario at a tertiary hospital's sleep medicine center, conducting non-contact continuous sleep behavior analysis and quality assessment tasks on outpatients. The participants in this project were 32 adults aged 23 to 58 years with mild to moderate sleep disorders. For two consecutive weeks, the participants completed nighttime sleep monitoring in the hospital's sleep monitoring room according to their natural sleep-wake cycles. During this period, traditional polysomnography signals and non-invasive video signal data constructed using the vibration photography and temporal modeling method described in this invention were simultaneously collected.

[0034] In this application scenario, the present invention uses a commercial-grade infrared night vision camera installed in each monitoring room to collect a sequence of sleep video frames at a frame rate of 15Hz and a resolution of 1280×720, covering the entire sleep process from pre-sleep preparation to natural awakening. The raw video data is stored in a local cache module and then input into a vibration image construction module in chronological order. Inter-frame pixel difference and windowed accumulation operations are performed to generate a vibration image frame sequence. Each vibration image frame is further input into a frequency domain statistical feature extraction module to extract the mean frequency, frequency dispersion, dominant frequency, respiratory band power ratio, and body movement energy index, generating a vibration image feature vector sequence. An improved residual temporal convolutional network is used to perform temporal modeling and sleep behavior recognition on the vibration image feature vector sequence. The recognition results include sleep stage label sequences, micro-awakening and body movement event label sequences, and their corresponding confidence distributions. To address the issue of data drift across nights, a drift detection mechanism based on Gaussian kernel density estimation and Jensen-Shannon divergence is introduced. When the detection result exceeds a threshold, Z-score normalization is triggered to ensure distribution consistency in multi-night modeling.

[0035] To further verify the accuracy and stability of the system of the present invention in sleep behavior recognition and quality index extraction, the performance of the system of the present invention was compared with that of a sleep analysis system based on optical flow method. The experimental results are shown in Table 1.

[0036] Table 1. Performance Comparison of the Two Invented Systems in Sleep Monitoring Tasks

[0037] As shown in Table 1, the system of this invention significantly outperforms the sleep analysis system based on optical flow in multiple performance indicators. In sleep stage recognition, compared to the optical flow-based sleep analysis system, this invention improves the accuracy of sleep stage recognition by over 16% by introducing frequency domain feature modeling and an improved temporal convolution structure. In the micro-awakening event recognition task, due to the system's fine sensitivity to vibration amplitude changes, the F1 score is improved by nearly 0.18. The recall rate for body movement event recognition is improved by 17.7%, demonstrating that the system of this invention has higher recall capability and can effectively cover more subtle movement states.

[0038] In long-term tracking and modeling tasks, the system of this invention effectively improves the stability and comparability of multi-night data by introducing cross-night drift analysis and normalization mechanisms, increasing the consistency coefficient of cross-night quality indicators to 0.942. The mean deviation of sleep efficiency is significantly reduced to only ±2.6%, which is 4.7 percentage points lower than that of sleep analysis systems based on optical flow, demonstrating that the system of this invention has higher quantitative accuracy. Through an enhanced residual time-series modeling structure and confidence correction mechanism, the system of this invention significantly improves the consistency of trend classification, achieving a sleep trend classification consistency rate of 86.9%, which is 21.4% higher than that of sleep analysis systems based on optical flow, making it more suitable for long-term health monitoring scenarios.

[0039] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A long-term tracking and modeling system for sleep quality based on temporal convolutional networks using vibration photography, characterized in that, include: The video acquisition module is used to acquire a sequence of sleep video frames from the subject. The vibration image construction module is used to perform inter-frame pixel difference on the sleep video frame sequence and accumulate it frame by frame within a set window length to generate a vibration image frame sequence. The statistical feature extraction module is used to perform frequency domain statistics and spatiotemporal feature extraction on the vibration image frame sequence to generate a vibration image feature vector sequence. The sleep behavior modeling module is used to model sleep behavior and perform multi-task recognition by passing the vibration image feature vector sequence through an improved residual temporal convolutional network, and output a set of sleep behavior recognition sequences. The cross-night drift correction module is used to perform distribution drift analysis on a sequence of vibration images from consecutive nights using Gaussian kernel density estimation and Jensen-Shannon divergence, and to generate a drift correction feature vector sequence. The sleep index calculation module is used to calculate the set of sleep quality indexes for the current night. The trend modeling and visualization module is used to perform second-order regression modeling on a set of sleep quality indicators for consecutive nights, generate sleep quality trend curves, and visualize them.

2. The long-term tracking and modeling system for vibration photography sleep quality based on temporal convolutional networks according to claim 1, characterized in that, The modules are connected in the following way: Step 1: Collect sleep video frame sequences from the subjects; Step 2: Perform inter-frame difference on the sleep video frame sequence and accumulate frame by frame within a set window length to generate a vibration image frame sequence; Step 3: Perform frequency domain statistics and spatiotemporal feature extraction operations on the vibration image frame sequence to construct a vibration image feature vector sequence; Step 4: Input the vibration image feature vector sequence into the improved residual temporal convolutional network to perform sleep behavior feature modeling and generate a set of sleep behavior recognition sequences. The improved residual temporal convolutional network introduces spectral domain guided residual, drift perception residual and attention guided residual in each residual block. Step 5: Perform distribution drift analysis and cross-night normalization on the vibration image frame sequence of consecutive nights using Gaussian kernel density estimation and Jensen-Shannon divergence to generate a drift correction feature vector sequence; Step 6: Calculate the set of sleep quality indicators for the current night based on the drift correction feature vector sequence and the sleep behavior recognition sequence set; Step 7: Model the trend changes of sleep quality index sequences for consecutive nights using a second-order regression method, generate sleep quality trend curves, and perform visualization analysis.

3. The long-term tracking and modeling system for vibration photography sleep quality based on temporal convolutional networks according to claim 2, characterized in that, The sleep video frame sequence was collected during a period covering the subject's entire nighttime sleep, and the subject maintained a static, supported posture during the collection process, with the subject's body not in contact with any rigid structure other than the mattress.

4. The long-term tracking and modeling system for vibration photography sleep quality based on temporal convolutional networks according to claim 2, characterized in that, Step two specifically includes: The sleep video frame sequence is arranged in chronological order to form a sleep image frame set, which includes several sleep images, each frame corresponding to a time step. Based on the set of sleep image frames, the sleep images of adjacent frames are subjected to inter-frame pixel difference to obtain a difference sleep image sequence, which is composed of difference sleep images; Set the cumulative window length, and within each time step, accumulate the difference sleep images of all frames within the cumulative window length frame by frame at the pixel position to generate the vibration image frame of the corresponding time step. The vibration image frames are arranged into a vibration image frame sequence according to time steps.

5. The long-term tracking and modeling system for vibration photography sleep quality based on temporal convolutional networks according to claim 2, characterized in that, Step three specifically includes: Perform a two-dimensional Fourier transform on the image space coordinates of each vibration image frame to obtain a two-dimensional complex spectrum. The complex modulus of each two-dimensional complex spectrum is taken to obtain the spectrum amplitude distribution; The frequency mean of each frame of the vibration image is obtained by weighting and summing all frequency points in the two-dimensional complex spectrum according to the spectral amplitude distribution and dividing by the total number of frequency points. Based on the spectral amplitude, the squared deviation of each frequency point relative to the frequency mean distribution is calculated, and the root mean square value of the weighted squared deviation is used as the frequency dispersion. Based on the spectral amplitude distribution, the frequency point with the largest spectral amplitude is taken as the dominant frequency; Based on the spectral amplitude distribution, the frequency range of the corresponding breathing frequency band is set, and the sum of the spectral amplitudes of all frequency points within the frequency range is taken as the spectral energy of the breathing band. Divide the respiratory band spectral energy by the sum of the spectral amplitudes of all frequency points in the current two-dimensional complex spectrum to obtain the respiratory band power ratio of each frame of the vibration image. The body energy index of each vibration image frame is obtained by summing all pixel values ​​pixel by pixel and dividing by the total number of pixels. The frequency mean, frequency dispersion, dominant frequency, respiratory zone power ratio, and body energy index of each vibration image frame are used to form a vibration image feature vector. The vibration image feature vectors are then arranged in chronological order to form a vibration image feature vector sequence.

6. The long-term tracking and modeling system for vibration photography sleep quality based on temporal convolutional networks according to claim 2, characterized in that, Step four specifically includes: The sleep behavior recognition sequence set includes sleep stage label sequences, micro-awakening event label sequences, body movement event label sequences, sleep confidence sequences, micro-awakening confidence sequences, and body movement confidence sequences; The vibration image feature vector sequence is input into an improved residual temporal convolutional network, which is composed of several stacked residual blocks. The input of the first residual block is the vibration image feature vector at each time step, and the input of the remaining residual blocks is the updated feature vector output by the previous residual block. Use the updated feature vector of the last residual block at the current time step as the sleep prediction feature vector; The sleep prediction feature vector is input into the sleep stage classification branch, the micro-arousal event discrimination branch, and the body movement event discrimination branch, respectively; Sleep stage labels and classification confidence scores are obtained through sleep stage classification branches; the sleep stage labels include 0: wakefulness, 1: non-rapid eye movement (NREM) stage 1, 2: NREM stage 2, 3: NREM stage 3, and 4: rapid eye movement (REM) sleep. Micro-awakening event labels and their classification confidence scores are obtained through the micro-awakening event discrimination branch; body movement event labels and their classification confidence scores are obtained through the body movement event discrimination branch. The sleep stage labels, micro-awakening event labels, body movement event labels, classification confidence scores of sleep stage labels, classification confidence scores of micro-awakening event labels, and classification confidence scores of body movement event labels for all time steps are respectively used to form sleep stage label sequences, micro-awakening event label sequences, body movement event label sequences, sleep confidence sequences, micro-awakening confidence sequences, and body movement confidence sequences.

7. The long-term tracking and modeling system for vibration photography sleep quality based on temporal convolutional networks according to claim 6, characterized in that, The steps for generating updated feature vectors for each residual block in the improved residual temporal convolutional network specifically include: Based on the feature vector sequence of vibration images, the difference in the dominant frequency between two adjacent time steps is calculated to obtain the rate of change of the dominant frequency. Set an adjustment coefficient, multiply the rate of change of the main frequency by the adjustment coefficient, add it to 1, and then use the Round function to generate the hole rate of the current time step; Based on the void ratio, the vibration image feature vector at the current time step is subjected to one-dimensional void convolution to obtain the main path feature vector. Extract the ratio of dominant frequency to respiratory band power from the feature vector of vibration images to form a frequency domain gating factor vector; Perform a one-dimensional convolution on the vibration image feature vector to obtain the intermediate residual feature vector; The gate factor vector is input into the MLP structure to generate the gate weight vector, and then normalized using the Sigmoid activation function to generate the normalized gate weight vector. The intermediate residual feature vector is multiplied element-wise with the normalized gate weight vector to obtain the gated response feature vector. The Gaussian kernel density estimation method is used to calculate the probability distribution of the vibration image feature vector at the current time step and the probability distribution of the vibration image feature vector at the historical time steps within a set window length, so as to obtain the feature distribution and baseline distribution at the current time step. Calculate the Kullback-Leibler divergence between the feature distribution and the baseline distribution at the current time step to obtain the drift divergence value. Input the drift divergence value into the MLP structure to generate the residual adjustment factor, and normalize it through the Sigmoid activation function to obtain the normalized residual adjustment factor. The drift residual feature vector is obtained by multiplying the gated response feature vector element by element with the normalized residual adjustment factor. The drift residual feature vector is globally averaged in the channel dimension, and channel attention is mapped through a single MLP structure to obtain the channel attention weight vector. The drift residual feature vectors of the current time step and all time steps before the current time step are combined into a time block. The time dimension average pooling operation is performed on the time block, and the time attention is mapped through a layer MLP structure to generate a time attention weight vector. The drift residual feature vector and the channel attention weight vector are multiplied element-wise to obtain the channel adjustment feature vector; the channel adjustment feature vector and the temporal attention weight vector are multiplied element-wise to obtain the attention adjustment residual vector. The attention-adjusted residual vector is added to and fused with the main path feature vector to obtain the updated feature vector of the current residual block at the current time step.

8. The long-term tracking and modeling system for vibration photography sleep quality based on temporal convolutional networks according to claim 2, characterized in that, Step five specifically includes: Set the sliding window length, extract the feature vector sequence of vibration images from historical nights, and construct the historical window feature sequence; The Gaussian kernel density estimation method is used to estimate the current night feature distribution of the current night vibration image feature vector sequence and the historical baseline distribution of the historical window feature sequence; Obtain the average distribution of the current night feature distribution and the historical baseline distribution, and calculate the Kullback-Leibler divergence between the current night feature distribution and the historical baseline distribution and the average distribution, respectively, to generate the current night divergence index and the historical baseline divergence index; The Jensen-Shannon divergence value between the current night divergence index and the historical baseline divergence index is obtained by weighted averaging. The Jensen-Shannon divergence value is compared with a set drift threshold. If the Jensen-Shannon divergence value is greater than the drift threshold, cross-night normalization is performed, specifically as follows: The mean and standard deviation of the historical baseline distribution are calculated, and the vibration image feature vector of each time step of the current night is normalized by the Z-score normalization method, which introduces a regularization constant in the denominator. Set an update rate, and weight and fuse the current night feature distribution and the historical baseline distribution to obtain the historical baseline distribution for the next night; The normalized vibration image feature vector sequence for the current nighttime is used as the drift correction feature vector sequence.

9. The long-term tracking and modeling system for vibration photography sleep quality based on temporal convolutional networks according to claim 2, characterized in that, Step six specifically includes: The set of sleep quality indicators includes sleep efficiency, duration of wakefulness after falling asleep, number of awakenings, spectral fluctuation, number of micro-awakening events, and body movement ratio. The drift correction feature vector sequence is used to generate a confidence correction factor sequence through a two-layer MLP structure; The confidence correction factor sequence was weighted and corrected step-by-step with the sleep confidence sequence, micro-arousal confidence sequence, and body movement confidence sequence to obtain the corrected sleep confidence sequence, corrected micro-arousal confidence sequence, and corrected body movement confidence sequence. Calculate the cumulative value of the corrected sleep confidence corresponding to the time step with label 0 in the sleep stage label sequence, and divide it by the cumulative value of the corrected sleep confidence sequence to obtain the sleep efficiency; The position where the first non-zero label appears in the sleep stage label sequence is taken as the sleep onset time point, and the weighted sum of the corrected sleep confidence corresponding to the time steps with zero labels after the sleep onset time point is calculated as the wakefulness time after falling asleep. A transition point in the sleep stage label sequence where the state changes from non-zero to zero and the corrected sleep confidence of the transition point is greater than a set threshold is recorded as a valid event, and the total number of valid events is taken as the number of awakenings. Extract the dominant frequency value sequence from the drift correction feature vector sequence and calculate the standard deviation of the dominant frequency value sequence as an index of spectral volatility; A time step in the micro-awakening event label sequence where the label is 1 and the corresponding corrected micro-awakening confidence is greater than the set micro-awakening confidence threshold is counted as a micro-awakening event. The total number of micro-awakening events is counted as the number of micro-awakening events. The ratio of the cumulative value of the corrected body movement confidence score corresponding to label 1 in the body movement event label sequence to the cumulative value of the corrected body movement confidence sequence is used as the body movement ratio index.

10. The long-term tracking and modeling system for vibration photography sleep quality based on temporal convolutional networks according to claim 2, characterized in that, The sleep quality trend curves include the sleep efficiency trend curve, the wakefulness duration trend curve after falling asleep trend curve, the number of awakenings trend curve, the spectrum fluctuation trend curve, the number of micro-awakening events trend curve, and the body movement ratio trend curve.

Images