Obstructive sleep apnea syndrome touchless screening system

Abstract

The application provides a non-contact screening system for obstructive sleep apnea syndrome. The non-contact screening system for obstructive sleep apnea syndrome only needs to collect a user's face video through a data collection device, can extract multi-dimensional biological characteristics related to obstructive sleep apnea syndrome, including craniofacial morphological risk characteristics and real-time physiological characteristics, and can realize risk screening of obstructive sleep apnea syndrome without any body contact through fusion analysis of the characteristics by a deep learning model, thereby significantly improving the convenience and comfort of screening while maintaining high diagnostic accuracy.

Description

Technical Field

[0001] This invention relates to the field of sleep breathing recognition technology, and in particular to a contactless screening system for obstructive sleep apnea syndrome. Background Technology

[0002] Diagnosis of obstructive sleep apnea syndrome (OSAS) primarily relies on polysomnography (PSG), which requires overnight monitoring in a specialized sleep monitoring center. Patients wear multiple sensor electrodes, including over ten physiological signal acquisition devices such as electroencephalogram (EEG), electrooculogram (EOG), electromyography (EMG), and electrocardiogram (ECG). This monitoring method has significant limitations: firstly, the unfamiliar environment of a hospital laboratory can affect the patient's natural sleep state, leading to distorted monitoring data; secondly, the complex connection process of numerous sensors can cause discomfort for patients; and most importantly, this monitoring method requires the full involvement of professional technicians, and the equipment is expensive and resources are limited, making it difficult to meet the needs of large-scale screening. Although home portable sleep monitoring devices (HSAT) reduce the number of sensors, they still require the wearing of nasal tubes, chest straps, and other devices, failing to fundamentally solve the issues of comfort and convenience.

[0003] The core challenge facing current OSAS screening is how to completely eliminate the discomfort and usage barriers caused to patients by various contact sensors while maintaining diagnostic accuracy. Summary of the Invention

[0004] The purpose of this invention is to provide a non-contact screening system for obstructive sleep apnea syndrome, in order to solve at least one of the technical problems existing in the prior art.

[0005] To solve the above-mentioned technical problems, the present invention provides a contactless screening system for obstructive sleep apnea syndrome, comprising: Data acquisition device: used to continuously acquire facial video sequences of the identified object and capture the physiological activity signals of the identified object at a frame rate of 30-60fps; Static feature extraction device: Based on the three-dimensional network model of the facial video sequence components, it automatically calculates the morphological parameters of the object to be identified, and analyzes the facial fat distribution characteristics and modified Mahalanobis grading through a deep convolutional network to complete the extraction of the static features; Dynamic feature extraction device: Based on remote photoplethysmography, the heart rate and blood oxygenation trend of the identified object are extracted, the respiratory rate and abnormalities are captured by optical flow method, and the eye dynamics are analyzed by an attention mechanism-enhanced LSTM network to detect the rapid eye movement sleep period, thus completing non-contact physiological signal monitoring and extraction of the dynamic features. Fusion analysis device: Construct a multimodal deep learning model that combines convolutional neural networks and recurrent neural networks. It extracts spatiotemporal features through a spatiotemporal feature extraction network and captures temporal relationships through a temporal modeling module. It uses an adaptive feature weighting mechanism to fuse static and dynamic features and outputs the predicted value of respiratory disorder index and risk level.

[0006] As a further improvement of the present invention, the static feature extraction device includes: Three-dimensional facial modeling unit: Extracts facial key points of the recognition object based on the facial video sequence acquired by the data acquisition device, and constructs a physically based deformable mesh model; Morphological parameter calculation unit: Calculates the morphological parameters of the identified object based on the deformable mesh model. The morphological parameters include at least the neck circumference, mandibular retrusion angle, and facial obesity index. The hierarchical evaluation unit extracts the mouth-opening action video of the identified object during sleep based on the facial video sequence, preprocesses the mouth-opening action video, and then performs a hierarchical evaluation. Feature fusion unit: used to integrate multi-source features through a gated attention mechanism; including: concatenating the feature vector of the current feature with the context feature vector of the context feature of the current feature, generating dynamic weights through linear transformation and sigmoid activation function, and dynamically calculating the importance weight of the current feature.

[0007] As a further improvement of the present invention, the hierarchical evaluation unit is configured to perform the following steps: Based on the facial video sequence, extract the mouth opening action video of the identified object during sleep, and preprocess the mouth opening action video to extract video segments containing temporal information as input; The spatiotemporal features of the video are extracted using a 3D convolutional neural network to generate a feature map that simultaneously encodes the spatial appearance and temporal dynamics. An attention pooling mechanism is used to adaptively weight the feature map, highlighting key spatiotemporal regions related to the degree of mouth opening and suppressing irrelevant information. Based on the weighted features, regression or classification is performed through a fully connected layer to output a graded score of 1 to 4, which quantitatively assesses the amplitude or severity of mouth opening movements.

[0008] As a further improvement of the present invention, the dynamic feature extraction device includes: Photoplethysmography signal processing unit: Based on the facial video sequence, extract the signal of the region of interest in the forehead, and reconstruct and synthesize the physiological signal through three-channel separation, phase decoupling, adaptive filtering and signal fusion; Respiratory motion analysis unit: Combining a hybrid method of dense optical flow and deep learning, it calculates the optical flow field of the nasal alar region of the identified object; reconstructs the breathing waveform of the identified object; and detects abnormal breathing based on the local mutation index; Eye movement sleep detection unit: Extracts the region of interest sequence of the eyes based on the facial video sequence, captures the spatiotemporal dynamic information of eye movements through a 3D convolutional neural network, and focuses on key time points and feature channels through a multi-head attention mechanism, outputting the probability of rapid eye movement based on attention-weighted features; Multi-signal synchronization unit: Employs dynamic time warping to align physiological signals, enabling the extraction and output of dynamic features.

[0009] As a further improvement of the present invention, the three-channel separation is specifically as follows: Based on the signal from the aforementioned region of interest in the forehead, hemoglobin absorption features are separated in the HSV color space, wherein: Specifically, ΔH represents the hue value or the angle value associated with the hue. In color spaces, hues are typically represented in angular form from 0° to 360°. arctan: The arctangent function, used to convert calculation results into radian angle values; R, G, B: These represent the values ​​of the red, green, and blue color channels, respectively, and are usually in the range of 0 to 255 or 0 to 1. (GB) and (2R-GB): are calculated based on the difference between RGB channels, representing the relative relationship between color channels.

[0010] As a further improvement of the present invention, the fusion analysis device includes a spatiotemporal feature extraction unit, a temporal modeling unit, and an adaptive feature fusion unit.

[0011] As a further improvement of the present invention, the spatiotemporal feature extraction unit is used to perform the following steps: A 3D residual convolutional neural network with a depth of 34 layers is used as the backbone network. The preprocessed facial video sequence is used as input to obtain the residual blocks contained in the backbone network structure. For each residual block, a 3D convolution kernel is used to extract spatiotemporal features: in, 3D convolution kernel, For the input cube, BN represents batch normalization; Each residual block outputs a 256-dimensional feature map, which is then processed by 3D global average pooling to obtain a 512-dimensional spatiotemporal feature vector.

[0012] As a further improvement of the present invention, the timing modeling unit is used to perform the following steps: A bidirectional long short-term memory network is used to process physiological signal time series, wherein the physiological signals include at least heart rate, respiration, and blood oxygenation. In a time series of length T, the input sequence is defined as follows: Each This represents the multi-channel physiological signal vector collected at time point T; the unit formula for updating the bidirectional long short-term memory network is based on the input sequence.

[0013] As a further improvement of the present invention, the unit formula for updating the bidirectional long short-term memory network based on the input sequence includes: in, The output of the forget gate is in the range [0,1] and is used to determine the information to be discarded from the cell state; It is the sigmoid activation function; Here is the weight matrix for the forget gate; To concatenate the hidden state from the previous time step with the current input; For the bias term of the forget gate; in, The input gate output is in the range [0,1] and is used to determine the new information added to the cell state; It is the sigmoid activation function; This is the weight matrix of the input gate; To concatenate the hidden state from the previous time step with the current input; This is the bias term for the input gate; in, This is the output gate, with a range between [0,1], used to determine the output value; It is the sigmoid activation function; This is the weight matrix of the output gate; To concatenate the hidden state from the previous time step with the current input; This is the bias term for the output gate; in, It represents the current cell state, used to update the cell state and preserve long-term memory; ⊙ represents element-wise multiplication; The cell state at the previous time step; tanh is the hyperbolic tangent activation function, used to generate new candidate values; This is the weight matrix for cell states; To concatenate the hidden state from the previous time step with the current input; This is a bias term for the cell state; in, The hidden state at the current moment; used to generate the output at the current moment, containing short-term memory.

[0014] As a further improvement of the present invention, the adaptive feature fusion unit is used to perform the following steps: Define a feature importance weight calculation module for static anatomical features. Dynamic physiological characteristics and video spatiotemporal features Perform weighted fusion: in, is the attention weight of the i-th feature, used to measure the importance of this feature during fusion, with a value range of [0,1], and the sum of all weights is 1; and These are learnable weight vectors and bias terms used to adjust features. Perform a linear transformation and score it.

[0015] For the i-th type of input feature, such as text features or image features; exp() is an exponential function used to amplify the fractional differences between features.

[0016] The summation of the exponential scores of all features s, d, and v is used as a normalization factor. This is the final feature vector after fusion; To weight each feature according to its attention weight; To sum all the weighted features, we obtain the fused features.

[0017] By adopting the above technical solution, the present invention has the following beneficial effects: The non-contact screening system for obstructive sleep apnea syndrome provided by this invention only requires a data acquisition device to collect facial video of the user, and can extract multi-dimensional biometric features related to obstructive sleep apnea syndrome, including craniofacial morphological risk features and real-time physiological features. At the same time, these features are fused and analyzed through a deep learning model to achieve risk screening for obstructive sleep apnea syndrome without any physical contact, which significantly improves the convenience and comfort of screening, while maintaining high diagnostic accuracy. Attached Figure Description

[0018] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is a schematic block diagram of a contactless screening system for obstructive sleep apnea syndrome. Detailed Implementation

[0020] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Those skilled in the art should understand that the following specific embodiments or implementation methods are a series of optimized configurations listed to further explain the specific content of the invention. These configuration methods can be combined or used in conjunction with each other, unless the invention explicitly states that some or a specific embodiment or implementation method cannot be associated with or used in conjunction with other embodiments or implementation methods. Furthermore, the following specific embodiments or implementation methods are merely optimized configurations and are not intended to limit the scope of protection of the invention.

[0022] Please see Figure 1 As shown, the contactless screening system for obstructive sleep apnea syndrome provided by the present invention includes: Data acquisition device: used to continuously acquire facial video sequences of the identified object and capture the physiological activity signals of the identified object at a frame rate of 30-60fps; Static feature extraction device: Based on a three-dimensional network model of facial video sequence components, it automatically calculates the morphological parameters of the object to be identified, and uses a deep convolutional network to analyze the facial fat distribution characteristics and a modified Mahalanobis grading to complete the extraction of static features; Dynamic feature extraction device: Based on remote photoplethysmography, it extracts the heart rate and blood oxygenation trends of the identified object, captures respiratory rate and abnormalities through optical flow method, and uses an attention-enhanced LSTM network to analyze eye dynamics to detect REM sleep, thus completing non-contact physiological signal monitoring and dynamic feature extraction. Fusion analysis device: Construct a multimodal deep learning model that combines convolutional neural networks and recurrent neural networks. It extracts spatiotemporal features through a spatiotemporal feature extraction network and captures temporal relationships through a temporal modeling module. It uses an adaptive feature weighting mechanism to fuse static and dynamic features and outputs the predicted value of respiratory disorder index and risk level.

[0023] In this application, the data acquisition device includes a high-sensitivity imaging unit for non-contact acquisition of physiological signals of the identified object; in this application, the high-sensitivity imaging unit includes a visible light RGB camera, suitable for monitoring scenarios with sufficient ambient light (>50 lux); and a near-infrared camera (wavelength 850 nm) in conjunction with an active supplemental lighting LED array to adapt to low-light environments.

[0024] The high-sensitivity imaging units in this application all support 1080p resolution and extend the dynamic range through hardware-level HDR technology to ensure that facial details can still be preserved in scenes with strong contrast between light and dark, and realize continuous acquisition of facial video sequences when the subject is sleeping; and the video acquisition of the high-sensitivity imaging units is performed at a configurable frame rate (30 / 45 / 60fps) and compressed into H.264 format in real time through a hardware encoder.

[0025] To ensure the quality of physiological signals from the high-sensitivity imaging unit, a dual verification mechanism is implemented: First, motion blur detection is performed on each frame (calculating the Laplacian variance), and the frame rate is automatically increased when the detected value is below a threshold; second, the displacement of facial feature points in adjacent frames is calculated using optical flow, and dynamic frame rate adjustment is triggered when the displacement exceeds a preset threshold (e.g., 5 pixels). Simultaneously, the data acquisition device continuously monitors facial coverage (based on a semantic segmentation network), and issues a real-time alert when more than 30% of the face is detected to be obscured.

[0026] In this application, the data acquisition device also includes an intelligent exposure control unit, which employs a three-level adjustment mechanism: First, based on image histogram analysis, the brightness distribution of the current scene is calculated, dividing the image acquired by the high-sensitivity imaging unit into a facial region (ROI) and a background region; then, local exposure compensation is performed for the facial region, dynamically adjusting the sensor gain and exposure time through a PID control algorithm; finally, a deep learning-based noise suppression algorithm (using a pre-trained DnCNN network) is used to eliminate image noise under low light conditions. This system can complete exposure adjustment within 0.2 seconds, maintaining a constant brightness (target value 70-90 gray levels) in the facial region.

[0027] In a preferred embodiment of this application, a high-sensitivity imaging unit can be installed at the head of the bed in the bedroom of the person being identified, approximately 0.8-1.2 meters away from the face of the person being identified; after the non-contact screening system for obstructive sleep apnea syndrome is activated, the data acquisition device performs the following process: An ambient light sensor detects the current illuminance and automatically selects either RGB or infrared mode. The identified subject completes a 5-second calibration action (slowly turning its head towards the high-sensitivity imaging unit) via voice prompts, during which the data acquisition device establishes a baseline facial model; During the continuous monitoring phase, a video clip (including timestamps and body position markers) is automatically saved every 10 minutes. For example, in a nighttime environment (illuminance of 5 lux), the data acquisition device switches to infrared mode, activates an 850nm fill light (intensity automatically adjusted to 10 lux), and acquires video at a frame rate of 45fps. When the identified object turns to the side, causing partial occlusion of the face, the data acquisition device adjusts the ROI area through 3D pose estimation to maintain an effective signal acquisition window.

[0028] The static feature extraction device employs a multimodal feature fusion strategy to achieve automated measurement and static feature extraction of anatomical risk features.

[0029] Specifically, the static feature extraction device is used to construct a three-dimensional mesh model based on facial key points, automatically calculate morphological parameters such as neck circumference and mandibular retraction angle, and analyze facial fat distribution characteristics and improved Mahalanobis grading through deep convolutional networks.

[0030] In this application, the static feature extraction device includes: 3D facial modeling unit: Extracts key facial points of the object to be identified based on facial video sequences acquired by the data acquisition device, and constructs a physically based deformable mesh model; Morphological parameter calculation unit: Calculates the morphological parameters of the identified object based on the deformable mesh model. The morphological parameters include at least the neck circumference, mandibular retrusion angle, and facial obesity index. The hierarchical evaluation unit extracts videos of the mouth opening movements of the identified object during sleep based on facial video sequences, preprocesses the mouth opening movement videos, and then performs hierarchical evaluation. Feature fusion unit: used to integrate multi-source features through a gated attention mechanism; including: concatenating the feature vector based on the current feature with the context feature vector of the current feature's context features, generating dynamic weights through linear transformation and sigmoid activation function, and dynamically calculating the importance weight of the current feature.

[0031] Specifically, the 3D facial modeling unit can initially generate facial mesh deformation by using the Blendshape interpolation method to extract 106 key points from facial video sequences acquired by a data acquisition device; then, physical constraints are introduced for optimization to ensure that the deformation results conform to biomechanical laws; then, a 3D facial mesh containing 512 vertices is reconstructed; finally, the mesh surface is smoothed and geometric details are enhanced by a curvature optimization algorithm, ultimately generating a high-quality 3D facial reconstruction mesh result that conforms to the motion of a real human face.

[0032] Furthermore, in the process of optimizing the solution by introducing physical constraints, a deformation energy function is introduced. In this application, the deformation energy function is defined as: in: Matching energy: Used to ensure that key points match the target key points; This represents the i-th vertex (or key point) on the model. This represents the corresponding key points on the target shape, specifically a data fitting term. By minimizing this energy, the model's key points are made as close as possible to the target position. 106 is used to represent the total number of key points; Smooth Energy: Used to maintain smoothness and continuity; This represents all connected pairs of vertices in the grid; This represents the target offset between vertices i and j in the original mesh. It is used to penalize the deviation between the offsets of adjacent vertices and the ideal state, prevent mesh distortion during the optimization process, and ensure that the optimized mesh maintains reasonable local geometric relationships. Prior energy: Used to constrain the model within a reasonable statistical shape space; The set of vertices representing the average shape (or prior shape); Typically, it is a weight matrix or transformation matrix, which may come from statistical methods such as principal component analysis (PCA). It is used to prevent the model from being optimized to an unreasonable shape, and to regularize the optimization process using prior knowledge. It also helps to deal with data noise or missing key points. Parameter settings: , , .

[0033] The morphological parameter calculation unit is used at least to measure neck circumference, mandibular retrusion angle, and facial obesity index. In this application, the measurement of neck circumference specifically includes: Based on the horizontal plane of the C3 vertebra located on the 3D facial reconstruction mesh, calculate the perimeter of the neck cross section passing through the laryngeal protuberance: in, For the cross-sectional contour points, N=36 uniform sampling points.

[0034] The measurement of the mandibular retrusion angle is specifically as follows: connect the subnasal point (Sn), the anterior chin point (Pog), and the cervical vertebra point (Cv), and calculate the included angle in three-dimensional space: arccos is the inverse cosine function, used to convert the dot product result into an angle (ranging from 0 to radians). and It represents the vector connecting anatomical landmarks.

[0035] The facial obesity index is calculated by extracting facial fat distribution features through a ResNet-18 network, outputting a 128-dimensional embedding vector, and then calculating the cosine similarity with a standard obesity template.

[0036] The grading assessment unit is configured to perform the following steps: Based on facial video sequences, the video of the mouth opening action of the identified object during sleep is extracted, and the video of the mouth opening action is preprocessed to extract video segments containing temporal information as input; The spatiotemporal features of the video are extracted using a 3D convolutional neural network to generate a feature map that simultaneously encodes the spatial appearance and temporal dynamics. Specifically, the 3D convolutional neural network employs an improved RankingLoss model for improvement, namely: in, This is the marginal threshold; For regression weights, , y represents the model's prediction score (e.g., relevance score) for samples i and j; y represents the sample's true label vector. An attention pooling mechanism is used to adaptively weight the feature map, highlighting key spatiotemporal regions related to the degree of mouth opening and suppressing irrelevant information. Based on the weighted features, regression or classification is performed through a fully connected layer to output a graded score of 1 to 4, which quantitatively assesses the amplitude or severity of mouth opening movements.

[0037] The feature fusion unit integrates multi-source features using a gated attention mechanism, specifically: in, There are 4 input features The weighted sum of (i=1~4); These are the corresponding weighting coefficients; For each feature vector, For contextual features, σ is the sigmoid activation function, which compresses the output to the [0,1] interval; ; ] indicates that the features With context features Then, the parts are assembled.

[0038] Furthermore, contextual features In a specific task or scenario, it refers to the current feature. Auxiliary features that provide supplementary information, contextual relationships, or global environment; contextual features. Instead of directly describing the features of the current target itself, information is usually extracted from a broader range (such as previous and next frames in a time series, the surrounding area in space, or other relevant modalities) to help understand the meaning and importance of the current features.

[0039] This process can be understood as: based on the current features and context information The importance weight of the feature is dynamically calculated.

[0040] In a preferred embodiment of this application, feature extraction for a BMI=28 target includes: The 3D facial modeling unit performs 3D facial modeling: a 512-vertex mesh is generated from 106 key points, and the deformation energy is reduced to 0.03 after 10 iterations of optimization; Morphological parameter calculation unit performed morphological measurements: the calculated neck circumference was 38.2cm, the mandibular retraction angle was 17.8°, and the facial obesity index was 0.67 (normal range 0.3-0.7). The grading assessment unit performed the Mahalanobis grading: analysis of the mouth-opening movement video, with a network output score of 2.3 (corresponding to clinical assessment level II). Feature fusion of the feature fusion unit: the gating weights are assigned as [0.35, 0.25, 0.3, 0.1], and a 128-dimensional comprehensive feature vector is finally generated; The static feature extraction device automatically generates anatomical risk warning labels such as "short and thick neck, and slightly receding mandible" based on the fusion features.

[0041] The dynamic feature extraction device is used to achieve non-contact physiological signal monitoring, including: The photoplethysmography signal processing unit extracts signals from the forehead region of interest based on facial video sequences, and reconstructs and synthesizes physiological signals through three-channel separation, phase decoupling, adaptive filtering, and signal fusion. Respiratory motion analysis unit: Combining a hybrid approach of dense optical flow and deep learning, it calculates the optical flow field of the nasal alar region of the identified object; reconstructs the breathing waveform of the identified object; and detects abnormal breathing based on the local mutation index; Eye movement sleep detection unit: Extracts region of interest sequences of the eyes based on facial video sequences, captures spatiotemporal dynamic information of eye movements through a 3D convolutional neural network; and focuses on key time points and feature channels through a multi-head attention mechanism to output the probability of rapid eye movement based on attention-weighted features. Multi-signal synchronization unit: Employs dynamic time warping to align physiological signals, enabling the extraction and output of dynamic features.

[0042] Specifically, the rPPG signal processing unit uses an improved rPPG technique to extract heart rate and blood oxygenation trends. It first extracts signals from the region of interest in the forehead and decomposes them into three independent channels. Then, it separates the physiological signal components in each channel using phase decoupling technology to eliminate mutual interference. Next, it performs adaptive filtering on each channel, dynamically adjusting the filtering parameters according to signal characteristics to enhance effective information and suppress noise. Finally, it fuses the processed three-channel signals to reconstruct a high-quality, low-noise synthetic physiological signal, achieving robust extraction and enhancement of forehead physiological information.

[0043] The three-channel separation is specifically as follows: Based on the signal from the region of interest in the forehead, hemoglobin absorption characteristics are separated in the HSV color space, where: Specifically, ΔH represents the hue value or the angle value associated with the hue. In color spaces, hues are typically represented in angular form from 0° to 360°. arctan: The arctangent function, used to convert calculation results into radian angle values; R, G, B: These represent the values ​​of the red, green, and blue color channels, respectively, and are usually in the range of 0 to 255 or 0 to 1. (GB) and (2R-GB): are calculated based on the difference between RGB channels, representing the relative relationship between color channels.

[0044] Phase decoupling specifically involves constructing a phase decoupling matrix: in, , It covers the typical heart rate range.

[0045] Adaptive filtering employs variational mode decomposition (VMD) for signal enhancement. Set the number of modes K=3 and the penalty factor α=2000. This indicates that the objective function to be minimized is [the function to be optimized]. A set of functions; A set of frequency parameters; Summing over k components means this is a multi-component optimization problem; The square of the L2 norm (i.e., energy) represents the energy measure of the expression within the parentheses; It is the Dirac delta function (unit impulse function); It is a kernel function related to the Hilbert transform (the impulse response of the Hilbert transform is...) Here, there is an additional imaginary unit j, used to represent a specific linear transformation (e.g., the construction of a certain analytic signal); To analyze the signal, Calculate the time derivative of the convolution result, that is, calculate the instantaneous rate of change of the transformed signal. This is a complex exponential term, essentially a frequency of... The inverse Fourier transform kernel is used to convert a signal from the frequency domain to the time domain, or to perform frequency shifting (demodulation) on a signal.

[0046] The respiratory motion analysis unit is used to detect abnormal breathing using a hybrid approach combining dense optical flow and deep learning; it includes optical flow field calculation in the nasal alar region, respiratory waveform reconstruction, and abnormal breathing detection; specifically, the optical flow field calculation in the nasal alar region involves: in, For Gaussian kernel function, For feature point weights, ( , Let be the input feature of the i-th training sample. For input (x,y) and training samples ( , The difference between the two is used to calculate their similarity.

[0047] The reconstructed breathing waveform is used to encode the optical flow sequence using a GRU network. The output breathing waveform is then generated by a 1D deconvolution layer.

[0048] This represents the hidden state at time step t, which is the output of the GRU and contains sequence information up to the current time step. The computation function of the GRU unit controls the flow and updating of information through a gating mechanism; This indicates that the input features (or input data) at time step t are the inputs of the current time step; This represents the hidden state at time step t−1, carrying information from all previous time steps, and is used to characterize the memory properties of the RNN.

[0049] Abnormal respiration detection specifically involves calculating the local mutation index: when The event was marked as a suspected sleep apnea event.

[0050] Expressing expectations, These represent the signals at "time t" and "time t+1" in the time series data, respectively. It represents the "change" (difference) of a signal from time t to t+1, reflecting the instantaneous dynamics of the signal; L2 norm (Euclidean norm) is used to convert a "change" from a vector to a scalar (i.e., to calculate the "magnitude" or "amplitude" of the change). like For scalars (such as single-sensor data), the L2 norm degenerates into an absolute value, i.e. =∣ |

[0051] The eye movement sleep detection unit is used to extract the region of interest sequence of the eyes based on the facial video sequence, capture the spatiotemporal dynamic information of eye movements through a 3D convolutional neural network, and focus on key time points and feature channels through a multi-head attention mechanism to output the probability of rapid eye movement based on attention-weighted features.

[0052] Specifically, the 3D convolutional neural network is a three-dimensional residual convolutional neural network with a depth of 18 layers (3DResNet-18); furthermore, the multi-head attention mechanism is calculated as follows: in, The attention weights, calculated at time step t, are typically a probability distribution (the sum of all weights is 1) used to represent the importance of different pieces of information. softmax() is the softmax activation function that transforms an input vector into a probability distribution; It is a learnable weight vector (row vector) used to linearly transform the output of tanh, compressing it into a scalar or low-dimensional vector; tanh() is the hyperbolic tangent activation function, which maps the input to the range [-1,1], increasing the nonlinear expressive power; It is a learnable weight matrix. This represents the hidden state of the LSTM at time step t, containing timing information. It is a learnable weight matrix. It is a context vector.

[0053] The multi-signal synchronization unit uses dynamic time warping (DTW) to align various physiological signals, thereby enabling the extraction and output of dynamic features.

[0054] Specifically, dynamic time warping (DTW) aligns various physiological signals as follows: Set the step size constraint to ±10% of the sampling rate.

[0055] in, Represents a sequence The former Elements and sequences The former The cumulative distance between elements; For the first sequence The 1 element (which can be a scalar or a vector); For the second sequence The 1 element (which can be a scalar or a vector); This is the local distance between two elements (such as Euclidean distance, Manhattan distance, etc.), which measures the difference between elements at the current position.

[0056] To select the path with the minimum cumulative distance from three possible preceding states, this embodies the "optimal substructure" property of dynamic programming: : indicates skipping The cumulative distance at that time (corresponding to the "delete" operation); : indicates skipping The cumulative distance at that time (corresponding to the "insert" operation); : indicates matching and The cumulative distance at that time (corresponding to "replace" or "match" operation).

[0057] In a preferred embodiment of this application, the extraction of dynamic feature information using a dynamic feature extraction device specifically involves: Analyze and process a 30-minute nighttime monitoring video; The rPPG signal processing unit of the photoplethysmography (PPG) procedure includes: extracting the signal from the forehead region, decomposing it using VMD to obtain a heart rate of 62 bpm, and showing a blood oxygen trend that dropped below 88% three times. The respiratory motion analysis unit performs respiratory analysis, including: based on nasal ala optical flow detection, a respiratory rate of 12 breaths / min is detected, and two respiratory arrest events lasting >10 seconds are marked; The eye movement sleep detection unit performed eye movement detection, including detecting REM sleep features with a probability of 87% during the period from 23:15 to 23:20 based on an attention LSTM network. The multi-signal synchronization unit performs signal alignment, including correcting the time deviation between apnea events and blood oxygen decline to <0.5 seconds using DTW; The dynamic feature extraction device automatically generates clinical event association information for "three simultaneous occurrences of decreased blood oxygenation and apnea".

[0058] The fusion analysis device includes a spatiotemporal feature extraction unit, a temporal modeling unit, and an adaptive feature fusion unit.

[0059] The spatiotemporal feature extraction unit is used to perform the following steps: A 3D residual convolutional neural network with a depth of 34 layers is used as the backbone network. The preprocessed facial video sequence is used as input to obtain the backbone network structure containing residual blocks. Spatiotemporal features are extracted for each residual block using a 3D convolution kernel: in, 3D convolution kernel, For the input cube, BN represents batch normalization; Each residual block outputs a 256-dimensional feature map, which is then processed by 3D global average pooling to obtain a 512-dimensional spatiotemporal feature vector.

[0060] The timing modeling unit is used to perform the following steps: A bidirectional long short-term memory network is used to process physiological signal time series, which include at least heart rate, respiration, and blood oxygenation. In a time series of length T, the input sequence is defined as follows: Each Represents the multi-channel physiological signal vector collected at time point T; the unit formula for updating the bidirectional long short-term memory network based on the input sequence.

[0061] The unit formulas for updating a bidirectional long short-term memory network based on input sequences include: in, The output of the forget gate is in the range [0,1] and is used to determine the information to be discarded from the cell state; It is the sigmoid activation function; Here is the weight matrix for the forget gate; To concatenate the hidden state from the previous time step with the current input; For the bias term of the forget gate; in, The input gate output is in the range [0,1] and is used to determine the new information added to the cell state; It is the sigmoid activation function; Here is the weight matrix of the input gate; To concatenate the hidden state from the previous time step with the current input; This is the bias term for the input gate; in, This is the output gate, with a range between [0,1], used to determine the output value; It is the sigmoid activation function; This is the weight matrix of the output gate; To concatenate the hidden state from the previous time step with the current input; This is the bias term for the output gate; in, It represents the current cell state, used to update the cell state and preserve long-term memory; ⊙ represents element-wise multiplication; The cell state at the previous time step; tanh is the hyperbolic tangent activation function, used to generate new candidate values; This is the weight matrix for cell states; To concatenate the hidden state from the previous time step with the current input; This is a bias term for the cell state; in, The hidden state at the current moment; used to generate the output at the current moment, containing short-term memory.

[0062] The adaptive feature fusion unit is used to perform the following steps: Define a feature importance weight calculation module for static anatomical features. Dynamic physiological characteristics and video spatiotemporal features Perform weighted fusion: in, is the attention weight of the i-th feature, used to measure the importance of this feature during fusion, with a value range of [0,1], and the sum of all weights is 1; and These are learnable weight vectors and bias terms used to adjust features. Perform a linear transformation and score it.

[0063] For the i-th type of input feature, such as text features or image features; exp() is an exponential function used to amplify the fractional differences between features.

[0064] The summation of the exponential scores of all features s, d, and v is used as a normalization factor. This is the final feature vector after fusion; To weight each feature according to its attention weight; To sum all the weighted features, we obtain the fused features.

[0065] The fusion analysis device ultimately outputs two prediction results through a fully connected layer: the AHI regression value and the risk level probability.

[0066] AHI regression value: , is a linear transformation that maps the fused features to the predicted values ​​of AHI.

[0067] in, The AHI (apnea-hypopnea index) value predicted by the model is a continuous variable and is the output of the regression task.

[0068] The weight vector used for AHI prediction is the parameter that the model needs to learn.

[0069] This is the fused feature vector (possibly from the previously mentioned multi-feature fusion mechanism); This is a bias term used to adjust the baseline value of the regression results.

[0070] Risk level probability: in, Let be the probability that a sample belongs to the k-th risk level, with a value range of [0,1]. The sum of the probabilities of all risk levels is 1. softmax() is the softmax activation function, used to convert a linear output into a probability distribution; The weight matrix used for risk level prediction is the parameter that the model needs to learn. This is the bias term for risk level prediction; This represents the probability of taking the k-th element of the softmax output vector, i.e., the probability of the k-th class.

[0071] In a preferred embodiment of this application, a non-contact screening system for obstructive sleep apnea syndrome is used to perform overnight monitoring and analysis on individuals suspected of having obstructive sleep apnea syndrome. Specifically: Video data processing using a data acquisition device includes: dividing an 8-hour video into 900 segments of 32 frames each, extracting spatiotemporal features using 3D-ResNet, and obtaining a 900×512 feature matrix after average pooling; The static feature extraction device performs physiological signal processing including: inputting heart rate and respiratory signals into a bidirectional LSTM in 5-second segments, and outputting 3600×256 temporal features; The collected static features are fused to calculate and obtain the mandibular retraction angle of the identified object as 12°, the neck circumference as 42cm, etc., and the anatomical features are standardized to form a 16-dimensional vector. Dynamic feature information is extracted using a dynamic feature extraction device and adaptively weighted. The feature weights are calculated as follows: , , ; The fusion analysis device calculates AHI=28.7 (moderate OSAS) through the fusion features via a fully connected layer, with a risk level probability distribution of [0.02, 0.15, 0.68, 0.15] corresponding to low, mild, medium, and high risks. Furthermore, the report generated by the non-contact screening system for obstructive sleep apnea syndrome states that "video features show frequent apnea, highly synchronized with events of decreased blood oxygenation, and anatomical features suggest a risk of upper airway stenosis."

[0072] In summary, the contactless screening system for obstructive sleep apnea syndrome provided by this invention only requires the acquisition of facial video of the user through a data acquisition device to extract multi-dimensional biometric features related to obstructive sleep apnea syndrome, including craniofacial morphological risk features and real-time physiological features. At the same time, by fusing and analyzing these features through a deep learning model, it can achieve risk screening for obstructive sleep apnea syndrome without any physical contact, significantly improving the convenience and comfort of screening while maintaining high diagnostic accuracy.

[0073] It should be noted that any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a specific logical function or process, and the scope of the preferred embodiments of this solution includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which the embodiments of this solution pertain. The processor performs the various methods and processes described above. For example, the method embodiments of this solution can be implemented as software programs tangibly contained in a machine-readable medium, such as memory. In some embodiments, part or all of the software program can be loaded and / or installed via memory and / or a communication interface. When the software program is loaded into memory and executed by the processor, one or more steps of the methods described above can be performed. Alternatively, in other embodiments, the processor can be configured to perform one of the methods described above by any other suitable means (e.g., by means of firmware).

[0074] The logic and / or steps represented in the flowchart or otherwise described herein may be specifically implemented in any readable storage medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0075] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A contactless screening system for obstructive sleep apnea syndrome, characterized in that, include: Data acquisition device: used to continuously acquire facial video sequences of the identified object and capture the physiological activity signals of the identified object at a frame rate of 30-60fps; Static feature extraction device: Based on the three-dimensional network model of the facial video sequence components, it automatically calculates the morphological parameters of the object to be identified, and analyzes the facial fat distribution characteristics and modified Mahalanobis grading through a deep convolutional network to complete the extraction of the static features; Dynamic feature extraction device: Based on remote photoplethysmography, the heart rate and blood oxygenation trend of the identified object are extracted, the respiratory rate and abnormalities are captured by optical flow method, and the eye dynamics are analyzed by an attention mechanism-enhanced LSTM network to detect the rapid eye movement sleep period, thus completing non-contact physiological signal monitoring and extraction of the dynamic features. Fusion analysis device: Construct a multimodal deep learning model that combines convolutional neural networks and recurrent neural networks. It extracts spatiotemporal features through a spatiotemporal feature extraction network and captures temporal relationships through a temporal modeling module. It uses an adaptive feature weighting mechanism to fuse static and dynamic features and outputs the predicted value of respiratory disorder index and risk level.

2. The contactless screening system for obstructive sleep apnea syndrome according to claim 1, characterized in that, The static feature extraction device includes: Three-dimensional facial modeling unit: Extracts facial key points of the recognition object based on the facial video sequence acquired by the data acquisition device, and constructs a physically based deformable mesh model; Morphological parameter calculation unit: Calculates the morphological parameters of the identified object based on the deformable mesh model. The morphological parameters include at least the neck circumference, mandibular retrusion angle, and facial obesity index. The hierarchical evaluation unit extracts the mouth-opening action video of the identified object during sleep based on the facial video sequence, preprocesses the mouth-opening action video, and then performs a hierarchical evaluation. Feature fusion unit: used to integrate multi-source features through a gated attention mechanism; including: concatenating the feature vector of the current feature with the context feature vector of the context feature of the current feature, generating dynamic weights through linear transformation and sigmoid activation function, and dynamically calculating the importance weight of the current feature.

3. The contactless screening system for obstructive sleep apnea syndrome according to claim 2, characterized in that, The hierarchical evaluation unit is configured to perform the following steps: Based on the facial video sequence, extract the mouth opening action video of the identified object during sleep, and preprocess the mouth opening action video to extract video segments containing temporal information as input; The spatiotemporal features of the video are extracted using a 3D convolutional neural network to generate a feature map that simultaneously encodes the spatial appearance and temporal dynamics. An attention pooling mechanism is used to adaptively weight the feature map, highlighting key spatiotemporal regions related to the degree of mouth opening and suppressing irrelevant information. Based on the weighted features, regression or classification is performed through a fully connected layer to output a graded score of 1 to 4, which quantitatively assesses the amplitude or severity of mouth opening movements.

4. The contactless screening system for obstructive sleep apnea syndrome according to claim 1, characterized in that, The dynamic feature extraction device includes: Photoplethysmography signal processing unit: Based on the facial video sequence, extract the signal of the region of interest in the forehead, and reconstruct and synthesize the physiological signal through three-channel separation, phase decoupling, adaptive filtering and signal fusion; Respiratory motion analysis unit: Combining a hybrid method of dense optical flow and deep learning, it calculates the optical flow field of the nasal alar region of the identified object; reconstructs the breathing waveform of the identified object; and detects abnormal breathing based on the local mutation index; Eye movement sleep detection unit: Extracts the region of interest sequence of the eyes based on the facial video sequence, captures the spatiotemporal dynamic information of eye movements through a 3D convolutional neural network, and focuses on key time points and feature channels through a multi-head attention mechanism, outputting the probability of rapid eye movement based on attention-weighted features; Multi-signal synchronization unit: Employs dynamic time warping to align physiological signals, enabling the extraction and output of dynamic features.

5. The contactless screening system for obstructive sleep apnea syndrome according to claim 1, characterized in that, The three-channel separation specifically refers to: Based on the signal from the aforementioned region of interest in the forehead, hemoglobin absorption features are separated in the HSV color space, wherein: Specifically, ΔH represents the hue value or the angle value associated with the hue. In color spaces, hues are typically represented in angular form from 0° to 360°. arctan: The arctangent function, used to convert calculation results into radian angle values; R, G, B: These represent the values ​​of the red, green, and blue color channels, respectively, and are usually in the range of 0 to 255 or 0 to 1. (GB) and (2R-GB): are calculated based on the difference between RGB channels, representing the relative relationship between color channels.

6. The contactless screening system for obstructive sleep apnea syndrome according to claim 1, characterized in that, The fusion analysis device includes a spatiotemporal feature extraction unit, a temporal modeling unit, and an adaptive feature fusion unit.

7. The contactless screening system for obstructive sleep apnea syndrome according to claim 6, characterized in that, The spatiotemporal feature extraction unit is used to perform the following steps: A 3D residual convolutional neural network with a depth of 34 layers is used as the backbone network. The preprocessed facial video sequence is used as input to obtain the residual blocks contained in the backbone network structure. For each residual block, a 3D convolution kernel is used to extract spatiotemporal features: in, 3D convolution kernel, For the input cube, BN represents batch normalization; Each residual block outputs a 256-dimensional feature map, which is then processed by 3D global average pooling to obtain a 512-dimensional spatiotemporal feature vector.

8. The contactless screening system for obstructive sleep apnea syndrome according to claim 6, characterized in that, The timing modeling unit is used to perform the following steps: A bidirectional long short-term memory network is used to process physiological signal time series, wherein the physiological signals include at least heart rate, respiration, and blood oxygenation. In a time series of length T, the input sequence is defined as follows: each This represents the multi-channel physiological signal vector collected at time point T; the unit formula for updating the bidirectional long short-term memory network is based on the input sequence.

9. The contactless screening system for obstructive sleep apnea syndrome according to claim 7, characterized in that, The formula for updating the unit of the bidirectional long short-term memory network based on the input sequence includes: in, The output of the forget gate is in the range [0,1] and is used to determine the information to be discarded from the cell state; It is the sigmoid activation function; Here is the weight matrix for the forget gate; To concatenate the hidden state from the previous time step with the current input; For the bias term of the forget gate; in, The input gate output is in the range [0,1] and is used to determine the new information added to the cell state; It is the sigmoid activation function; This is the weight matrix of the input gate; To concatenate the hidden state from the previous time step with the current input; This is the bias term for the input gate; in, This is the output gate, with a range between [0,1], used to determine the output value; It is the sigmoid activation function; This is the weight matrix of the output gate; To concatenate the hidden state from the previous time step with the current input; This is the bias term for the output gate; in, It represents the current cell state, used to update the cell state and preserve long-term memory; ⊙ represents element-wise multiplication; The cell state at the previous time step; tanh is the hyperbolic tangent activation function, used to generate new candidate values; This is the weight matrix for cell states; To concatenate the hidden state from the previous time step with the current input; This is a bias term for the cell state; in, The hidden state at the current moment; used to generate the output at the current moment, containing short-term memory.

10. The contactless screening system for obstructive sleep apnea syndrome according to claim 6, characterized in that, The adaptive feature fusion unit is used to perform the following steps: Define a feature importance weight calculation module for static anatomical features. Dynamic physiological characteristics and video spatiotemporal features Perform weighted fusion: in, is the attention weight of the i-th feature, used to measure the importance of this feature during fusion, with a value range of [0,1], and the sum of all weights is 1; and These are learnable weight vectors and bias terms used to adjust features. Perform a linear transformation and score it. For the i-th type of input feature, such as text features or image features; exp() is an exponential function used to amplify the fractional differences between features. The summation of the exponential scores of all features s, d, and v is used as a normalization factor. This is the final feature vector after fusion; To weight each feature according to its attention weight; To sum all the weighted features, we obtain the fused features.

Images