A method and system for estimating video breathing frequency considering the effects of illumination

Abstract

This invention discloses a method and system for estimating video breathing frequency considering the influence of illumination. The method includes: acquiring candidate corner points in the video to be estimated; selecting target corner points based on the color score, texture score, and GFTT quality of each candidate corner point; optimizing the target corner points by combining inter-frame geometric constraint error and census matching error to obtain the optimal inter-frame corner point matching result; and using adaptive robust Kalman filtering for dynamic weighted fusion based on the optimal inter-frame corner point matching result to obtain the final estimated breathing signal. This invention addresses the challenge of signal reliability degradation under extreme illumination changes in video-based breathing estimation, and promotes the development of illumination-robust video breathing frequency estimation methods.

Description

Technical Field

[0001] This invention relates to the field of medical respiratory monitoring technology, and in particular to a video respiratory rate estimation method and system that takes into account the influence of light. Background Technology

[0002] Breathing is a key indicator of physiological health and is widely used in scenarios such as driver monitoring, lie detection, and long-term health assessment. Non-contact respiratory rate (RR) estimation can be achieved through various sensing methods, including radar, WiFi, and RGB cameras. Among these, RGB cameras have attracted particular attention due to their widespread deployment and enormous application potential. RGB video-based respiratory rate estimation faces two major challenges: motion artifacts and ambient lighting variations. The former can be viewed as a decoupling problem between the breathing component and the noise component, and signal processing and deep learning methods have extensively studied this area. The latter affects the reliability of breathing-related features (such as corner trajectories and color changes) captured by existing methods, thus significantly reducing performance. Despite the importance of this issue, related research is limited, and effective solutions are lacking.

[0003] Video-based respiratory rate estimation methods can be categorized into motion-based, remote photoplethysmography (rPPG)-based, and deep learning-based methods. Motion-based methods extract respiratory displacement signals from pixel motion in the abdominal or shoulder region through optical flow or pose estimation. The pyramid-based LK optical flow method, combining multiple corner detectors, has become the mainstream approach for extracting respiratory signals. Tan et al. and Cheng et al. demonstrated its application in intensive care units and motion-based scenarios. When significant shadows appear within the region of interest (ROI), existing corner selection strategies cannot distinguish between real corners and shadow-related pseudo-corners, leading to signal sampling failure. Furthermore, the LK optical flow method, which relies on the assumption of constant illumination, suffers from corner drift and tracking loss under varying illumination. Alnaggar et al. extracted respiratory signals from the shoulder-nose distance using pyramid-based motion amplification and pose estimation. While this method reduces dependence on illumination, the pyramid process simultaneously amplifies illumination fluctuations, potentially leading to face detection failures. Due to the diversity of head movements, its applicability in real-world environments remains limited. Methods based on rPPG and deep learning capture breathing by analyzing subtle color changes in facial skin. The former relies on optical decomposition models, while the latter employs data-driven fitting. However, in low-light and complex lighting conditions, such as nighttime driving, rPPG-based methods fail to extract stable skin features, while deep learning methods lack generalization ability due to insufficient scene diversity. Furthermore, the lack of public datasets with ground-proof breathing values ​​in extreme lighting scenarios further hinders the development of lighting-robust video breathing frequency estimation methods.

[0004] Currently, only a few studies address the challenges of illumination variation in video breathing frequency estimation, and most focus on low-light conditions while neglecting dynamic illumination changes. Wang et al. applied a Retinex-based enhancement method for video breathing frequency estimation under low-light conditions, but its effectiveness is not applicable to scenarios with varying illumination. Shao et al. developed a self-supervised spatiotemporal filtering framework for acquiring rPPG signals in outdoor environments and inferring breathing from them. However, this work has not been publicly published and does not include evaluations under extreme illumination conditions. In real-world scenarios, rapid and extreme illumination fluctuations are common. Nighttime driver monitoring is a typical example, where various light sources, such as ambient light inside the vehicle, streetlights, and headlights from other vehicles, introduce significant and unstructured illumination noise under low illumination conditions. The robustness of enhancement algorithms under extreme illumination changes is crucial for advancing video breathing frequency estimation towards practical applications. Summary of the Invention

[0005] To address the aforementioned shortcomings in the existing technology, the present invention provides a video breathing frequency estimation method and system that considers the influence of illumination, solving the problems of low signal reliability and insufficient breathing estimation accuracy caused by extreme illumination noise.

[0006] To achieve the above-mentioned objective, this invention provides a video breathing frequency estimation method that considers the influence of illumination, comprising: Obtain candidate corner points in the video to be estimated; The target corner is selected based on the color score, texture score, and GFTT quality of each candidate corner. The target corner point is optimized by combining the inter-frame geometric constraint error and the census matching error to obtain the optimal inter-frame corner point matching result; Based on the optimal inter-frame corner matching results, an adaptive robust Kalman filter is used for dynamic weighted fusion to obtain the final estimated respiratory signal.

[0007] Secondly, the present invention also provides a system for implementing a video breathing frequency estimation method that takes into account the influence of illumination, comprising: The data processing module is used to obtain candidate corner points in the video to be estimated; The color-texture constrained corner optimization module is used to select target corners based on the color score, texture score, and GFTT quality of each candidate corner. The geometric constraint survey transformation module is used to optimize the target corner points by jointly minimizing the geometric consistency error and the survey matching error, so as to obtain the optimal inter-frame corner point matching result; The multi-channel adaptive Kalman filter module is used to dynamically weight and fuse the signals based on the optimal inter-frame corner matching results using adaptive robust Kalman filtering to obtain the final estimated respiratory signal.

[0008] The present invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs a video breathing frequency estimation method that takes into account the influence of illumination.

[0009] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform a video breathing frequency estimation method that takes into account the effects of illumination.

[0010] The beneficial effects of this invention are as follows: 1. A task-oriented corner selection strategy specifically tailored for breathing extraction was established through the color-texture constraint corner optimization module. By applying color difference and texture variation constraints, corners located near clothing edges or skin-clothing boundaries were selected. These corners exhibited good traceability and strong correlation with breathing displacement, resulting in a more reliable set of breathing-related corners.

[0011] 2. By introducing geometric constraints through the geometric constraint survey transformation module, and utilizing the inherent stability of the distance, angle and local geometry between corner points, the matching problem is improved from independent point-by-point decision-making to joint optimization that maintains the structure. The combination of CT illumination robustness and global geometric consistency effectively suppresses drift and point jump caused by illumination perturbation, so that the corner point trajectory can maintain the real breathing-driven deformation pattern over long sequences.

[0012] 3. By employing a process noise standard deviation update strategy, the Kalman filter model maintains low process noise and improves smoothness when breathing is stable, while increasing process noise to enhance response capability when illumination interference intensifies or tracking temporarily fails. Through this joint adaptive covariance update, the Kalman filter can simultaneously distinguish channel quality and adapt to time-varying noise conditions, ensuring that the fused output breathing signal has high stability, strong robustness, and effective self-recovery capability. Attached Figure Description

[0013] Figure 1 A flowchart of a video breathing frequency estimation method considering the influence of illumination is provided as an example. Figure 2 A comparison image showing the corner points selected by the corner point selection method in this invention and the corner points selected by a traditional detector; Figure 3 This is a performance comparison between the average fusion method and the self-optimizing fusion method (M-ARKF) proposed in this invention; Figure 4 This is a comparison chart between the respiratory signals predicted by the method provided in this invention and the actual signals, based on the SCSO-Driver (nighttime) dataset. Figure 5 This is a comparison chart between the respiratory signals predicted by the method provided in this invention and the actual signals, based on the SCSO-Driver (daytime) dataset. Figure 6 This is a comparison chart between the respiratory signals predicted by the method provided in this invention and the actual signals, based on the COHFACE dataset. Figure 7 This is a comparison diagram between the respiratory signals predicted by the method provided in this invention and the actual signals, based on the COHFACE dataset simulating changes in light intensity. Detailed Implementation

[0014] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0015] like Figure 1 As shown, in one embodiment of the present invention, a video breathing frequency estimation method considering the influence of illumination includes the following steps: S1. Obtain candidate corner points in the video to be estimated.

[0016] Specifically, acquire the human movement and breathing video to be estimated; Based on the pose estimation network HRNet, a region of interest (ROI) is selected from the first frame of the video to be estimated. The neck-collar region is selected as the ROI from the first frame of the video. Breathing-related ROIs typically include the shoulder-neck, abdomen, and chest regions. The shoulder region is highly susceptible to motion interference, the abdomen is often occluded, and the chest region suffers from variations in corner quality due to differences in clothing texture among different subjects. Therefore, this embodiment selects the neck-collar region as a more stable ROI, as this region naturally contains multiple corners at the clothing-skin boundary, and its geometry provides additional constraints for subsequent corner prediction.

[0017] The GFTT algorithm is applied to the region of interest to obtain several candidate corner points, and each candidate corner point corresponds to a GFTT quality score.

[0018] S2. Select the target corner point based on the color score, texture score and GFTT quality of each candidate corner point.

[0019] Specifically, including: For each candidate corner point, a 25×25 pixel window is defined, and the local features of each candidate corner point are captured with the candidate corner point as the center to obtain a local image patch; Calculate the gradient covariance matrix of a local image patch. Its expression is:

[0020] In the formula, ▽ represents a local image patch, and ▽ represents the gradient operator. This represents the gradient of a local image patch in the x-direction. This represents the gradient of a local image patch in the y-direction.

[0021] based on Determine the principal orientation of a local image patch, and derive the surface normal vector of the local image patch based on the principal orientation. ; Surface normal vectors based on local image patches The local image patch is divided into the first sub-region. Second sub-region ; To calculate the color features of candidate corner points, local image patches are transformed to the illumination-robust CIELAB color space, and color scores for candidate corner points are calculated, including color stability score and color discrimination score, with the following expressions:

[0022] In the formula, Score for color stability. The color discrimination score characterizes the color consistency of the window around the corner point. This represents the variance of parameter 'a' in the entire CIELAB color space. This represents the variance of the b parameter in the entire CIELAB color space. This represents the mean value of parameter 'a' in the CIELAB color space within the region. This represents the mean value of the b parameter in the CIELAB color space for the region. Significant color discontinuities occur when corner points are located at the boundaries of clothing and skin, while points caused by shadows show less color variation in the color space.

[0023] Texture score quantifies the texture consistency between two sub-regions defined by surface normals. The texture score is calculated through the following steps.

[0024] A gray-level co-occurrence matrix (GLCM) is constructed for the first and second sub-regions, and contrast, energy, entropy and correlation are extracted based on the GLCM. Texture feature vectors are constructed based on contrast, energy, entropy, and correlation. Its expression is:

[0025] In the formula, For contrast, For energy, For entropy, This indicates a correlation.

[0026] Texture scores are calculated based on texture feature vectors, and the expression is as follows:

[0027] In the formula, Score the texture. This is the texture feature vector corresponding to the first sub-region. This is the texture feature vector corresponding to the second sub-region; Based on color score, texture score, and GFTT quality score, an average ranking weighted strategy is used to select the top 10% of corner points from the candidate corner points. The color-texture constraint optimization consistently identifies the correct breathing-related corner points under different lighting conditions, demonstrating its superior corner selection capability.

[0028] Breathing estimation relies on capturing subtle chest wall movements, making the quality of selected corner points crucial for achieving high signal-to-noise ratios and robustness. However, mainstream corner point or feature detectors exhibit significant limitations in this scenario. General-purpose feature extractors (such as SIFT, KAZE, and ORB) are designed for cross-image invariance rather than breathing tracking, while traditional GFTT detection relies solely on grayscale gradients, often selecting pseudo-corners from uniform or shaded regions that do not reflect breathing motion but instead introduce noise. This invention addresses these issues by establishing a task-oriented corner point selection strategy specifically tailored for breathing extraction. By jointly applying constraints based on color difference and texture variation, corner points located near clothing edges or skin-clothing boundaries are selected, exhibiting good trackability and strong correlation with breathing displacement. Figure 2 A comparison of the corner selection method proposed in this invention with the corner selection method of traditional detectors shows that the present invention produces a more reliable set of respiration-related corners.

[0029] S3. Optimize the target corner point by combining the inter-frame geometric constraint error and the survey matching error to obtain the optimal inter-frame corner point matching result.

[0030] Census Transform (CT) is a non-parametric method for representing local structure in images. While CT is inherently robust to both linear and non-linear illumination changes due to its non-parametric local intensity comparison mechanism, this characteristic also introduces inherent limitations when applied to breathing tracking. CT only encodes the relative intensity order within a small neighborhood, meaning its discriminative ability depends entirely on the stability of local texture patterns. However, in real-world driving scenarios, the textures around collars, neck skin, and shadow boundaries are often simple, repetitive, or affected by dynamic shadows, leading to frequent reversals of grayscale order. Therefore, CT-based single-point matching is prone to blurring, point skipping, and being mistakenly attracted to regions with similar texture patterns but unrelated to breathing motion. More importantly, breathing-related corner points often cluster along clothing-skin boundaries or geometrically significant structures of the neck, and their inter-frame motion follows a consistent global pattern. Traditional CT matching processes each corner point independently, making only locally optimal decisions without imposing any global spatial consistency. Therefore, even if some corner points are mismatched due to illumination interference, the system cannot correct this through collective structural cues. This leads to topological collapse, distortion of the overall corner configuration, and ultimately instability in respiratory displacement estimation. Therefore, introducing geometric constraints is crucial. By leveraging the inherent stability of inter-corner distances, angles, and local geometry, the matching problem is elevated from independent point-by-point decisions to joint optimization that preserves the structure. This combines the illumination robustness of CT with global geometric consistency, effectively suppressing drift and point jumps caused by illumination perturbations. As a result, corner trajectories can maintain true respiratory-driven deformation patterns over long sequences, rather than being corrupted by transient local brightness fluctuations.

[0031] Specifically, including: Assuming the target corner point in the first frame is Extract a 5×5 local neighborhood centered on the target corner point; where the coordinates of adjacent pixels are... The corresponding gray intensity is .

[0032] The relative grayscale relationship between each neighboring pixel and the target corner point is encoded to obtain a binary descriptor, the expression of which is:

[0033] In the formula, The final output binary descriptor represents the integer value of a 24-bit binary descriptor obtained by encoding the relative grayscale relationship between the 24 non-center pixels and the center pixel within a 5×5 neighborhood. Let the coordinates be those of a target corner point. For the target corner point The coordinates of adjacent pixels, This indicates the calculation of grayscale intensity values; It is a linear index (ranging from 0 to 23) of each non-center pixel, traversed in the neighborhood in row-major order; This indicates an indicator function that returns 1 if the condition within the parentheses is met, and 0 otherwise.

[0034] The inter-frame corner matching error based on CT is quantified using Hamming distance. The census matching error is calculated based on binary descriptors, and its expression is as follows:

[0035] in, Indicates census matching error. This represents a counting function used to count the number of bits that are 1 in the XOR result, which is the final Hamming distance error. This represents the binary descriptor corresponding to a target corner point within the search window of the t-th frame in the video to be estimated. This indicates a bitwise XOR operation.

[0036] For the target corner points in the first frame, after being filtered by the color-texture constrained corner point optimization module, a minimum spanning tree (MST) is constructed to characterize the intrinsic spatial geometric relationships between the points. Subsequently, the side length and angle features are used as the geometric constraint benchmarks for inter-frame tracking. The inter-frame geometric constraint error is calculated, and its expression is:

[0037] in, For inter-frame geometric constraint error, This is the side length constraint error sequence. These are the angle constraint error sequences. They represent the deviations of the predicted corner point sequence in the current frame from the side lengths and angles of the reference MST structure, respectively. Accordingly, the corner point prediction for each frame is formulated as a multi-objective weighted minimization problem, aiming to find the optimal combination of candidate points that achieves the highest feature matching consistency while maintaining spatial structure stability.

[0038] The corner matching optimization problem is constructed based on the census matching error and the inter-frame geometric constraint error, and its expression is:

[0039] In the formula, This represents the optimal set of predicted corner points. Indicates the first The set of candidate pixels for each corner point within the search window of frame t. . Indicates the cost of feature matching. The geometric constraint cost is expressed as follows:

[0040] In the formula, This represents the census matching error corresponding to the index of the selected candidate pixel within the search window corresponding to the i-th target corner point. This represents the angular constraint error of the k-th edge in the constructed minimum spanning tree. This represents the edge length constraint error of the k-th edge in the constructed minimum spanning tree. This indicates the number of valid edges in the constructed MST.

[0041] Solving the corner matching optimization problem yields the optimal matching pixel position of the target node in each frame, i.e., the optimal inter-frame corner matching result. In this embodiment, a greedy algorithm is used to solve the problem to reduce the computational complexity of the discrete optimization problem.

[0042] S4. Based on the optimal inter-frame corner matching results, an adaptive robust Kalman filter is used for dynamic weighted fusion to obtain the final estimated respiratory signal.

[0043] Specifically, including: Based on the optimal inter-frame corner point matching results, the displacement of each target corner point from the previous frame to the next frame is calculated; the obtained displacements are arranged in chronological order to obtain a time series. The obtained time series is decomposed along the Y-axis and bandpass filtered to suppress non-respiratory components, resulting in a multi-channel respiratory sub-signal. Each sub-signal independently represents a respiratory movement.

[0044] The noise level and residual energy of each channel of the multi-channel respiratory sub-signal are evaluated by adaptive robust Kalman filtering, and adaptive weighted fusion is performed to obtain the final estimated respiratory signal.

[0045] Specifically, the self-optimization fusion of multi-source respiratory signals through adaptive robust Kalman filtering is achieved as follows: Given the small inter-frame variation, a quasi-uniform motion model is used to construct the state equation of the Kalman filter, which is defined as:

[0046]

[0047] In the formula, For state vectors, This represents the actual respiratory signal in frame t. The rate of change of the actual respiratory signal in frame t. The process noise is represented by a Gaussian distribution. The process noise covariance matrix is... fps is the inter-frame interval, and FPS is the number of video frames transmitted per second. Let be the state transition matrix.

[0048] The observation equation is:

[0049] In the formula, This represents the multi-source observation vector of frame t. It is the observation matrix, with the first column being 1 and the second column being 0. The observed noise is represented by zero mean and covariance is . The Gaussian distribution.

[0050] The target variable to be estimated in the Kalman filtering process is defined as:

[0051] In the formula, This is the updated optimal estimated state. The predicted value is obtained based on the state of the previous frame. Represents the observation residual. It is the Kalman gain of frame t.

[0052] In the above equation, Since the Kalman gain is an unknown quantity, and other parameters depend on constants, the previous state, and the current observation, estimating the Kalman gain for each frame becomes a crucial step in solving the Kalman filter problem.

[0053] The specific calculation is as follows:

[0054] In the formula, To predict the error covariance, we need to consider the process noise covariance. , To observe the noise covariance matrix, both need to be estimated during the filtering process. This is due to the residuals of each signal channel. Calculateable and All of these are based on adaptive design using the statistical properties of these residuals. (Channels) n of Defined as:

[0055] In the formula, This represents the covariance corresponding to the respiratory signal in the nth channel. The smoothing coefficient is set to 0.9 based on experience. Let be the covariance corresponding to the respiratory signal of the (n-1)th channel. This represents the observation vector of channel n in frame t. This represents the median of the observed values.

[0056] The observation noise intensity is dynamically adjusted by measuring the deviation of each channel observation from the median of all channels. This allows for adaptive weighting between channels based on signal quality. Furthermore, since a smaller residual between the channel measurement signal and the predicted signal indicates higher reliability, and a larger residual indicates lower reliability, the weights of the n channels are defined based on the Lorentz weighting function as follows:

[0057] In the formula, This represents the weight of the nth channel. This represents the residual of the nth channel in the t-th frame. It is the observation noise covariance of the nth channel in the tth frame. A constant to ensure numerical stability. Therefore, the covariance of the nth channel signal in the t-th frame. In the initial stage, the multi-channel filtering process is initialized with equal observation noise variances, which are set to the variance of each channel itself to ensure consistent starting conditions.

[0058] For process covariance The estimation uses a white noise acceleration model, the expression of which is:

[0059] In the formula, The process noise variance represents the acceleration uncertainty. This invention proposes an adaptive process noise update mechanism based on residual energy, defining the process noise standard deviation as:

[0060] In the formula, To control the smoothing coefficient for update stability, it is empirically set to 0.9. The average residual energy at time t reflects the magnitude of the overall prediction error.

[0061] In summary, the self-optimization fusion of multi-source respiratory signals through adaptive robust Kalman filtering includes the following steps: A1. Forming multi-channel observation vectors based on multi-channel respiration sub-signals ; A2. Predict the current observation based on the state of the previous frame. ; A3. Update the covariance matrices of observation noise and process noise; A4. Calculate the dynamic weights (Kalman gain) based on the updated observation noise and updated process noise covariance matrices. ; A5. Based on dynamic weights (Kalman gain) The state is updated to obtain the final respiratory signal output, which is equivalent to calculating the optimal estimated state. .

[0062] In the fusion of multi-source respiratory-related corner point trajectories, the signal exhibits strong dynamic heterogeneity due to local texture differences, varying illumination interference, and random tracking errors. Therefore, noise characteristics not only differ between channels but also change over time, making traditional fixed-parameter filtering models ill-suited to handle this complexity. Simple averaging or static weighting cannot achieve optimal estimation. To address this issue, the proposed adaptive robust Kalman filter models motion as a quasi-uniform linear dynamic system, leveraging the inherent smooth temporal evolution and small inter-frame displacement of breathing. This formula allows the prediction step to effectively capture the continuity and low-frequency characteristics of real respiratory motion. The key advantage of this model lies in its adaptive updating of observation noise covariance and process noise covariance. First, on the observation side, time-varying covariance is constructed by measuring the degree to which each corner point deviates from the multi-channel median. This allows channels with higher noise levels to automatically receive lower weights, while high-quality channels contribute more, achieving adaptive quality competition among channels. Furthermore, Lorentz weighting is applied to suppress extreme biases, ensuring robustness to large transient errors in individual channels. Second, on the prediction side, process noise is dynamically estimated based on residual energy. This allows the model to maintain low process noise and improve smoothness when breathing is stable, while increasing process noise and enhancing responsiveness when illumination interference intensifies or tracking temporarily fails. Through this joint adaptive covariance update, the Kalman filter can simultaneously distinguish channel quality and adapt to time-varying noise conditions, ensuring that the fused output has high stability, strong robustness, and effective self-recovery capability. Figure 3 This paper presents a performance comparison between the averaging fusion method and the self-optimizing fusion method (M-ARKF) proposed in this invention. The results show that averaging fusion is highly sensitive to channels with large deviation errors, leading to amplitude distortion and phase shift in the fused signal. In contrast, even with severe damage to some channels, self-optimizing fusion of multi-source breathing signals via adaptive robust Kalman filtering maintains a stable waveform and achieves optimal fusion.

[0063] To verify the effectiveness and performance of the present invention, the large-scale real-world driver breathing dataset SCSO-Driver, the public COHFACE dataset, and the dataset of simulated light changes derived from COHFACE were collected as experimental data.

[0064] SCSO-Driver Dataset: To comprehensively evaluate the proposed framework under extreme real-world lighting conditions, a large-scale driver breathing dataset was constructed. This dataset covers driving scenarios both day and night, with all videos collected during actual road driving. Light variations are naturally introduced by a variety of light sources, including sunlight, traffic lights, oncoming vehicles, taillights, and in-vehicle ambient light, exhibiting randomness and diversity. The acquisition device used a Logitech C920 Pro camera mounted on the right side of the dashboard to capture the driver's upper body. Videos were recorded at a frequency of 30Hz with a resolution of 640×540 pixels. Ground truth breathing signals were acquired via a wireless piezoelectric breathing band attached to the participant's abdomen, sampled at a frequency of 50Hz. Video and breathing band signals were acquired synchronously and transmitted to a laptop placed on the passenger seat to ensure precise time alignment. The dataset contains data from 10 participants in both day and night scenarios, with each participant provided 100 15-second driving videos for each scenario.

[0065] COHFACE: This dataset contains 160 RGB videos, each 60 seconds long, collected from 40 subjects. Each video captures the upper body and facial area of ​​the subject, recorded using a Logitech HD C525 camera at a resolution of 640×480 pixels. The true respiratory signal was acquired via a breathing belt (SA9311M) at a sampling rate of 256 Hz.

[0066] COHFACE Simulates Illumination Variation: The COHFACE dataset was originally recorded under stable lighting conditions. To evaluate illumination robustness against public benchmarks, a version simulating illumination variation was constructed by introducing controlled noise perturbations. This dataset simulates two common types of illumination variation: global brightness fluctuations and localized irregular moving shadows. Specifically, global brightness variations are achieved by applying sinusoidal modulation to the overall image intensity. Localized irregular shadows are simulated by introducing a movable rectangular shadow region that drifts slowly between frames, its position updated randomly, and then Gaussian blurring is applied to create natural light and shadow transitions. Each video in COHFACE is divided into three illumination perturbation phases: the first 20 seconds contain only global brightness variations, the next 20 seconds contain only localized moving shadow perturbations, and the final 20 seconds contain both effects simultaneously.

[0067] This embodiment uses the three datasets mentioned above as experimental data, employing four deep learning-based methods (such as MTTs-CAN, PhyFormer, Cliffphys, and Spiking-PhyFormer), three signal processing-based methods (such as IOF-NFCP, Alnaggar et al., and OVRM), and the method provided in this invention for respiratory signal estimation. The experimental results are shown in Table 1 (performance comparison of each method on the SCSO-Driver dataset) and Table 2 (performance comparison of each method on the COHFACE dataset and the COHFACE dataset simulating light changes).

[0068] Table 1

[0069] Table 2

[0070] Overall, signal processing-based methods exhibit stronger robustness than network-based methods in video breathing estimation, but all methods suffer significant performance degradation under extreme lighting variations. Alnaggar et al. demonstrated the smallest performance degradation due to the inherent robustness of pose-based measurements. However, their reliance on Euler Video Magnification (EVM) amplifies both motion and lighting changes, leading to overexposure or underexposure and pose detection failures. OVRM achieves a high detection rate but suffers from poor RMSE performance because its block-based adaptive ROI selection cannot guarantee the presence of valid targets, making it sensitive to environmental disturbances and prone to output loss when block features deviate from a preset threshold. Network-based methods experience a sharp performance drop once lighting perturbations are introduced, reflecting their limited cross-domain generalization capabilities. In contrast, the method presented in this invention achieves the lowest MAE and the highest PCC and DR on COHFACE, outperforming all baseline methods. Compared to the best traditional method (IOF-NFCP), it further reduces MAE by approximately 8.3%, demonstrating its superior accuracy and stability. On datasets simulating varying illumination, the advantages of the method provided in this invention are even more pronounced. When existing methods experience a sharp decline in performance, the method provided in this invention maintains significant stability and accuracy, far surpassing all competitors. These results demonstrate that the method provided in this invention effectively suppresses illumination-induced performance degradation, maintaining strong robustness and signal reliability even under challenging illumination conditions where deep learning and optical flow-based methods fail. Figure 4 , Figure 5 , Figure 6 , Figure 7As shown, the comparison results between the predicted respiratory signal generated by the method provided by the present invention and the real respiratory signal are presented, indicating that the waveform generated by the present invention is stable and consistent, and highly consistent with the real respiratory signal band.

[0071] In summary, this invention achieves robust corner selection under shadows, robust inter-frame matching under illumination, and optimal multi-channel signal fusion through a color-texture constrained corner optimization module, a geometric constraint survey transformation module, and a multi-channel adaptive Kalman filter module, respectively, thus solving the challenge of signal reliability degradation under extreme illumination changes in video-based breathing estimation.

Claims

1. A video respiration rate estimation method considering the effect of illumination, characterized in that, include: Obtain candidate corner points in the video to be estimated; The target corner is selected based on the color score, texture score, and GFTT quality of each candidate corner. The target corner point is optimized by combining the inter-frame geometric constraint error and the census matching error to obtain the optimal inter-frame corner point matching result; Based on the optimal inter-frame corner matching results, an adaptive robust Kalman filter is used for dynamic weighted fusion to obtain the final estimated respiratory signal.

2. The method of claim 1, wherein, Obtain candidate corner points from the video to be estimated, including: Obtain the video to be estimated; Based on the pose estimation network HRNet, the region of interest is selected from the first frame of the video to be estimated; The GFTT algorithm is applied to the region of interest to obtain several candidate corner points, and each candidate corner point corresponds to a GFTT quality score.

3. The method of claim 2, wherein, Corner selection is performed based on the color score, texture score, and GFTT quality score of each candidate corner point to obtain the target corner points, including: For each candidate corner point, with size as k × k A pixel window captures the local features of each candidate corner point to obtain a local image patch; Calculate the gradient covariance matrix of a local image patch. ; based on Determine the principal direction of the local image patch, and derive the surface normal vector of the local image patch based on the principal direction of the local image patch; The local image patch is divided into a first sub-region and a second sub-region based on the surface normal vector of the local image patch. The local image patch is transformed to the illumination-robust CIELAB color space, and the color scores of the candidate corner points are calculated, including color stability score and color discrimination score. A gray-level co-occurrence matrix is ​​constructed for the first and second sub-regions, and contrast, energy, entropy, and correlation are extracted based on the gray-level co-occurrence matrix; Texture feature vectors are constructed based on contrast, energy, entropy, and correlation. Calculate texture score based on texture feature vector; Based on color score, texture score and GFTT quality score, the top x corner points are selected from the candidate corner points using an average ranking weighting strategy to obtain the target corner points.

4. The method according to claim 3, characterized in that, The target corner points are optimized by combining inter-frame geometric constraint error and survey matching error to obtain the optimal inter-frame corner point matching result, including: Extracting points centered on the target corner l × l Local neighborhood; The relative grayscale relationship between each neighboring pixel and the target corner point is encoded to obtain a binary descriptor, the expression of which is: In the formula, This is the final output binary descriptor. Let the coordinates be those of a target corner point. For the target corner point The coordinates of adjacent pixels, This indicates the calculation of grayscale intensity values; It is a linear index for each non-center pixel; This indicates an indicator function that returns 1 if the condition within the parentheses is met, and 0 otherwise. The expression for calculating the census matching error based on binary descriptors is as follows: in, Indicates census matching error. Represents a counting function. This represents the binary descriptor corresponding to a target corner point within the search window of the t-th frame in the video to be estimated. Indicates a bitwise XOR operation; By constructing a minimum spanning tree to represent the spatial geometric relationship between target corner points, the inter-frame geometric constraint error is calculated, and its expression is as follows: in, For inter-frame geometric constraint error, This is the side length constraint error sequence. This is the angle constraint error sequence; The corner matching optimization problem is constructed based on the census matching error and the inter-frame geometric constraint error, and its expression is: © In the formula, This represents the optimal set of predicted corner points. Indicates the first The set of candidate pixels for each corner point within the search window of frame t. ; Indicates the cost of feature matching. The geometric constraint cost is expressed as follows: In the formula, This represents the census matching error corresponding to the index of the selected candidate pixel within the search window corresponding to the i-th target corner point. This represents the angular constraint error of the k-th edge in the constructed minimum spanning tree. This represents the edge length constraint error of the k-th edge in the constructed minimum spanning tree. This represents the number of valid edges in the constructed minimum spanning tree; Solve the corner matching optimization problem to obtain the optimal matching pixel position of the target corner in each frame, i.e. the optimal inter-frame corner matching result.

5. The method according to claim 4, characterized in that, Based on the optimal inter-frame corner matching results, an adaptive robust Kalman filter is used for dynamic weighted fusion to obtain the final estimated respiratory signal, including: Based on the optimal inter-frame corner point matching results, the displacement of each target corner point from the previous frame to the next frame is calculated; the obtained displacements are arranged in chronological order to obtain a time series. The obtained time series is decomposed along the Y-axis and bandpass filtered to obtain multi-channel respiratory sub-signals; The noise level and residual energy of each channel of the multi-channel respiratory sub-signal are evaluated by adaptive robust Kalman filtering, and adaptive weighted fusion is performed to obtain the final estimated respiratory signal.

6. The method according to claim 5, characterized in that, The noise level and residual energy of each channel of the multi-channel respiratory sub-signal are evaluated using adaptive robust Kalman filtering, and adaptive weighted fusion is performed to obtain the final estimated respiratory signal, including: A multi-channel observation vector is formed based on the multi-channel respiration sub-signals, and its expression is as follows: ， In the formula, This represents the multi-source observation vector of frame t. It is the observation matrix. For state vectors, Let be a state vector, with zero mean and covariance. Gaussian distribution; This represents the actual respiratory signal in frame t. The rate of change of the actual respiratory signal in frame t; The expression for predicting the observation value of the current frame based on the state of the previous frame is as follows: In the formula, The observation value for the current frame. This refers to the state of the previous frame. This is the state transition matrix; Assess observation noise: Calculate the median of the observation vector; update the observation noise covariance based on the median of the observation vector, expressed as: ； In the formula, Let represent the observation noise covariance corresponding to the respiratory signal of the nth channel in frame t. For smoothing coefficients, Let be the observation noise covariance corresponding to the respiratory signal of the (n-1)th channel in frame t. This represents the observation vector of channel n in frame t. This represents the median of the observed values; The channel weights are calculated and the observation noise is updated; the expression is as follows: In the formula, This represents the weight of the nth channel. This represents the residual of the nth channel in the t-th frame. It is the observation noise covariance of the nth channel in the tth frame. , Constants to ensure numerical stability; Updated observation noise; assessment process noise: The standard deviation of noise during the update process is expressed as: In the formula, The updated process noise standard deviation, To control the smoothing coefficient for update stability, Represents the average residual energy. The standard deviation of process noise before the update; The process noise covariance matrix is ​​updated based on the updated process noise standard deviation, and its expression is as follows: In the formula, The updated process noise covariance; The dynamic weights are calculated based on the updated observation noise and the updated process noise covariance, and their expression is as follows: In the formula, To predict the error covariance, For the updated observation noise, ; The state is updated based on dynamic weights and the observations of the current frame to obtain the final estimated respiratory signal, which is expressed as follows: In the formula, This is the final estimated respiratory signal.

7. A system for implementing the video breathing frequency estimation method considering the effect of illumination as described in any one of claims 1 to 6, characterized in that, include: The data processing module is used to obtain candidate corner points in the video to be estimated; The color-texture constrained corner optimization module is used to select target corners based on the color score, texture score, and GFTT quality of each candidate corner. The geometric constraint survey transformation module is used to optimize the target corner points by jointly minimizing the geometric consistency error and the survey matching error, so as to obtain the optimal inter-frame corner point matching result; The multi-channel adaptive Kalman filter module is used to dynamically weight and fuse the signals based on the optimal inter-frame corner matching results using adaptive robust Kalman filtering to obtain the final estimated respiratory signal.

8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, When the computer program is executed by the processor, it causes the processor to perform the method described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it causes the processor to perform the method described in any one of claims 1 to 6.

Images