A deep learning-based event camera pulse wave detection method

Abstract

The present application belongs to the field of physiological signal processing and artificial intelligence technology, and specifically relates to an event camera pulse wave detection method based on deep learning, aiming to realize reliable and high-precision non-contact pulse wave detection in complex real scenes. The method comprises collecting event stream data of the radial artery region of the wrist of a subject by using an event camera; wherein the event stream data refers to a collection of event signals containing pixel coordinates, time stamps and polarity information of events; performing a preprocessing operation on the event stream data to obtain a preprocessed count signal; the preprocessing includes region of interest extraction, time binning, detrending and standardization; inputting the preprocessed count signal into a deep learning network to recover the pulse wave signal in an end-to-end manner; using a photoplethysmography signal to supervise the training of the deep learning network, and calculating physiological indicators based on the recovered pulse wave signal.

Description

Technical Field

[0001] This invention relates to the fields of physiological signal processing and artificial intelligence technology, and in particular to a deep learning-based method for detecting pulse waves using an event camera. Background Technology

[0002] The pulse wave is a key physiological signal reflecting the state of the human cardiovascular system. Its waveform characteristics are closely related to multiple health indicators such as heart rate, heart rate variability, blood pressure, and vascular elasticity. Therefore, high-precision pulse wave detection technology has wide-ranging applications in health monitoring, disease early warning, and assisted diagnosis.

[0003] Currently, mainstream pulse wave detection methods are mainly divided into two categories: contact and non-contact. Contact methods, represented by photoplethysmography, acquire signals through skin-contact sensors. While offering high accuracy, the constraint of wearing these sensors affects user experience and the applicability for long-term monitoring. Non-contact methods, primarily based on remote photoplethysmography, utilize ordinary cameras to capture subtle changes in light intensity on the skin surface. However, traditional cameras, with their frame-based imaging mechanisms, suffer from inherent limitations such as limited temporal resolution, low dynamic range, and high data redundancy. This makes it difficult to stably extract microsecond-level pulse optical information under interference from ambient light changes and minute target movements, resulting in insufficient signal quality and robustness.

[0004] Event cameras, as an emerging bio-inspired visual sensor, output brightness change information in the form of an asynchronous "event" stream. They possess unique advantages such as microsecond-level temporal resolution, high dynamic range, and low data redundancy, making them theoretically well-suited for capturing instantaneous optical pulsations caused by the pulse. However, current technologies have not fully utilized these characteristics of event cameras to achieve direct reconstruction of high-fidelity pulse waveforms, lacking mature solutions for effectively recovering continuous and smooth physiological waveforms from sparse, asynchronous event streams. Summary of the Invention

[0005] The purpose of this invention is to provide a deep learning-based event camera pulse wave detection method, which aims to achieve reliable and high-precision non-contact pulse wave detection in complex real-world scenarios.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: This invention provides a deep learning-based event camera pulse wave detection method, comprising: S1: acquiring event stream data of the radial artery region of a subject's wrist using an event camera; wherein, the event stream data refers to a set of event signals containing the pixel coordinates, timestamps, and polarity information of the event; S2: performing preprocessing operations on the event stream data to obtain a preprocessed count signal; the preprocessing includes region of interest extraction, time binning, detrending, and standardization, wherein the region of interest refers to the selected effective data region containing the radial artery region; S3: inputting the preprocessed count signal into a deep learning network to recover the pulse wave signal in an end-to-end manner; wherein, the deep learning network refers to a neural network system used to realize the mapping from the count signal to the pulse wave signal; S4: supervising the training of the deep learning network using photoplethysmography (PPG) signals, and calculating physiological indicators based on the recovered pulse wave signal; wherein, the PPG signals refer to physiological signals that provide a reference for the real pulse wave, and the physiological indicators include heart rate and heart rate variability.

[0007] In step S1, a laser light source is used to illuminate the radial artery region; a time signal of the radial artery region of the subject's wrist is acquired using an event camera. in Indicates the recorded event signal, For pixel coordinates, For timestamps, It is polar.

[0008] In step S2, the region of interest is determined based on the event density map.

[0009] In step S2, time binning refers to dividing the event stream data within the region of interest into fixed time windows. The data is divided and transformed into a continuous event counting signal. : in, for Event count signal within the time window, For impulse functions, Indicates the first The timestamps of each event and the event count signal are time-series signals obtained by counting the number of events that occur within each time window.

[0010] In step S3, the deep learning network includes an event feature extraction module. The event feature extraction module adopts a structure that combines a three-dimensional convolutional neural network with a sparse attention mechanism. The event feature extraction module is used to extract multi-scale spatiotemporal features from the input counting signal.

[0011] The deep learning network also includes a temporal modeling and fusion module, which adopts a structure that combines an improved temporal converter structure with a gated recurrent unit. The improved temporal converter structure refers to a structure that includes a multi-head temporal attention layer, a positional encoding layer, and a residual normalization layer. The temporal modeling and fusion module is used to model the temporal dependencies of multi-scale spatiotemporal features and output temporal features.

[0012] The deep learning network also includes a pulse waveform reconstruction module, which includes deconvolutional layers and multilayer perceptron output layers. The pulse waveform reconstruction module is used to decode temporal features into continuous pulse wave signals, and a residual regression structure and time smoothing constraints are introduced in the pulse waveform reconstruction module to ensure the continuity and physiological rationality of the pulse waveform.

[0013] In step S4, supervised training includes aligning the photoplethysmography signal with the predicted pulse wave signal and adjusting the network parameters using a correlation index as a loss function.

[0014] In step S4, the heart rate is calculated by analyzing the frequency characteristics of the pulse wave signal, and the heart rate variability is calculated by analyzing the time interval between adjacent waveform peaks in the pulse wave signal.

[0015] The performance evaluation step is included after step S4: the accuracy is verified by comparing the waveform similarity between the recovered pulse wave signal and the photoplethysmography signal, the effectiveness is verified by comparing the consistency of physiological indicators, and the adaptability is verified by comparing the detection results under different acquisition conditions.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This application provides a deep learning-based event camera pulse wave detection method. It uses an event camera as the signal acquisition terminal, leveraging its non-frame sparse event stream output characteristics to fully utilize its core advantages of high temporal resolution and low redundancy. Combined with directional illumination of the radial artery region using a laser light source, it effectively enhances the optical change signal caused by pulse pulsation. Furthermore, targeted preprocessing operations such as region of interest extraction, bandpass filtering, and standardization filter background noise and invalid data at the source. By combining deep learning networks for accurate modeling of event signals, this method can stably output high signal-to-noise ratio and high-fidelity pulse wave signals even under complex lighting and environmental fluctuations, completely overcoming the environmental adaptability limitations of existing non-contact detection technologies.

[0017] 2. The deep learning network of this invention achieves precise mapping from event counting signals to pulse wave signals through modular collaborative design. The event feature extraction module adopts a structure combining a three-dimensional convolutional neural network and a sparse attention mechanism, which can capture the local dynamic spatial features of the counting signal and model the global temporal dependency, efficiently extracting multi-scale spatiotemporal features. The temporal modeling and fusion module, through the synergistic effect of an improved temporal converter and a gated recurrent unit, strengthens the capture of the periodicity and local temporal dependency of the pulse wave on the one hand, and fully explores the correlation information of the event signal in the temporal dimension by temporal fusion and updating of different event features on the other hand, improving the completeness of the temporal feature expression in terms of temporal continuity and global consistency. The pulse waveform reconstruction module, through the combination of deconvolutional layers, multilayer perceptrons and residual regression structures, and temporal smoothing constraints, effectively avoids non-physiological anomalies such as waveform jumps and oscillations, ensuring the continuity and physiological rationality of the output pulse wave in the time domain. The end-to-end technical design ensures that the recovered pulse wave signal is highly consistent with the real signal in terms of core features such as morphology and frequency, providing a reliable foundation for subsequent physiological index calculations.

[0018] 3. This invention uses photoplethysmography (PPG) signals as a supervised reference. Through the design of signal alignment and correlation index loss functions, the deep learning network is precisely trained to ensure the consistency between the predicted pulse wave signal and the real pulse wave signal. Based on this, heart rate is calculated by analyzing the frequency characteristics of the pulse wave signal, and heart rate variability is calculated by extracting waveform peaks and analyzing the time intervals between adjacent peaks. The calculation logic of these two core physiological indicators is highly matched with the physiological characteristics of the pulse wave signal, resulting in accurate and stable calculation results. Compared to the problem of large errors in physiological indicators due to signal distortion in existing technologies, the heart rate and heart rate variability output by this method can truly reflect the state of the subject's cardiovascular system. This provides reliable data support for health management scenarios such as heart rate monitoring, arteriosclerosis assessment, and blood pressure estimation, as well as for the auxiliary diagnosis of related diseases, enhancing the clinical reference value of non-contact health monitoring. Attached Figure Description

[0019] Figure 1 This is a flowchart of an event camera pulse wave detection method based on deep learning provided in an embodiment of this application; Figure 2 This is a schematic diagram showing the comparison between the counting signal, the PPG signal, and the prediction result provided in an embodiment of this application; Figure 3 This is a schematic diagram showing the comparison results of the time-frequency representation of the prediction signal and the PPG signal provided in an embodiment of this application; Figure 4 This is a spatiotemporal visualization diagram of event stream data provided in an embodiment of this application. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0021] For example, such as Figure 1 As shown in the figure, this application provides a deep learning-based method for detecting pulse waves in an event camera, including: S1: Event flow data of the radial artery region of the subject's wrist was acquired using an event camera.

[0022] Event stream data refers to a collection of event signals that include the pixel coordinates, timestamps, and polarity information of the events.

[0023] For example, in step S1, a laser light source is used to illuminate the radial artery region to enhance the optical changes caused by pulse pulsation; a time signal of the radial artery region of the subject's wrist is acquired using an event camera. in Indicates the recorded event signal, For pixel coordinates, For timestamps, It is polar.

[0024] S2: Perform preprocessing operations on the event stream data to obtain the preprocessed count signal; preprocessing includes region of interest extraction, time binning, detrending and standardization, and the region of interest refers to the effective data region containing the radial artery region selected.

[0025] In step S2, the region of interest is determined based on the event density map. The event density map is an image formed by statistically analyzing the density of the spatial distribution of events. It is used to accurately locate the region where the radial artery is located, and to remove redundancy in order to improve detection accuracy and speed up processing.

[0026] In step S2, time binning refers to dividing the event stream data within the region of interest into fixed time windows. The process of partitioning transforms an irregular event stream into a continuous time series, which in turn transforms it into a continuous event counting signal. : in, for Event count signal within the time window, For impulse functions, Indicates the first The timestamps of each event and the event count signal are time-series signals obtained by counting the number of events that occur within each time window.

[0027] After time binning, the resulting time series is detrended to eliminate the influence of background illumination variations. The specific formula is as follows: in, This represents the event count signal after detrending processing. This represents the total number of sampling points within the time window.

[0028] Finally, the preprocessing steps are completed by standardizing the detrended event count signal. in, This represents the event count signal after detrending and standardization, i.e., the count signal. This represents the mean. It represents the standard deviation.

[0029] S3: Input the preprocessed counting signal into the deep learning network to recover the pulse wave signal in an end-to-end manner.

[0030] Deep learning networks refer to neural network systems used to map count signals to pulse wave signals, enabling end-to-end reconstruction from event camera-acquired, preprocessed event count signals to pulse waveforms. For example, a deep learning network includes three main functional modules: an event feature extraction module, a temporal modeling and fusion module, and a pulse waveform reconstruction module. These three modules are connected sequentially to form a complete feature extraction and signal reconstruction path.

[0031] The event feature extraction module is used to extract multi-scale spatiotemporal features from the input counting signal. As a possible implementation, given the sparsity and non-frame-based nature of the event data output by the event camera, to fully capture the event's changes in both time and space, the event feature extraction module employs a structure combining a 3D convolutional neural network (3D-CNN) and a sparse attention mechanism. The 3D convolutional part can extract dynamic changes between consecutive time frames within a local range, while the sparse attention mechanism can capture global dependencies over long time spans, thereby achieving high temporal resolution dynamic modeling of pulse waveforms. The calculation process is as follows: in, Indicates at time Extracted high-dimensional spatiotemporal feature mapping, This represents a 3D convolution operation. It represents a sparse attention mechanism; it is used to model the correlation between events in different time periods; through the event feature extraction module, the deep learning network can robustly model the optical pulsation changes in the event stream and effectively suppress the interference of illumination fluctuations and background noise.

[0032] The temporal modeling and fusion module is used to model the temporal dependencies of multi-scale spatiotemporal features and establish the dynamic evolution of event sequences. As a possible implementation, considering the obvious periodicity and local dynamic dependencies of pulse wave signals, the temporal modeling and fusion module adopts a structure combining an improved time-series converter structure and a gated recurrent unit. The improved time-series converter structure includes a multi-head temporal attention layer, a position encoding layer, and a residual normalization layer, enabling it to capture complex pulse wave rhythm changes over a longer time range. The gated recurrent unit further enhances the modeling capability for local temporal dependencies. (Based on time...) The high-dimensional spatiotemporal feature mapping, and the temporal fusion hidden state update process are represented as: in, This represents the hidden state representation after temporal fusion. Indicates a gated loop unit. This represents an improved time-series converter structure used to characterize the physiological dynamics at the current moment. The improved time-series converter structure is used to model global dependencies, while the gated recurrent unit is used to capture short-term dynamic changes. The two work together to enable the deep learning network to maintain long-term temporal correlations while being sensitive to local morphological changes.

[0033] The pulse waveform reconstruction module decodes temporal features into a continuous pulse wave signal. As one possible implementation, the module includes a deconvolution layer and a multilayer perceptron output layer. Furthermore, it incorporates a residual regression structure and temporal smoothing constraints to ensure the continuity and physiological plausibility of the pulse waveform, preventing waveform jumps or oscillations during decoding. The output calculation formula for the pulse waveform reconstruction module is as follows: in, Indicates the predicted pulse wave signal. This represents a multilayer perceptron. Indicates a deconvolution layer. To smooth the weights of regularization terms, A time second derivative constraint term is used to predict the pulse wave signal, constraining the smoothness of the waveform between adjacent time points. This constraint effectively avoids abnormal fluctuations in the model caused by noise, ensuring the morphological consistency between the output waveform and the physiological signal waveform of the real pulse wave reference.

[0034] S4: Supervised training of the deep learning network is performed using photoplethysmography (PPG) signals, and physiological indicators are calculated based on the recovered pulse wave signals; where PPG signals refer to physiological signals that provide a real pulse wave reference, and physiological indicators include heart rate and heart rate variability.

[0035] In step S4, supervised training includes aligning the photoplethysmography (PPG) signal with the predicted pulse wave signal and adjusting network parameters using a correlation index as the loss function. Network parameters refer to: in the event feature extraction module, the kernel weights and bias parameters of the 3D convolutional neural network, and the attention weight parameters in the sparse attention mechanism; in the temporal modeling and fusion module, the weights, position encoding parameters, residual normalization layer parameters of the improved temporal converter structure, and the state transition weights and bias parameters of the gated recurrent unit; in the pulse waveform reconstruction module, the kernel weights and bias parameters of the deconvolution layer, and the fully connected weights and bias parameters of the multilayer perceptron output layer; and the regularization correlation parameters in the residual regression structure and temporal smoothing constraints. These parameters are iteratively adjusted through backpropagation of the gradient of the loss function (correlation index), ultimately achieving consistency optimization between the predicted pulse wave signal and the PPG signal, ensuring the accuracy of pulse wave detection.

[0036] For example, the negative Pearson correlation coefficient is used as the loss function. : in, The average physiological signal value is used as a reference for the true pulse wave. To predict the mean value of the pulse wave signal.

[0037] In step S4, the heart rate is calculated using a fast Fourier transform by analyzing the frequency characteristics of the pulse wave signal. : in, Represents the Fast Fourier Transform. Indicates in Within the frequency range, find the frequency corresponding to the maximum amplitude of the Fourier transform result. This frequency is the dominant frequency corresponding to the heart rate, and the final heart rate is calculated from the dominant frequency.

[0038] Heart rate variability (HRV) is calculated by analyzing the time interval (RR interval) between adjacent peaks in the pulse wave signal. The R-wave peak is determined by threshold detection of the recovered pulse wave signal (the threshold is 80%-90% of the signal peak value). The interval between adjacent heartbeats. The average value across all intervals. This represents the number of RR intervals.

[0039] For example, after step S4, a performance evaluation step is also included: verifying accuracy by comparing the waveform similarity between the recovered pulse wave signal and the photoplethysmography signal, verifying effectiveness by comparing the consistency of physiological indicators, and verifying adaptability by comparing the detection results under different acquisition conditions.

[0040] Reference Figure 2 By comparing the recovered pulse wave signal with the physiological signal of the real pulse wave reference in terms of waveform similarity and frequency characteristics, the accuracy and stability of the method provided in the embodiments of this application can be systematically analyzed. Figure 2 It can be seen that the PPG signal and the predicted result, i.e. the recovered pulse wave signal, are extremely similar in waveform, which shows that the method provided in the embodiments of this application is very reliable in terms of accuracy.

[0041] Reference Figure 3 By comparing the power spectra of the recovered pulse wave signal with the physiological signal referenced by the real pulse wave in the time-frequency domain, the performance of the method provided in the embodiments of this application can be further verified. Figure 3 It can be seen that the recovered pulse wave signal is highly consistent with the real PPG signal in terms of the position of the dominant frequency marked by the dashed line, the dynamic trend of frequency change (the small fluctuation characteristics of the curve over time), and the energy distribution pattern (the spatial distribution and intensity gradient of the high-energy region). This indicates that the method provided in this application embodiment can accurately capture the main frequency components of the pulse wave and their dynamic change characteristics over time. This result verifies the reliability of the method at the frequency feature level and further confirms the accuracy and stability of the method.

[0042] Reference Figure 4 By demonstrating the spatiotemporal distribution of the event stream output by the event camera and the processing results in the time dimension, the preprocessing logic of the event stream can be clearly demonstrated. Figure 4 As can be seen, the left figure shows the joint distribution of events in the spatial and temporal dimensions. The dense strip structure of events along the time axis intuitively reflects the temporal aggregation characteristics of events under time binning processing. The right figure shows the aggregation result of the event flow in the spatial dimension. The spatial variation law of the original event flow with time can be clearly observed. This law corresponds to the periodic fluctuation of the skin optical signal caused by the radial artery pulse, indicating that the processing method of this application embodiment can effectively capture the spatiotemporal characteristics of the pulse.

[0043] Experimental results show that with increasing light interference, the signal-to-noise ratio of the traditional rPPG method decreases significantly, while the method provided in this application still maintains high signal fidelity and morphological consistency. Under different subjects and various acquisition conditions, the pulse waves recovered by the method provided in this application exhibit good consistency in key physiological indicators such as heart rate and heart rate variability, fully verifying the robustness and practicality of the method.

[0044] In the description of this specification, specific features, structures, materials, or characteristics may be combined in any suitable manner in one or more embodiments or examples.

[0045] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A deep learning-based method for detecting pulse waves from an event camera, characterized in that, include: S1: Event stream data of the radial artery region of the subject's wrist is acquired using an event camera; whereby event stream data refers to a set of event signals containing pixel coordinates, timestamps, and polarity information of events; S2: Preprocessing is performed on the event stream data to obtain a preprocessed count signal; the preprocessing includes region of interest extraction, time binning, detrending, and standardization, whereby the region of interest refers to the selected effective data region containing the radial artery region; S3: The preprocessed count signal is input into a deep learning network to recover the pulse wave signal in an end-to-end manner; whereby the deep learning network refers to a neural network system used to realize the mapping from the count signal to the pulse wave signal; S4: The deep learning network is supervisedly trained using photoplethysmography (PPG) signals, and physiological indicators are calculated based on the recovered pulse wave signal; whereby the PPG signals refer to physiological signals that provide a real pulse wave reference, and the physiological indicators include heart rate and heart rate variability; in step S3, the deep learning network includes an event feature extraction module, which uses a three-dimensional convolutional neural network and sparse... The deep learning network incorporates an attention mechanism and an event feature extraction module to extract multi-scale spatiotemporal features from the input counting signal. The network also includes a temporal modeling and fusion module, which combines an improved temporal converter structure with a gated recurrent unit. The improved temporal converter structure includes a multi-head temporal attention layer, a position encoding layer, and a residual normalization layer. The temporal modeling and fusion module models the temporal dependencies of the multi-scale spatiotemporal features and outputs temporal features. The deep learning network also includes a pulse waveform reconstruction module, which includes a deconvolution layer and a multilayer perceptron output layer. This module decodes the temporal features into a continuous pulse wave signal, and incorporates a residual regression structure and temporal smoothing constraints to ensure the continuity and physiological rationality of the pulse waveform. In step S4, supervised training includes aligning the photoplethysmography signal with the predicted pulse wave signal and using a correlation index as a loss function to adjust the network parameters.

2. The deep learning-based event camera pulse wave detection method according to claim 1, characterized in that, In step S1, a laser light source is used to illuminate the radial artery region; a time signal of the radial artery region of the subject's wrist is acquired using an event camera. in Indicates the recorded event signal, For pixel coordinates, For timestamps, It is polar.

3. The deep learning-based event camera pulse wave detection method according to claim 1, characterized in that: In step S2, the region of interest is determined based on the event density map.

4. The deep learning-based event camera pulse wave detection method according to claim 1, characterized in that, In step S2, time binning refers to dividing the event stream data within the region of interest into fixed time windows. The data is divided and transformed into a continuous event counting signal. : in, for Event count signal within the time window, For impulse functions, Indicates the first The timestamps of each event, and the event counting signal is a time sequence signal obtained by counting the number of events occurring within each time window.

5. The deep learning-based event camera pulse wave detection method according to claim 1, characterized in that, In step S4, the heart rate is calculated by analyzing the frequency characteristics of the pulse wave signal, and the heart rate variability is calculated by analyzing the time interval between adjacent waveform peaks in the pulse wave signal.

6. The deep learning-based event camera pulse wave detection method according to claim 1, characterized in that, The performance evaluation step is included after step S4: the accuracy is verified by comparing the waveform similarity between the recovered pulse wave signal and the photoplethysmography signal, the effectiveness is verified by comparing the consistency of physiological indicators, and the adaptability is verified by comparing the detection results under different acquisition conditions.

Images