Heart rate detection method based on dual-modal cardiac impulse signals

Abstract

This invention provides a heart rate detection method based on dual-modal cardiac impact signals. The heart rate detection method includes: performing quality assessments on waveform signals of contact cardiac impact image segments acquired through a contact acquisition device and remote cardiac impact image segments acquired through video clips; selecting the cardiac impact image segment with the higher quality score from the contact and remote cardiac impact image segments based on the quality assessment results; and inputting the cardiac impact image segment with the higher quality score into a neural network model to obtain an electrocardiogram (ECG) image segment for heart rate extraction. This invention proposes to perform quality assessment and optimal selection of two modalities, cBCG and rBCG signals, to detect heart rate, thereby improving the accuracy of ECG generation.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and more specifically, to a heart rate detection method based on dual-modal cardiac impulse signals. Background Technology

[0002] Cardiac impaction mapping (BCG) is a non-contact, real-time method for detecting cardiac function. It is used to detect body vibration signals synchronized with the heartbeat caused by the heart's pulsation and blood circulation in the aorta. Based on contact cardiac impaction mapping (cBCG) or remote cardiac impaction mapping (rBCG), it can achieve heart rate extraction, heart rate variability, and emotion recognition.

[0003] rBCG signals are generated by cardiac contraction and the ejection of blood from the ventricles into the vascular system. They can be extracted by continuously capturing the subject's face in a video, calculating the BCG waves of the head movement in the previous and next facial images, and tracing the movement of facial points in a two-dimensional trajectory. rBCG is invariant to changes in lighting and skin color, but is limited by motion artifacts, including facial expressions and voluntary head movements.

[0004] cBCG has achieved significant research results in heart rate extraction and heart rate variability, and can be applied to lie detection systems. It can be combined with other non-contact acquisition devices to build a multi-channel lie detection system based on cBCG signals. However, cBCG detection is limited by contact requirements.

[0005] BCG is a dynamic measurement of blood circulation, while medical testing tends to favor ECG (electrocardiographic measurement of cardiac activity), which carries additional cardiac information. Therefore, a non-invasive monitoring solution is needed to convert the BCG information detected by the BCG sensor into ECG information to achieve continuous and reliable cardiac monitoring.

[0006] Reconstructing BCG waveforms from videos recording head movements is a key research focus for heart rate extraction using rBCG. Videos typically require tracking 500–1000 facial feature points for identification and tracking. Solving for such a large number of feature points is time-consuming, and the processing of tracking trajectories for different feature points during signal reconstruction is still in the basic research stage, using only simple operations such as summation and averaging. While current research on heart rate detection based on single-point rBCG signals overcomes the need to track a large number of feature points, increased unreliability under complex lighting conditions leads to poor algorithm robustness.

[0007] The application of deep learning models to generate electrophysiological signals from cardiac dynamics is currently limited to the generation of gECG (Generative ECG) from cardiac electrocardiograms (SCG) using GAN network models, lacking research on ECG generation using BCG. Furthermore, GAN networks suffer from poor interpretability and training instability. Summary of the Invention

[0008] This summary is provided to introduce the selected concepts in a simplified form, and these concepts are further described in the following detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to help determine the scope of the claimed subject matter.

[0009] To address the issue that single-modal BCG information is easily affected by subjective actions or external factors, this invention proposes a method for quality evaluation and optimal selection of both cBCG and rBCG signals to detect heart rate, thereby improving the accuracy of heart rate detection. This invention proposes signal quality assessment of both cBCG and rBCG signals, retaining the higher-quality signal within a given time segment, thus achieving quality evaluation and optimal selection of dual-modal signals.

[0010] To address the issue of poor robustness to facial expressions caused by the fixed selection of ROIs in existing rBCG reconstruction algorithms, this invention proposes an adaptive reconstruction algorithm for rBCG signals to identify and track facial feature points. The quality of the rBCG signal reconstructed using 468 facial feature points is evaluated, and then the facial feature points are sorted, automatically selecting the number and location of each feature point. By establishing a mathematical model of rBCG, the reconstructed rBCG waveform is obtained using principal component analysis based on the three-dimensional motion trajectories of the selected feature points.

[0011] To address the need for BCG to generate ECG, this invention proposes a U-shaped neural network model for BCG to generate ECG, in order to further improve the accuracy of BCG heart rate detection.

[0012] One aspect of the present invention provides a heart rate detection method based on dual-modal cardiac impact signals, comprising: performing a first quality assessment on the waveform signals of a contact cardiac impact image segment acquired by a contact acquisition device and a remote cardiac impact image segment acquired by a video clip, respectively; selecting a cardiac impact image segment with a higher quality score from the contact cardiac impact image segment and the remote cardiac impact image segment based on the result of the first quality assessment; and inputting the cardiac impact image segment with the higher quality score into a neural network model to obtain an electrocardiogram (ECG) image segment.

[0013] Preferably, obtaining the remote cardiac impact image segment from the video clip includes: establishing a facial model and marking multiple facial feature points; recording the position information of the multiple facial feature points in each frame of the video clip; tracking the multiple facial feature points based on the position information and drawing the motion trajectory of each of the multiple facial feature points in the vertical axis direction; performing a second quality assessment on the waveform signal of the motion trajectory; and selecting the motion trajectory with a higher quality score based on the result of the second quality assessment to further perform principal component analysis to reconstruct the remote cardiac impact image signal.

[0014] Preferably, the first quality assessment or the second quality assessment determines the quality fraction based on the percentage of power spectral density within a predetermined frequency range relative to the total power spectral density.

[0015] Preferably, the neural network model includes a U-shaped neural network model with a symmetrical structure that generates electrocardiogram segments based on cardiac impact image segments. The U-shaped neural network model has a three-layer data path consisting of a downsampled compression path and an upsampled expansion path. The shallow and middle layers of the three data paths have skip links that adaptively weight the features of the compression path and stack them with the features of the expansion path.

[0016] Preferably, the skip links that adaptively weight the features of the compressed path and stack them with the features of the expanded path include spatial attention and channel attention mechanisms to adaptively weight the features. The channel attention mechanism performs global average pooling on the input features to obtain a feature vector in the channel dimension, then passes it through a fully connected layer and an activation function to obtain a channel attention weight vector, and multiplies the channel attention weight vector with the input features to obtain the enhanced features. The spatial attention mechanism performs convolution on the input features to obtain a feature vector in the spatial dimension, then passes it through a fully connected layer and an activation function to obtain a spatial attention weight matrix, and multiplies the spatial attention weight matrix with the input features to obtain the enhanced feature representation.

[0017] Preferably, the input signal of the U-shaped neural network model is a one-dimensional signal, and the size of the convolution kernel gradually decreases as the number of layers increases.

[0018] Preferably, the U-shaped neural network model further includes a residual structure, which adds the input signal to the output signal and is used to skip connections between features that are convolutionally separated by one step in the compression path.

[0019] Preferably, the U-shaped neural network model includes both a cB2E U-shaped neural network model and an rB2E U-shaped neural network model. The cB2E U-shaped neural network model and the rB2E U-shaped neural network model are trained by using contact cardiac impact image segments or remote cardiac impact image segments of the subject in a state of no voluntary movement as input and electrocardiogram image segments as labels.

[0020] Another aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the heart rate detection method as described above.

[0021] Another aspect of the present invention provides a computer device, the computer device comprising: a processor; and a memory storing a computer program, wherein when the computer program is executed by the processor, the heart rate detection method described above is implemented.

[0022] This invention proposes a method for quality evaluation and optimal selection of both cBCG and rBCG signals to detect heart rate, thereby improving the accuracy of heart rate detection. The invention involves signal quality assessment of both cBCG and rBCG signals, retaining the higher-quality signal within a given time segment, thus achieving quality evaluation and optimal selection of dual-modal signals.

[0023] This invention proposes an adaptive reconstruction algorithm for rBCG signals to evaluate the quality of rBCG signals reconstructed from facial feature points. It automatically selects the number and position of feature points, filters out invalid facial feature point information, and avoids the disadvantage of large computational load caused by a large number of facial feature points.

[0024] This invention proposes a U-shaped neural network model to generate ECG from BCG, in order to further improve the accuracy of BCG heart rate detection.

[0025] In this invention, the size of the convolution kernel is gradually reduced as the number of layers increases in the U-shaped neural network model. This is beneficial for the network to extract high-dimensional details of features, and to obtain deep information while retaining shallow information. Attached Figure Description

[0026] The above and other aspects, features and advantages of the present invention will be more clearly understood from the following detailed embodiments, taken in conjunction with the accompanying drawings, in which:

[0027] Figure 1 This is a flowchart of a heart rate detection method based on dual-modal cardiac impulse signals according to an embodiment;

[0028] Figure 2 This is a flowchart of a method for obtaining remote cardiac impact image segments from video clips according to an embodiment;

[0029] Figure 3 It is a network diagram of the U-shaped neural network model according to the embodiment; and

[0030] Figure 4 This is a block diagram of a heart rate detection method based on dual-modal cardiac impulse signals. Detailed Implementation

[0031] The following detailed descriptions are provided to aid the reader in gaining a comprehensive understanding of the methods, apparatus, and / or systems described herein. However, various modifications, variations, and equivalents of the methods, apparatus, and / or systems described herein will be apparent to those skilled in the art. For example, the order of operations described herein is merely illustrative and is not limited to the order set forth herein; changes that will be apparent to those skilled in the art may be made, except for operations that must be performed in a specific order. Furthermore, for clarity and brevity, descriptions of features and structures well-known to those skilled in the art may be omitted. The features described herein may be implemented in different forms and will not be construed as being limited to the examples described herein. Rather, the examples provided herein make the invention thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0032] Technical terminology definition

[0033] BCG: Ballistocardiogram

[0034] cBCG: Contact Ballistocardiogram

[0035] rBCG: Remote Ballistocardiogram

[0036] PCA: Principal Component Analysis

[0037] ROI: Region of Interest

[0038] ECG: Electrocardiogram

[0039] SCG: Seismocardiogram

[0040] GAN: Generative Adversarial Network

[0041] MSE: Root Mean Squared Error

[0042] In the process of extracting heart rate from BCG signals, rBCG or cBCG signals are generally used. However, rBCG signals are easily interfered with by the subject's voluntary facial expressions, and cBCG signals are easily interfered with by the subject's voluntary movements, resulting in degraded signal waveforms that cannot be analyzed. Furthermore, BCG information from a single modality is easily affected by subjective actions or external factors. This invention proposes to evaluate the quality of dual-modality cBCG and rBCG signals and select the optimal signal to generate ECG signals.

[0043] This invention proposes first constructing a U-shaped neural network model for generating ECG signals based on BCG signals. The model is trained using rBCG and cBCG signals in a state without voluntary movement as inputs, and synchronous ECG signals as labels. A dual-modal BCG signal quality evaluation and optimization algorithm is applied to assess the quality of rBCG and cBCG signals within each time window. Signal segments with higher quality are used as input to generate gECG segments under measured conditions, and heart rate is then calculated.

[0044] Figure 1 This is a flowchart of a heart rate detection method based on dual-modal cardiac impulse signals according to an embodiment. (Refer to...) Figure 1 The heart rate detection method based on dual-modal cardiac impact signals according to the embodiment performs the following steps. In operation 101, quality assessments are performed on the waveform signals of contact cardiac impactogram (cBCG) segments acquired via a contact acquisition device and remote cardiac impactogram (rBCG) segments acquired via video segments, respectively. In operation 102, based on the quality assessment results, the cardiac impactogram (cBCG) segment with the higher quality score from the contact cardiac impactogram (cBCG) segment and the remote cardiac impactogram (rBCG) segment is selected. In operation 103, the cardiac impactogram (rBCG) segment with the higher quality score is input into a neural network model to obtain a generated electrocardiogram (gECG) segment.

[0045] This invention proposes a quality assessment of cBCG and rBCG fragments. The heart rate detection method based on dual-modal cardiac impulse signals according to the embodiments can realize mutual quality evaluation and optimal selection of dual-modal signals, improve the effectiveness and accuracy of heart rate detection, and reduce the problem of heart rate measurement failure caused by signal interference when using single-modal cBCG or rBCG signals to measure heart rate.

[0046] Figure 2 This is a flowchart of a method for obtaining remote cardiac impact image segments from video clips according to an embodiment.

[0047] Reference Figure 2According to an embodiment, the method for obtaining remote cardiac impaction image segments from video clips includes: Step 201, establishing a facial model and labeling multiple facial feature points. For example, a facial model can be established using a MediaPipe model, and 468 facial feature points can be labeled in the facial model. The facial model used in this invention labels significantly fewer facial feature points than conventional facial models. Step 202, recording the position information of the multiple facial feature points in each frame of the video clip. Step 203, tracking the multiple facial feature points based on the position information and depicting the motion trajectory of each of the multiple facial feature points in the vertical axis direction. The vertical axis direction can also be represented as the direction of gravity. Step 204, performing a quality assessment on the waveform signal of the motion trajectory. Optionally, filtering processing for the frequency range of the cardiac impaction image can be performed on the motion trajectory before the quality assessment, for example, the frequency range of the cardiac impaction image can be 0.5Hz-20Hz. The quality score may include a weighted sum of multiple scoring functions. Step 205, based on the results of the quality assessment, selecting the motion trajectory with a higher quality score for further principal component analysis to reconstruct the remote cardiac impaction image signal. Principal component analysis can be performed on the top m (e.g., m could be 25, but is not limited to) trajectory signals with relatively good quality, and the top five principal components can be summed and averaged. Through quality evaluation, the number of facial feature points labeled in the facial model can be further reduced, significantly reducing the computational cost of the network model.

[0048] The method for acquiring remote cardiac impact image segments from video clips according to the embodiments can retain high-quality signals per unit time, enabling mutual quality evaluation and optimal selection of dual-modal signals.

[0049] In the embodiments, in Figure 1 Quality assessment of images of central impact or Figure 2 The quality assessment of motion trajectories can be determined based on the percentage of the power spectral density of the waveform signal within a predetermined frequency range (e.g., 0.5 Hz-20 Hz) relative to the total power spectral density.

[0050] Figure 3 This is a network diagram of a U-shaped neural network model according to an embodiment.

[0051] Reference Figure 3In this embodiment, the neural network model includes a U-shaped neural network model with a symmetrical structure that generates electrocardiogram (ECG) segments based on cardiac impact image segments. The U-shaped neural network model has a three-layer data path consisting of a downsampled compression path 301 and an upsampled expansion path 302. For example, the signal can be downsampled using max pooling to extract features, and the signal can be upsampled using interpolation to reconstruct the signal. The shallow and middle layers of the three data paths have skip links 303 that adaptively weight the features from the compressed path and stack them with the features from the expansion path. Skip links 303 enable the combination of shallow and deep features, resulting in a more comprehensive feature representation.

[0052] The skip link 303, which adaptively weights the features of the compressed path and stacks them with the features of the expanded path, includes a dual attention module. The dual attention module mainly contains two attention mechanisms (AM, which refers to all attention mechanisms in general): channel attention and spatial attention.

[0053] Channel attention mechanisms enhance the model's focus on important features by calculating the importance of features along the channel dimension. Specifically, the channel attention mechanism performs global average pooling on the input features to obtain a feature vector along the channel dimension, and then passes it through a fully connected layer and activation function to obtain a channel attention weight vector. Multiplying this weight vector with the input features yields the enhanced features. Spatial attention mechanisms enhance the model's focus on spatial locations by calculating the importance of features at different locations. Specifically, the spatial attention mechanism performs convolution on the input features to obtain a feature vector along the spatial dimension, and then similarly passes it through a fully connected layer and activation function to obtain a spatial attention weight matrix. Multiplying this weight matrix with the input features yields the enhanced feature representation. Combining the weighted features of channel and spatial attention can further enhance the model's feature representation and discriminative abilities, improving model performance. By adaptively weighting the features from the encoding region into a dual attention mechanism and stacking them onto a feature layer of the same size as the decoding region, the fusion of features at different depths can be achieved, improving the quality of generated features. The spatial attention mechanism learns the weights for each spatial location and adjusts the representation of each location in the features, thereby better capturing spatial structural information. Channel attention mechanisms learn the weights of each channel and adjust the representation of each channel to better capture the relationships between features. The introduction of a dual attention module can improve the expressiveness of the U-shaped neural network model, thereby improving the quality of ECG generation results, preserving shallow information while obtaining deep information.

[0054] The input signal of the U-shaped neural network model is a one-dimensional signal, and the size of the convolutional kernel gradually decreases with the increase of the number of layers. Since the dimension of the input signal and the feature size in each convolutional layer gradually decrease, this invention proposes to modify the convolutional kernels so that their size gradually decreases with the increase of the number of layers. For example, the kernel sizes of three convolutional layers can be 14×1, 10×1, 6×1 or 7×1, 5×1, 3×1, respectively. The gradual decrease in kernel size with the increase of layers is beneficial for the network to extract high-dimensional details of features. The input signal length of the U-shaped neural network model can be 2000 sampling points. After three downsampling operations, the feature size changes from 2000 sampling points to 20 sampling points. 2000 sampling points is used as an example here, but this invention is not limited to this; other numbers of sampling points can also be used. In the B2E U-shaped network of this invention, the downsampling part reduces the amount of data by subsampling (subsampling is applicable to one dimension) of every n samples in the input signal (or by averaging n samples, but not limited to this). For example, in one-dimensional data, the first or average of every n samples is taken. The downsampling factor is 10, 5, and 2 each time, and after three downsampling operations, the feature size changes from 2000 to 20.

[0055] In the B2E U-shaped network of this invention, the width of each convolutional layer is represented as the feature size, and the number of feature channels (maps) is represented numerically. For example, in Figure 3 In this process, the single-channel input signal of a high-quality BCG fragment is calculated using first-order differentiation, second-order differentiation, and single-integration, and then merged with the original signal to obtain 4-channel feature data. This is then convolved to generate 12-channel feature data (mapping). Different convolutional kernels are then used to generate higher-dimensional 12-channel feature data. This data can be generated in two ways: firstly, through skip-link 303 weighting to produce 12-channel feature data, and then concatenating the 12-channel and 48-channel feature data to reduce the dimensionality back to 12-channel feature data; secondly, by downsampling to reduce the size and then performing dimensionality-up convolution to generate 48-channel feature data. Mapping refers to the output of the network's convolutional layers. Convolutional layers perform convolution operations on the input data using sliding convolutional kernels to obtain convolutional feature maps. Feature maps represent feature information at different locations and scales in the input data. Each feature map corresponds to a convolutional kernel in the convolutional layer and can be considered as a feature extraction from the input data. By continuously stacking multiple convolutional layers, more complex and deeper features can be extracted from the feature maps. The U-shaped neural network model also includes a residual structure 304, which adds the input and output signals to skip connections between features that are one convolution apart in the compressed path. This preserves information from the input signal, improving the quality of feature representation while retaining the original information, thus further enhancing the model's performance.

[0056] Figure 4 This is a block diagram of a heart rate detection method based on dual-modal cardiac impulse signals.

[0057] The neural network models include both cB2E U-shaped neural network models and rB2E U-shaped neural network models. Specifically, the cB2E (cBCG to ECG) U-shaped neural network model and the rB2E (rBCG to ECG) U-shaped neural network model are trained by taking the contact cardiac impact image segments or remote cardiac impact image segments of the subject in a state of no voluntary movement as input and the ECG image segments as labels.

[0058] Reference Figure 4 The heart rate detection method based on dual-modal cardiac impact signals includes: using cBCG signal 401 and rBCG signal 402 in the involuntary movement state of the subject as model inputs, and synchronously measured ECG signal as labels, to train U-shaped neural network model 403 of cB2E (cBCG to ECG) and U-shaped neural network model 404 of rB2E (rBCG to ECG) respectively.

[0059] In the actual heart rate measurement process, the measured cBCG signal 405 and the measured rBCG signal 406 are synchronously processed 407. For example, a time window with a width of 2000 and a step size of Fs can be used to synchronously extract bimodal signal segments, where Fs represents the sampling frequency. Quality assessment is performed on the bimodal signal segments 408. The quality score of the quality assessment may include a weighted sum of multiple scoring functions. Signal segments with higher quality scores within the same time window are input into the corresponding trained model to obtain gECG segments and calculate the heart rate 409. The gECG segments can be spliced ​​together to generate continuous ECGs.

[0060] In detail, the heart rate detection method based on dual-modal cardiac impulse signals according to the embodiments may include the following steps:

[0061] Step 1: Acquire video clips of the subjects, each video clip containing at least n frames of data;

[0062] Step 2: Build a facial model and label multiple facial feature points. Here, the video stream can be input into the MediaPipe network to obtain the location information (x, y) of facial feature points i in each frame. i ,y i ), forming a precise facial grid, where 1≤i≤468;

[0063] Step 3: Record the position coordinates of feature point i on the vertical axis, and track the vertical motion trajectory signal rBCG corresponding to each feature point. i ;

[0064] Step 4: For rBCG i(t) is processed using the Hanning window to obtain the signal rBCG with window function w(n). i w(n), where the window length is L. rBCG i w(n)=rBCG i (t)*w(n), where n = 0, 1, ..., L-1, and t represents time.

[0065] Step 5: For rBCG i Performing a Fast Fourier Transform on w(n) yields the frequency domain representation rBCG. i (k).

[0066] Step 6: For rBCG i (k) is squared to obtain the power spectral density P(k) = |rBCG|. i (k)|^2

[0067] Step 7: Integrate the power spectral density P(k) over the frequency range of 0.5Hz to 20Hz to obtain the power spectral density P(0.5Hz-20Hz) = sum(P(f))*df in the range of 0.5Hz-20Hz. The mass fraction S(i) is obtained as a percentage of the power spectral density within a predetermined frequency range f relative to the total power spectral density. f can be in the range of 0.5-20Hz.

[0068] Step 8: Sort according to S(i), and select the 25 rBCGs with the best quality. i Principal component analysis was performed on the trajectory.

[0069] Step 9: Add 25 rBCGs i The trajectories are arranged into a matrix X, where each row represents the same trajectory and each column represents the same time point. The matrix size is 25×n, where n is the trajectory length or the number of video frames.

[0070] Step 10: Perform mean centering on matrix X, that is, subtract the average value of each column from the average value of that column.

[0071] Step 11: Perform principal component analysis on the centered matrix X to obtain its first K principal components, with corresponding weight vectors w1, w2, ..., wK, where K is the number of principal components to be retained.

[0072] Step 12: For each rBCG trajectory, perform a linear combination of its corresponding K principal components to obtain the projection vector of the trajectory in the principal component space, denoted as y1, y2, ..., yK. The specific calculation method is: yi = [x1i, x2i, ..., x25i] · wi, where xi is the value of the i-th rBCG trajectory at time point j, and wi is the weight vector of the i-th principal component.

[0073] Step 13: Sum and average the projection vectors of all trajectories in the principal component space to obtain the projection vector of the average value of all trajectories in the principal component space at this moment, denoted as y_mean. The specific calculation method is: y_mean = (y1 + y2 + ... + y25) / 25.

[0074] Step 14: Project y_mean back into the original signal space to obtain the average rBCG signal, rBCG=w1·y_mean(1)+w2·y_mean(2)+...+wK·y_mean(K) / K.

[0075] Step 15: Select K=5 to obtain the rBCG signal of the sum of the first 5 principal components.

[0076] The rBCG signal can be obtained from the video data through steps 1 to 15 above.

[0077] Step 16: Segment the rBCG, cBCG, and ECG signals of the subject in a state of no voluntary movement to obtain a signal segment with 2000 sampling points.

[0078] Step 17: Calculate the first derivative drBCG, the second derivative ddrBCG, and the first integral irBCG of the rBCG signal.

[0079] Step 18: Calculate the first-order differential dcBCG, the second-order differential ddcBCG, and the first-order integral icBCG of the cBCG signal.

[0080] Step 19: Build a U-shaped neural network model with an input signal length of 2000. Use two downsampling operations, with each downsampling operation having a multiplier of 10, 5, and 2. After three downsampling operations, the feature map size changes from 2000 to 20.

[0081] Step 20: In order to obtain the feature representation of the waveform, the convolution kernel of the U-shaped neural network is set to 14×1, 10×1 and 6×1 respectively. As the number of layers increases, the size of the convolution kernel is gradually reduced, which is beneficial for the network to extract high-dimensional details of features.

[0082] Step 21: In order to obtain deep information while preserving shallow information, a dual attention module and a residual structure are added to the jump connections of the U-shaped neural network model.

[0083] Step 22: Train the rB2E U-shaped neural network model and the cB2E U-shaped neural network model by taking the synchronized rBCG, drBCG, ddrBCG, irBCG and cBCG, dcBCG, ddcBCG, icBCG as model inputs and ECG as labels.

[0084] The rB2E U-shaped neural network model and the cB2E U-shaped neural network model can be trained through steps 16 to 22 above.

[0085] Step 23: Synchronize the measured cBCG signal with the measured rBCG signal, and use a time window with a sampling frequency of 2000 sampling points and a sampling step size Fs to segment the dual-mode signal.

[0086] Step 24: Perform the operations from Step 4 to Step 8 on the dual-mode signals to obtain their respective quality fractions S.

[0087] Step 25: Select a high-quality signal segment with a high quality score S as the model input, input it into the U-shaped neural network model trained in Step 22, and output the gECG segment within the corresponding time window.

[0088] Step 26: Repeat steps 23 to 25 to achieve continuous, unobtrusive heart rate measurement by splicing gECG segments.

[0089] An exemplary embodiment of the present invention also provides a computer-readable storage medium storing a computer program. The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to execute the heart rate detection method based on a dual-modal cardiac impulse signal according to the present invention. The computer-readable recording medium is any data storage device capable of storing data readable by a computer system. Examples of computer-readable recording media include: read-only memory, random access memory, read-only optical disc, magnetic tape, floppy disk, optical data storage device, and carrier waves (such as data transmission via the Internet through wired or wireless transmission paths).

[0090] An exemplary embodiment of the present invention also provides a computer device. The computer device includes a processor and a memory. The memory stores a computer program. The computer program is executed by the processor, causing the processor to execute the computer program of the heart rate detection method based on a dual-modal cardiac impulse signal according to the present invention.

[0091] This invention proposes a method for quality evaluation and optimal selection of both cBCG and rBCG signals to detect heart rate, thereby improving the accuracy of heart rate detection. The invention involves signal quality assessment of both cBCG and rBCG signals, retaining the higher-quality signal within a given time segment, thus achieving quality evaluation and optimal selection of dual-modal signals.

[0092] This invention proposes an adaptive reconstruction algorithm for rBCG signals to evaluate the quality of rBCG signals reconstructed from facial feature points. It automatically selects the number and position of feature points, filters out invalid facial feature point information, and avoids the disadvantage of large computational load caused by a large number of facial feature points.

[0093] This invention proposes a U-shaped neural network model to generate ECG from BCG, in order to further improve the accuracy of BCG heart rate detection.

[0094] In this invention, the size of the convolution kernel is gradually reduced as the number of layers increases in the U-shaped neural network model. This is beneficial for the network to extract high-dimensional details of features, and to obtain deep information while retaining shallow information.

[0095] While this invention includes specific examples, it will be apparent to those skilled in the art that various changes in form and detail may be made to these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein are to be considered descriptive only and not for limiting purposes. The description of features or aspects in each example is to be considered applicable to similar features or aspects in other examples. Suitable results may be obtained if the described techniques are performed in a different order, and / or if components in the described system, architecture, apparatus, or circuit are combined in a different manner and / or if components in the described system, architecture, apparatus, or circuit are replaced or supplemented by other components or their equivalents. Therefore, the scope of this invention is not limited by the specific embodiments but by the claims and their equivalents, and all variations within the scope of the claims and their equivalents are to be construed as included in this invention.

Claims

1. A heart rate detection method based on dual-modal cardiac impulse signals, characterized in that, The heart rate detection method includes: The waveform signals of the contact cardiac impact image segments acquired by the contact acquisition device and the remote cardiac impact image segments acquired by the video clips are subjected to a first quality assessment. Based on the results of the first quality assessment, the cardiac impact image segment with the higher quality score among the contact cardiac impact image segments and the remote cardiac impact image segments is selected; and The high-quality cardiac impact image segments are input into a neural network model to obtain electrocardiogram (ECG) image segments. The acquisition of the remote cardiac impact image segment through the video clip includes: Create a facial model and label multiple facial feature points; Record the position information of the multiple facial feature points in each frame of the video segment; Based on the location information, the plurality of facial feature points are tracked and the motion trajectory of each of the plurality of facial feature points in the vertical axis direction is drawn; Perform a second quality assessment on the waveform signal of the motion trajectory; and Based on the results of the second quality assessment, the motion trajectory with the higher quality score was selected for further principal component analysis to reconstruct the remote cardiac impact map signal.

2. The heart rate detection method according to claim 1, wherein, The first quality assessment or the second quality assessment determines the quality score based on the percentage of the power spectral density within a predetermined frequency range of the waveform signal relative to the total power spectral density.

3. The heart rate detection method according to claim 1, wherein, The neural network model includes a U-shaped neural network model with a symmetrical structure that generates electrocardiogram (ECG) segments based on cardiac impact image segments. The U-shaped neural network model includes a three-layer data path consisting of a downsampled compression path and an upsampled expansion path. Among them, the shallow and middle-layer paths in the three-layer data path include skip links that adaptively weight the features of the compressed path and stack them with the features of the expanded path.

4. The heart rate detection method according to claim 3, wherein, The skip links that adaptively weight the features of the compressed path and stack them with the features of the expanded path include spatial attention and channel attention mechanisms to adaptively weight the features. The channel attention mechanism performs global average pooling on the input features to obtain feature vectors along the channel dimension. Then, it passes through a fully connected layer and an activation function to obtain channel attention weight vectors. Finally, it multiplies the channel attention weight vectors with the input features to obtain the enhanced features. The spatial attention mechanism performs convolution operations on the input features to obtain feature vectors in the spatial dimension, and then obtains a spatial attention weight matrix through a fully connected layer and an activation function. The spatial attention weight matrix is ​​multiplied by the input features to obtain the enhanced feature representation.

5. The heart rate detection method according to claim 3, wherein, The input signal of the U-shaped neural network model is a one-dimensional signal, and the size of the convolution kernel of the convolutional layer gradually decreases as the number of layers increases.

6. The heart rate detection method according to claim 3, wherein, The U-shaped neural network model also includes a residual structure, which adds the input signal to the output signal and is used to skip connections between features that are convolutionally separated by one step in the compression path.

7. The heart rate detection method according to claim 3, wherein, The U-shaped neural network model includes both a cB2E U-shaped neural network model and an rB2E U-shaped neural network model. The cB2E U-shaped neural network model and the rB2E U-shaped neural network model are trained by using contact cardiac impact image segments or remote cardiac impact image segments of the subject in a state of no voluntary movement as input and electrocardiogram segments as labels.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the heart rate detection method as described in any one of claims 1 to 7.

9. A computer device, characterized in that, The computer device includes: processor; A memory storing a computer program that, when executed by a processor, implements the heart rate detection method as described in any one of claims 1 to 7.

Images