A contactless heart rate estimation method, system and apparatus

Abstract

The application discloses a non-contact heart rate estimation method, system and device. The method comprises the following steps: acquiring a video frame sequence containing a face, and performing space-time block embedding on the video frame sequence to generate a Token sequence; detecting and locating key points of a face region in the video frame, extracting an interest region located in a forehead, left cheek, right cheek and chin region and excluding an eye, mouth, hair and background region according to the locating result, and generating a space mask based on the interest region, wherein the space mask indicates whether each Token in the Token sequence corresponds to a valid physiological region; and down-sampling the space mask to align with the feature map resolution of the Token sequence, and performing numerical mapping on the down-sampled mask. The application aims to solve the technical problems of high computational complexity of an existing video physiological measurement network based on a Transformer, difficulty in hardware deployment on an end side and lack of physiological constraints and environmental interference.

Description

Technical Field

[0001] This invention relates to the field of biomedical signal processing and computer vision, specifically to a non-contact heart rate estimation method, system and device, and in particular to a non-contact heart rate estimation technology that combines facial feature tracking and local sparse attention networks, suitable for low-power, real-time physiological signal monitoring scenarios on mobile and edge devices. Background Technology

[0002] Currently, in the interdisciplinary field of biomedical signal processing and computer vision, video-based non-contact heart rate estimation (i.e., remote photoplethysmography, rPPG) is typically implemented using deep learning networks based on the Transformer architecture. For example, Chinese patent application CN113408508A discloses a Transformer-based non-contact heart rate measurement method, which acquires a sequence of video frames containing facial information, obtains a sequence of facial region of interest (ROI) images using a facial landmark model, and then obtains the heart rate sequence using an end-to-end Transformer model. Chinese patent CN112200162B discloses a non-contact heart rate measurement method based on an end-to-end network, which acquires a sequence of video frames containing facial information, obtains a sequence of facial ROI images using an MTCNN face detection model and a facial landmark model, and obtains the heart rate value corresponding to each frame image based on the ROI image sequence using a trained end-to-end dual-branch network.

[0003] Building upon this foundation, to further improve the feature extraction efficiency of Transformer in long video sequences, improved schemes based on sparse attention mechanisms have emerged in existing technologies. Represented by RhythmFormer: Extracting patterned rPPG signals based on periodic sparse attention (Pattern Recognition, 2025), this type of scheme utilizes the inherent quasi-periodicity of rPPG signals to learn periodic patterns through a data-driven approach. It introduces a pre-attention stage before the traditional attention mechanism to filter out a large amount of irrelevant attention computation, enabling the network to extract periodic rPPG features with fine-grained precision across multiple time scales. Furthermore, Chinese patent application CN114596963A discloses a remote non-contact heart rate estimation method based on sparse structure representation. This method segments the face into multiple sub-regions and uses a structure dictionary composed of cosine bases of different frequencies and wavelet bases of different scales to solve for the coefficient matrix, thereby reconstructing the heart rate signal.

[0004] The above-mentioned existing technology has some shortcomings; please refer to the appendix. Figure 1The figure illustrates a typical rPPG signal extraction process based on global Top-K sparse attention in existing technologies. This process typically includes: 1. Input & Patch Embedding: The input is a video sequence containing faces. The network first divides the video frames into a large number of small three-dimensional image patches in the spatial and temporal dimensions, and flattens these image patches and maps them into a series of tokens (word vectors).

[0005] 2. Global Attention & Top-K Selection: At this stage, the tokens encompass all regions of the video frame (including facial skin, hair, eyes, background, clothing, etc.). The network's core attention module calculates cross-similarity for all tokens globally. Subsequently, through a data-driven approach, the network sorts the attention weights of all tokens and forcibly selects the top K (Top-K) tokens with the highest weights. Theoretically, these K tokens are considered by the network to contain the region with the strongest periodic rPPG signal.

[0006] 3. Feature Aggregation & Output: The network uses only the selected K tokens for subsequent spatiotemporal feature fusion and regression calculation, and finally outputs the predicted rPPG waveform or heart rate value.

[0007] However, this existing technical solution, which relies entirely on the network itself for global computation and data-driven filtering, has the following shortcomings in practical applications: First, the computational complexity is excessively high. In some existing schemes based on global sparse attention (such as those employing a global Top-K selection mechanism), before performing Top-K selection, the network must first complete the self-attention calculation among the global N tokens (with a complexity of O(N²)), followed by a global sorting operation. In face videos, the effective skin regions (forehead, cheeks, etc.) that actually contain rPPG signals typically account for less than 30% of the total image area, while irrelevant regions such as background, clothing, and hair make up the vast majority. Existing technologies force the network to expend a significant amount of computing power to calculate the attention weights of these background regions that lack physiological signals, resulting in a serious waste of computational resources.

[0008] Second, it is difficult to deploy in real time on mobile and edge devices. The computing power, memory bandwidth, and power consumption of mobile devices (such as smartphones and edge computing boxes) are strictly limited. In the aforementioned scheme based on the global Top-K mechanism, a large number of global matrix multiplications and sorting operations (sorting operations are not friendly to neural network processors (NPUs) and are difficult to be effectively accelerated by hardware instruction sets) result in extremely long model inference times and a surge in power consumption. In actual tests, this often leads to device overheating, frequency reduction, and frame drops, making it impossible to achieve high frame rates (such as above 30 frames per second) for real-time continuous heart rate monitoring on ordinary mobile devices. Although CN113408508A claims to reduce computing costs, it still uses a global Transformer architecture and does not fundamentally solve the computing power bottleneck problem of global attention calculation and sorting operations.

[0009] Third, the lack of prior physiological constraints makes it susceptible to environmental noise interference. Current Top-K screening is a purely data-driven "black box" process. When the video shows background screen flickering similar to the heart rate cycle, localized high-brightness reflection changes, or frequent blinking and mouth movements by the subject, the network is highly likely to misclassify these "high-contrast, highly periodic" noise areas as valid signals and assign them extremely high attention weights. Due to the lack of prior medical and physiological knowledge, this misleading can directly lead to serious deviations in heart rate estimation results, especially under complex lighting or motion scenarios, where the measurement accuracy and robustness of existing technologies significantly decrease.

[0010] To address the technical problems of existing technologies based on the global Top-K mechanism, such as redundant global attention computation, unfriendly sorting operations to edge hardware, and poor anti-interference ability due to lack of physiological prior constraints, this invention provides a non-contact heart rate estimation method, device, and storage medium that combines facial feature tracking and local sparse attention networks. The aim is to introduce a physiological prior mask generated by facial keypoint tracking technology to directly guide the network to focus on effective facial skin regions in a deterministic, non-learning manner, replacing the sparse strategy in existing technologies that rely on data-driven learning. This fundamentally reduces computational complexity and edge deployment barriers while ensuring heart rate estimation accuracy, and improves the model's anti-interference ability and robustness in complex scenarios. Summary of the Invention

[0011] To address the technical problems of existing rPPG signal extraction schemes based on global Top-K sparse attention, such as redundant and blind global attention computation, unfriendly sorting operations to edge hardware, and poor anti-interference ability due to lack of physiological prior constraints, this invention provides a non-contact heart rate estimation method, device, and storage medium that combines facial feature tracking and local sparse attention networks. The aim is to generate physiological prior masks by introducing lightweight facial keypoint tracking technology, directly guiding the network to focus only on effective facial skin regions. This significantly reduces computational complexity and edge deployment threshold while ensuring heart rate estimation accuracy, and improves the model's anti-interference ability and robustness in complex scenarios.

[0012] According to one aspect of the present invention, a non-contact heart rate estimation method is provided, the method comprising: Obtain a video frame sequence containing a face, and perform spatiotemporal block embedding on the video frame sequence to generate a token sequence; The facial regions in the video frames are detected and key points are located. Based on the location results, regions of interest are extracted from the forehead, left cheek, right cheek and chin areas, excluding the eyes, mouth, hair and background areas. A spatial mask is generated based on the regions of interest. The spatial mask indicates whether the image area corresponding to each token in the token sequence is a valid physiological area. The spatial mask is downsampled to align with the feature map resolution of the token sequence, and the downsampled mask is numerically mapped, wherein the effective physiological region is mapped to the first value and the invalid region is mapped to the second value; after the second value is activated by the Softmax function, the corresponding attention weight approaches zero; Attention computation is guided by the spatial mask to block attention interaction between invalid regions and other tokens, and local sparse attention computation is performed only on tokens within the valid physiological region. The feature sequence after multi-layer local sparse attention computation is spatiotemporally aggregated and decoded to output a volumetric pulse wave waveform signal. The heart rate estimate is determined based on the volumetric pulse wave waveform signal.

[0013] The above technical solution provides a non-contact heart rate estimation method. This method extracts the forehead, cheeks, and chin as effective physiological regions by locating facial key points, excluding the eyes, mouth, hair, and background regions, generating a spatial mask. This mask, after downsampling and numerical mapping, is used to guide the Transformer's self-attention calculation, ensuring the network only interacts with features within local skin regions strongly correlated with the rPPG signal, ultimately outputting a heart rate estimate.

[0014] Compared with existing technologies based on global Top Compared to the rPPG measurement scheme of K-sparse attention, the essential feature of this scheme is that the spatial mask is generated based on the facial key point localization results, and its effective area is the forehead, cheeks, and chin—that is, areas rich in microvessels and less affected by muscle movement, while actively excluding the eyes, mouth, hair, and background areas. After numerical mapping, this spatial mask is directly superimposed on the scaling dot product matrix of self-attention, so that the attention weight of invalid areas after Softmax activation is reduced to zero. This scheme replaces the data-driven paradigm of "global calculation and sorting and filtering" in the existing technology with "physiological prior anchoring".

[0015] Based on the aforementioned essential characteristics, this solution has at least the following three beneficial effects: First, in terms of computational efficiency, this scheme eliminates the need for the Transformer network to perform self-attention calculations (complexity O(N²)) and sorting operations on global tokens, thus reducing attention calculations to an effective skin area that occupies less than 30% of the total graph area. This scheme reduces the number of floating-point operations in the core attention module by approximately 60% to 80% and completely eliminates the global sorting operation. This efficiency improvement exceeds what a person skilled in the art could reasonably expect based on conventional optimization methods prior to the application date.

[0016] Secondly, regarding edge deployment, this solution transforms the dynamic sorting and dynamic memory allocation operations in existing technologies into mask addition and mask overlay operations for fixed-dimensional matrices. Experimental data shows that this solution can increase the inference frame rate from less than 15 frames per second to over 30 frames per second on ordinary smartphones. The numerical mapping in this solution—mapping the effective region to the first value and the invalid region to the second value—works in conjunction with the Softmax activation function, aiming to zero out the weights of invalid regions during self-attention computation. The parameter selection and placement of this numerical mapping are designed based on a deep understanding of the mathematical properties of the Softmax activation function in the Transformer self-attention mechanism, making the blocking effect of the mask mathematically deterministic.

[0017] Third, regarding anti-interference capabilities, this scheme uses anatomical priors (forehead, cheeks, and chin are valid regions, while eyes, mouth, hair, and background are invalid regions) to rigidly block the network's attention calculations on non-physiological regions, eliminating the pathway for the network to learn periodic noise such as background flickering and localized highlights. This is fundamentally different from the purely data-driven Top-K selection in existing technologies—the latter is prone to misjudging strong periodic noise as valid signals in complex lighting or motion scenarios.

[0018] In summary, this solution effectively addresses the technical problems of redundant computing power, difficult edge deployment, and poor anti-interference capability in existing rPPG solutions based on the global Top-K mechanism through the technical path of "physiological prior mask generation and numerical mapping → mask superposition to scaled dot product matrix → local sparse attention calculation". It achieves improvements over existing technologies in three aspects: computing efficiency, hardware deployment friendliness, and measurement robustness, and provides an feasible technical solution for the real-time edge application of non-contact heart rate estimation.

[0019] In some embodiments, the local sparse attention computation does not perform an attention weight sorting operation on the global tokens; the range of the effective physiological region is directly preset by the spatial mask.

[0020] In the above technical solution, it is clearly stated that the operation of sorting the global tokens by attention weight is not performed in the local sparse attention calculation, and the range of the effective physiological region is directly preset by the spatial mask, rather than dynamically determined by data-driven methods.

[0021] Compared with existing solutions based on global Top-K sparse attention, the essential feature of this solution is that: the local sparse attention calculation completely eliminates the step of sorting the attention weights of global tokens; the range of the effective physiological region is directly preset by the spatial mask, rather than being determined by filtering after sorting the global attention weights through the network.

[0022] First, regarding computational efficiency, the computational cost of the global sorting operation (time complexity O(N×log2N)) increases superlinearly with the number of tokens N. This solution pre-defines the effective region using a mask, allowing the network to determine the computation range without calculating the global attention weight matrix, fundamentally skipping the sorting step. This solution reduces the number of floating-point operations in the core attention module by approximately 60% to 80%. The prerequisite for skipping sorting in this solution is the mask pre-definement mechanism—which requires pre-determining the effective region without global attention calculation. Its implementation relies on a complete pre-process of facial keypoint localization and ROI extraction.

[0023] Second, in terms of hardware deployment, global sorting operations are not friendly to neural network processors (NPUs) and are difficult to be effectively accelerated by hardware instruction sets. In existing technologies, global sorting and dynamic filtering operations often become bottlenecks for edge-side inference. This solution transforms dynamic sorting into static mask presets, so that the scope of attention calculation is determined before inference, eliminating the need for dynamic decision-making at runtime, thereby reducing the memory scheduling overhead and inference latency of edge devices.

[0024] In summary, this solution further enhances the advantages of the aforementioned method in terms of computational efficiency and hardware deployment friendliness through the technical means of "no sorting + mask preset", and clearly distinguishes this solution from existing technical solutions that rely on global sorting in terms of implementation path.

[0025] In some embodiments, the attention calculation guided by the spatial mask includes: The token sequence is linearly transformed into a query matrix, a key matrix, and a value matrix, and the mapped spatial mask is superimposed on the scaled dot product matrix during self-attention computation.

[0026] The above technical solution clarifies a specific implementation method for attention computation guided by spatial masks: the token sequence is linearly transformed into a query matrix Q, a key matrix K, and a value matrix V, and the mapped spatial mask is superimposed on the scaled dot product matrix QK during self-attention computation. superior.

[0027] Compared with existing schemes based on global Top-K sparse attention, the essential feature of this scheme is that the superposition position of the spatial mask is the scaled dot product matrix QK. (That is, before the Softmax activation function), rather than post-processing the attention weights after Softmax, and not masking the token features at the input. This choice of overlay position allows the second value in the mask to directly participate in the exponential operation of the Softmax activation function, utilizing... The mathematical principles behind this approach fundamentally block the attention weights in ineffective regions.

[0028] The beneficial effects of this feature are reflected in the following two aspects: First, regarding the thoroughness of attention blocking, because the mask is superimposed before Softmax, the attention weights corresponding to invalid regions are set to zero during the normalization phase, rather than being masked or truncated after weight calculation. The value corresponding to the invalid region in the mask is... (In the specific embodiment, it is -1e9), its natural index after Softmax activation The corresponding attention weights are forced to zero. This mechanism ensures that invalid regions do not participate in subsequent feature aggregation under any circumstances, and its blocking effect is mathematically deterministic.

[0029] Secondly, regarding computational efficiency, since the attention weights in invalid regions are zeroed out during the Softmax stage, the network does not need to allocate any computational resources to invalid regions. Compared to existing technologies that first complete global attention calculation and then select the Top-K regions through ranking, this scheme achieves computational cost reduction during the attention calculation stage. The numerical mapping method of the mask in this scheme (0 for valid regions and -∞ for invalid regions) is designed in conjunction with the mathematical characteristics of the Softmax activation function. Their cooperation utilizes the monotonicity and zero-approaching characteristics of the exponential function during Softmax normalization, representing an adaptive improvement to the inherent computational logic of the Transformer self-attention mechanism.

[0030] In summary, this scheme clarifies a preferred technical path for "attention computation guided by spatial masks" by limiting the specific implementation method of mask superposition to scaled dot product matrix, thus providing structural support for the feasibility of the scheme.

[0031] In some embodiments, the attention calculation guided by the spatial mask includes: The spatial mask is convolutionally encoded and used as a query vector. The token sequence is linearly transformed and used as a key vector and a value vector. Physiologically relevant features that match the spatial mask are retrieved from the token sequence through a cross-attention mechanism.

[0032] In the above technical solution, another specific implementation method based on spatial mask-guided attention computation is provided: the spatial mask is used as a query vector after convolutional encoding, and the token sequence is used as a key vector and value vector after linear transformation. Physiologically relevant features that match the spatial mask are retrieved from the token sequence through a cross-attention mechanism.

[0033] Compared to the aforementioned default scheme (masks are superimposed onto the scaled dot product matrix), the essential characteristic of this scheme is that the masks are not directly superimposed onto the self-attention QK. Instead of using a matrix, the query vector is convolutionally encoded and used as an independent query vector. A cross-attention mechanism is then used to retrieve features matching the mask from the global token sequence. This scheme modifies the computation within the self-attention mechanism (QK). The + M) is transformed into an independent retrieval branch outside of attention (Query=mask, Key / Value=Token sequence), so that the mask is no longer a "correction" to the existing attention calculation, but actively "extracts" features related to physiological regions.

[0034] The beneficial effects of this feature are reflected in the following two aspects: First, in terms of architectural compatibility, the cross-attention scheme does not involve any modification to the self-attention calculation formula; physiological guidance can be achieved simply by adding an independent cross-attention layer to the network. This implementation has better compatibility with existing pre-trained Transformer weights—when fine-tuning using pre-trained weights, the parameters of the original self-attention layer do not need to be adjusted; only the newly added cross-attention layer needs to be trained.

[0035] Second, regarding the flexibility of feature retrieval, when the mask interacts with the global token sequence as a query vector in the cross-attention mechanism, the degree of matching between each token and the physiological prior mask can be quantitatively evaluated through the distribution of attention weights, rather than a binary determination of validity or invalidity. This mechanism, while preserving physiological prior constraints, allows the network to adaptively adjust the retrieval scope based on the actual image content. The input configuration of cross-attention in this scheme—the mask as the query vector and the token sequence as the key / value vector—has specific technical implications: this configuration explicitly limits the network's retrieval target to spatial locations matching the physiological prior, rather than arbitrary feature retrieval driven by data.

[0036] In summary, this scheme provides a parallel implementation path for "attention computation guided by spatial mask" that differs from the default scheme by limiting the implementation method of cross attention. While retaining the effect of physiological prior constraints, it also takes into account the compatibility of model architecture and the flexibility of feature retrieval.

[0037] In some embodiments, the attention calculation guided by the spatial mask includes: Before the token sequence is linearly transformed into a query matrix, a key matrix, and a value matrix, the spatial mask is multiplied element-wise with the token sequence to clear the token feature values ​​corresponding to invalid regions to zero.

[0038] In the above technical solution, another specific implementation method based on spatial mask-guided attention computation is provided: before the token sequence is linearly transformed into query matrix, key matrix and value matrix, the spatial mask is multiplied element by element with the token sequence, and the token feature value corresponding to the invalid region is cleared to zero.

[0039] Compared to the aforementioned default scheme (mask superimposed onto the scaled dot product matrix) and the aforementioned alternative scheme (cross-attention retrieval), the essential characteristic of this scheme is that the mask's application position is moved to the very beginning of the self-attention calculation—that is, the removal of invalid regions is completed before the token sequence enters the Q / K / V linear transformation, rather than during the self-attention calculation process (QK... Guidance can be provided either in the stage of attention or outside of the attentional layer (cross-attention layer).

[0040] The beneficial effects of this feature are reflected in the following two aspects: First, in terms of computational efficiency, since the token feature value corresponding to the invalid region is cleared to zero before the Q / K / V linear transformation, all subsequent computational steps—including the multiplication of the three linear transformation matrices Q / K / V, and the scaling of the dot product matrix QK—are efficient. The computation of the algorithm and Softmax normalization do not require allocating computational resources to invalid tokens. Compared to alternative solutions, this solution further advances the computational reduction from the self-attention computation stage to the feature projection stage. This solution also achieves the goal of skipping sorting and reducing computational power.

[0041] Secondly, regarding simplicity, the pre-masking scheme does not require modification of the self-attention computation logic. It only requires adding an element-wise multiplication operation after the token sequence is generated to completely remove invalid regions. This operation has minimal intrusion into the existing Transformer code structure, facilitating rapid integration into existing frameworks. The design intent of the pre-masking in this scheme—namely, achieving sparsity in all subsequent computational stages by zeroing out the tokens at the input—constitutes a different technical approach from the default and alternative schemes. Its advantage lies in its simplicity and independence from mathematical modifications to the self-attention mechanism. However, its cost is that the tokens are truncated before entering self-attention, potentially resulting in the loss of boundary information. This cost is acceptable in the application scenario described in this scheme because invalid regions are physiologically prior to be determined as not containing rPPG signals, and zeroing them out does not result in the loss of valid information.

[0042] In summary, this solution, by limiting the implementation method of the foreground mask, provides a solution that differs from the default solution (QK) for "attention computation guided by spatial masks". The third parallel implementation path, which combines superposition and cross-attention schemes, achieves the most thorough input-level computation pruning with minimal architectural changes while preserving the effects of physiological prior constraints.

[0043] In some embodiments, generating a spatial mask based on the region of interest includes: generating a binarized image using a polygon filling algorithm based on the boundary key points of the region of interest, with a pixel value of 1 for the effective region and a pixel value of 0 for the invalid region, and then downsampling the binarized image through a pooling operation.

[0044] The above technical solution clarifies the specific implementation method of generating spatial masks based on regions of interest: according to the boundary key points of the regions of interest, a polygon filling algorithm is used to generate a binarized image (the effective region pixel value is 1, and the invalid region pixel value is 0), and then the binarized image is downsampled through pooling operation.

[0045] Compared with existing technologies that directly use face detection boxes as ROI regions or use fixed-size rectangles to extract facial regions, the essential feature of this scheme is that the generation path of the binarized mask is "boundary key points → polygon filling → binarized image", and the shape of its effective region is determined by the boundary key points of the region of interest through a polygon filling algorithm.

[0046] The beneficial effects of this feature are reflected in the following two aspects: First, in terms of spatial accuracy, the boundaries of the effective region in the binarized mask are precisely determined by the keypoint localization results, accurately conforming to the actual contours of irregular facial areas such as the forehead, cheeks, and chin. The regions of interest are four polygonal regions extracted based on a keypoint index table; their shapes vary depending on individual facial features, accurately excluding areas such as the eyes, mouth, hair, and background. Compared to fixed templates or bounding boxes, this scheme avoids the problems of mistakenly including invalid regions in the effective region or mistakenly excluding effective edge regions.

[0047] Second, regarding architectural compatibility, the pooling operation downsamples the high-resolution binarized mask to align with the resolution of the token sequence feature map, allowing the mask to directly participate in the self-attention calculation of each Transformer layer. This alignment provides size-matched input for subsequent mask stacking in the backbone network. In this scheme, polygon filling converts discrete boundary keypoints into closed connected regions, while the pooling operation adjusts the mask to the tensor dimension usable by the network while preserving the spatial distribution of the regions. The specific order of this combination—filling before pooling—is to achieve resolution adaptation while maintaining regional connectivity; reversing the order (pooling keypoints before filling) would lead to distortion of region boundaries.

[0048] Third, in terms of computational efficiency, the numerical form of effective region pixel values ​​being 1 and invalid region pixel values ​​being 0 in the binarized mask makes subsequent pooling operations extremely efficient—it is only necessary to count whether there are non-zero pixels within the pooling window to determine whether the position belongs to the effective region after downsampling, without the need for complex numerical calculations.

[0049] In summary, this scheme provides an implementation method for "generating spatial masks based on regions of interest" by limiting the mask generation path to "polygon filling → binarization → pooling downsampling", ensuring the spatial accuracy of the mask and its architectural compatibility with the backbone network feature map.

[0050] In some embodiments, the spatial mask is a soft mask containing continuous probability weights, and the weight values ​​of each region are preset according to physiological prior knowledge. The soft mask is applied to the attention weight matrix calculated by self-attention through element-wise multiplication.

[0051] In the above technical solution, an alternative to hard mask (binarization mask) is provided: the spatial mask is a soft mask containing continuous probability weights, the weight values ​​of each region are preset according to physiological prior knowledge, and the soft mask is applied to the attention weight matrix after self-attention calculation by element-wise multiplication.

[0052] Compared to the aforementioned hard mask (where effective area pixel values ​​are 1 and invalid area pixel values ​​are 0), the essential features of this scheme are twofold: First, the mask's numerical form is a continuous probability weight, rather than a binary 0 / 1; second, the soft mask is applied to the attention weight matrix after self-attention calculation (after the Softmax output), rather than being superimposed on the scaled dot product matrix (before Softmax). Specifically, the weight values ​​for each region are preset based on physiological prior knowledge—for example, the cheek region has the richest microvessels, so its weight is set to 1.0; the forehead region is next, so its weight is set to 0.8; the mouth edge is easily affected by facial expressions and movements, so its weight is set to 0.2; and non-skin regions such as the background have a weight of 0.

[0053] The beneficial effects of this feature are reflected in the following two aspects: First, regarding the flexibility of feature utilization, soft masking assigns different continuous weight values ​​to different sub-regions within the effective physiological region, allowing the network to differentiate the retention or attenuation of attention weights based on the actual contribution of each region to the rPPG signal. This scheme "can both achieve physiological guidance and give the network a certain degree of feature self-regulation tolerance." Compared to the "all-or-nothing" blocking method of hard masking, soft masking retains the possibility for the network to make limited use of low-weight regions (such as the skin edge region near the mouth). When this region happens to contain a weak effective signal, the network can still consider it.

[0054] Second, in terms of fine-grained representation of anatomical priors, the continuous weight values ​​of the soft mask can more accurately reflect the differences in the contribution of different facial regions to the rPPG signal. According to medical prior knowledge, the richness of facial microcirculation varies in different regions, and different regions are affected to varying degrees by muscle movements (such as speaking and blinking). The soft mask quantifies these anatomical differences into prior information that the network can directly utilize through preset continuous weight values, rather than simplifying them into a uniform binary judgment of "valid" or "invalid." In this scheme, the weight values ​​of the soft mask originate from anatomical prior knowledge—the cheek region has a weight of 1.0, the forehead 0.8, and the mouth edge 0.2—these specific weight assignments reflect the differences in the reliability of physiological signals in different facial regions, rather than attention weights learned through data-driven methods. The two are fundamentally different in terms of information source and functional localization.

[0055] In summary, this scheme provides a parallel implementation path for "attention computation guided by spatial masks" that differs from hard masks by limiting the continuous weight form of soft masks and their role in the attention weight matrix. This approach gives the network greater flexibility in feature utilization while preserving physiological prior constraints.

[0056] In some embodiments, generating a spatial mask based on the region of interest includes: directly identifying facial skin regions in the video frame using an image semantic segmentation network, and using the identified facial skin regions as the region of interest to generate the spatial mask.

[0057] In the above technical solution, an alternative mask generation scheme to the "key point detection → polygon filling" path is provided: an image semantic segmentation network is used to directly identify the facial skin region in the video frame, and the identified facial skin region is used as the region of interest to generate a spatial mask.

[0058] Compared with the aforementioned scheme (through key point localization → ROI boundary extraction → polygon filling to generate a mask), the essential feature of this scheme is that the spatial mask is generated by directly outputting the facial skin region end-to-end through the image semantic segmentation network, and generating the spatial mask accordingly.

[0059] The beneficial effects of this feature are reflected in the following two aspects: First, regarding the simplicity of the data processing workflow, the semantic segmentation scheme skips three intermediate steps: keypoint detection, ROI boundary extraction, and polygon filling, directly mapping from the original video frame to the spatial mask. This end-to-end approach reduces the accumulated errors in intermediate steps—keypoint localization errors can be amplified by polygon filling, while the semantic segmentation network, through end-to-end training, can learn robustness to local detection noise, still outputting complete skin region segmentation results even when skin boundaries are blurred or there is local occlusion.

[0060] Secondly, regarding applicable scenarios, the semantic segmentation scheme is suitable for extremely low-computing-power scenarios where facial keypoint detection models are difficult to deploy, or scenarios where facial keypoint detection accuracy is insufficient (such as side profiles, large-angle head poses, and partial occlusion). This scheme can employ lightweight networks such as U-Net and BiSeNet based on MobileNetV3, whose inference speeds meet real-time requirements. The output objective of the semantic segmentation network in this scheme—"treating the forehead, cheeks, etc., as one category (high-value rPPG skin regions), and the background, hair, and facial features as another category"—is specifically designed for rPPG physiological signal measurement tasks and differs from the category definitions in general semantic segmentation tasks. This category definition is based on prior physiological knowledge (microvascular distribution and the degree of motion interference), rather than the semantic consistency of visual appearance.

[0061] In summary, this solution provides a parallel implementation path that does not rely on keypoint detection by limiting the semantic segmentation scheme as an alternative to mask generation. While simplifying the data processing flow, it also expands the applicability of this solution in scenarios where keypoint detection models are difficult to deploy or have insufficient detection accuracy.

[0062] According to another aspect of the present invention, a non-contact heart rate estimation system is provided, the system comprising, based on the above-described method: A facial physiological prior extraction subsystem is used to acquire video frames containing human faces, detect and locate key points in the facial regions of the video frames, extract regions of interest (ROIs) located in the forehead, left cheek, right cheek, and chin regions, excluding the eyes, mouth, hair, and background regions, based on the localization results, generate spatial masks based on the ROIs, downsample the spatial masks to align with the feature map resolution of the backbone network, and perform numerical mapping on the downsampled masks, wherein valid physiological regions are mapped to a first value, invalid regions are mapped to a second value, and the second value is activated by the Softmax function to make the corresponding attention weight approach zero; The local sparse attention measurement subsystem is used to perform spatiotemporal block embedding on the video frame sequence to generate a token sequence, guide attention calculation based on the spatial mask to block attention interaction between invalid regions and other tokens, perform local sparse attention calculation only on tokens within the valid physiological region, and perform spatiotemporal feature aggregation and decoding on the feature sequence after multi-layer local sparse attention calculation to output a volumetric pulse wave waveform signal for determining the heart rate estimate.

[0063] In order to better utilize the above method, this application proposes a non-contact heart rate estimation system. Each module corresponds to a step of the above method, and its specific principle has been described above and will not be repeated here.

[0064] According to another aspect of the present invention, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the above-described method.

[0065] In the above technical solution, to better operate and use the method, the method is stored in a computer-readable storage medium and implemented using a processor. It should be noted that the principle and effect of each step have been described above and will not be elaborated upon here. Attached Figure Description

[0066] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0067] Figure 1 This is a schematic diagram of a typical rPPG signal extraction process based on global Top-K sparse attention in existing technologies; Figure 2 This is a flowchart illustrating an embodiment of a non-contact heart rate estimation method according to the present invention; Figure 3 This is a schematic diagram comparing the prior art with the method flow of the present invention, which is an embodiment of the non-contact heart rate estimation method of the present invention. Figure 4 This is a schematic diagram of an embodiment of a non-contact heart rate estimation system according to the present invention; Figure 5 This is a schematic diagram of the system architecture of an embodiment of the non-contact heart rate estimation system of the present invention; Figure 6 This is a schematic diagram of the internal functional modules and data flow of the facial physiological prior extraction subsystem, which is an embodiment of a non-contact heart rate estimation system according to the present invention. Detailed Implementation

[0068] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be particularly noted that the following embodiments are for illustrative purposes only and do not limit the scope of the invention. Similarly, the following embodiments are only some, not all, embodiments of the present invention, and all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0069] Example 1 Please see Figure 2 A non-contact heart rate estimation method, the method comprising: S1. Obtain a video frame sequence containing a face, and perform spatiotemporal block embedding on the video frame sequence to generate a token sequence; For example, a video sequence containing T frames is received. A series of three-dimensional spatiotemporal patches are generated by sliding slices of the consecutive video frames in both spatial and temporal dimensions using a three-dimensional convolutional kernel. Each three-dimensional image patch is then linearly projected into a token sequence with consistent dimensions. At this point, the token sequence contains information about all regions of the entire image.

[0070] S2. Detect and locate key points in the facial region of the video frame. Based on the location results, extract the region of interest located in the forehead, left cheek, right cheek and chin area, excluding the eye, mouth, hair and background areas. Generate a spatial mask based on the region of interest. The spatial mask indicates whether the image area corresponding to each token in the token sequence is a valid physiological area. In this embodiment, generating a spatial mask based on the region of interest includes: generating a binarized image using a polygon filling algorithm based on the boundary key points of the region of interest, with the effective region pixel value being 1 and the invalid region pixel value being 0, and then downsampling the binarized image through a pooling operation.

[0071] For example, a lightweight facial landmark detection model (such as the MediaPipe Face Mesh network) is used to perform dense facial landmark detection on the current video frame, outputting a keypoint set containing multiple 3D coordinates to accurately depict the 3D topological geometry of the face. Based on prior medical knowledge, the areas with the richest facial microcirculation and least affected by facial muscle movement are mainly concentrated on the forehead, left and right cheeks, and chin. Boundary keypoints constituting these four core Regions of Interest (ROIs) are extracted using a predefined keypoint index table, while keypoints in the eyes (blinking generates strong periodic noise), mouth (speaking and breathing introduce deformation noise), hair, and background areas are excluded. A spatial mask is generated based on the extracted ROI boundary keypoints, specifically including: generating a 2D binary image at the original image resolution using a polygon filling algorithm based on the ROI boundary keypoints, where the pixel value of effective physiological areas (forehead, cheeks, etc.) is 1, and the pixel value of irrelevant areas (background, eyes, mouth, hair, etc.) is 0. Specifically: The MediaPipe Face Mesh network is used as the facial landmark detection model. This model is a lightweight real-time face detection method that can output a set of facial landmarks containing 468 3D coordinates, accurately depicting the 3D topological geometry of the face, covering the contour, eyes, mouth, and smooth skin areas, and has good robustness to spatial distortion, appearance distortion, head rotation, and body movement.

[0072] The video stream is input at a frame rate of 30 frames per second, with each frame having an image size of 224×224 pixels. The MediaPipe FaceMesh network performs inference on the current frame and outputs the 3D coordinates of 468 key points. ).

[0073] Based on prior medical knowledge, the areas with the richest facial microcirculation and least affected by facial muscle movement are mainly concentrated on the forehead, left and right cheeks, and chin. This embodiment uses a predefined keypoint index table to extract the boundary keypoints constituting these four core ROIs from 468 keypoints. Specifically: Forehead area: Extract key points corresponding to the forehead area (forehead contour points in the index range of approximately 10-109). Left cheek area: Extract key points corresponding to the left cheek (left cheek contour points within the index range of approximately 123-227). Right cheek area: Extract key points corresponding to the right cheek (right cheek contour points within the index range of approximately 352-397). Chin region: Extract key points corresponding to the chin (chin contour points within the index range of approximately 152-205).

[0074] At the same time, key points in the eye area (index number approximately 33-133, including the left and right eyes), key points in the mouth area (index number approximately 61-291, including the inner and outer lips), hair area, and background area are forcibly excluded.

[0075] Based on the extracted ROI boundary key points, a two-dimensional binarized image is generated using the scan line algorithm (polygon filling algorithm) at the original image resolution (224×224). Pixel values ​​for effective physiological regions (forehead, left and right cheeks, chin) are set to 1, while pixel values ​​for irrelevant regions (background, eyes, mouth, hair, etc.) are set to 0.

[0076] As an alternative, generating the spatial mask based on the region of interest includes: directly identifying facial skin regions in the video frame using an image semantic segmentation network, using the identified facial skin regions as the region of interest, and generating the spatial mask.

[0077] For example, as an alternative to the key point detection and ROI segmentation path in the above scheme, this example uses an image semantic segmentation network to directly generate a spatial mask, skipping the key point detection step.

[0078] Specifically, a lightweight U-Net semantic segmentation network based on MobileNetV3 as the backbone network is adopted. This network is pre-trained on a facial image dataset labeled with "high-value rPPG skin regions", and regions such as the forehead and cheeks are labeled as one category (foreground), while the background, hair, and facial features (eyes, mouth, etc.) are labeled as another category (background).

[0079] In actual inference, the semantic segmentation network directly receives the current video frame as input and outputs a segmentation probability map with the same resolution as the input image. The probability value of each pixel belonging to a "valid physiological region" is between 0 and 1. This probability map is then thresholded (threshold set to 0.5) to generate a binary mask. Pixels with a probability ≥ 0.5 are considered valid physiological regions (set to 1), and pixels with a probability < 0.5 are considered invalid regions (set to 0). Subsequent downsampling and numerical mapping steps are the same. This scheme is suitable for scenarios where end-to-end mask generation is desired, avoiding intermediate steps such as keypoint detection and polygon filling.

[0080] In this example, a lower computing power cost alternative can also be used, as detailed below: As an alternative for devices with extremely low computing power, this example uses traditional computer vision methods to generate spatial masks, completely abandoning deep learning models.

[0081] Specifically, firstly, a cascaded classifier based on Haar features or a face detector based on HOG features and SVM is used to roughly define the bounding box of the face in the current video frame. Then, the image within this face region is converted from the RGB color space to the YCrCb or HSV color space. In the YCrCb color space, the Cr and Cb components of skin color typically fall within specific ranges (e.g., 133 ≤ Cr ≤ 173, 77 ≤ Cb ≤ 127). Based on this skin color prior, skin color thresholding is performed on the bounded face region, marking pixels that conform to the skin color range as 1 (valid region) and pixels that do not conform to the skin color range (e.g., eyes, mouth, hair, background, etc.) as 0 (invalid region). To further improve segmentation accuracy, a Gaussian mixture model (GMM) can be used to model the skin color distribution, determining the probability of each pixel belonging to the skin color using a soft probability approach. After generating the binarized mask, the subsequent downsampling and numerical mapping steps are the same. This solution is suitable for edge devices with extremely limited computing power that cannot run any deep learning models (including MediaPipe).

[0082] As an alternative to relying on specific hardware, spatial masks can also be obtained directly using multimodal sensors, without the need for calculations from RGB images. Specifically: For devices equipped with infrared (IR) or depth (ToF) sensors, the foreground face can be directly separated from the background based on depth information: continuous areas closer to the camera in the depth image are considered the foreground (face), and farther areas are considered the background, thus generating a binary foreground mask. Alternatively, a thermal imaging camera can be used to identify facial temperature distribution: since the temperature distribution in the skin area is relatively uniform and there is a temperature difference with the background environment, a facial region mask can be generated by segmentation using a temperature threshold. The mask generated in the above methods can be directly used as an indicator of the physiologically valid area without any calculations based on RGB images. This solution is suitable for terminal devices with multimodal sensing capabilities (such as some smartphones, in-vehicle camera systems, etc.).

[0083] As an alternative, the spatial mask is a soft mask containing continuous probability weights, with the weight values ​​of each region preset according to physiological prior knowledge. The soft mask is applied to the attention weight matrix calculated by self-attention through element-wise multiplication.

[0084] As an alternative to hard masks (effective regions are mapped to 0 and invalid regions are mapped to -∞), this embodiment uses soft masks with continuous probability weights to guide attention computation.

[0085] Specifically, based on prior physiological knowledge, different facial regions contribute differently to the rPPG signal: the cheek region has the richest microvessels and thinnest skin, resulting in the highest signal intensity; the forehead region is next; while the edge regions near the mouth and eyes, although skin areas, are easily affected by facial expressions, leading to relatively low signal reliability. Based on this prior knowledge, this embodiment assigns continuous weight values ​​to each region based on the ROI segmentation in step 5.1.2: the cheek region is weighted at 1.0, the forehead region at 0.8, the chin region at 0.7, the edge skin regions near the mouth and eyes at 0.2~0.5, and non-skin regions such as the background, hair, eyes, and inside the mouth are weighted at 0.

[0086] The soft mask is generated as follows: after generating a binarized image based on the key points of the ROI boundary, the binarized image is not directly used as a mask. Instead, the pixels in each region are assigned values ​​according to the aforementioned physiological prior weights to generate a continuous probability weight map with the same resolution as the ROI. Then, the map is downsampled to be aligned with the resolution of the Token sequence feature map through pooling operations to obtain the soft mask tensor M_soft.

[0087] In self-attention computation, the soft mask does not use "overlay to QK". Instead of using the "" method, it applies the attention weight matrix calculated by self-attention through element-wise multiplication:

[0088] Here, ⊙ represents element-wise multiplication. For regions with a weight value of 0 (background, facial features, etc.), their attention weights are cleared to zero, equivalent to complete blocking by a hard mask. For skin regions with weight values ​​of 0.2 to 1.0, their attention weights are preserved or attenuated to varying degrees. This approach achieves both physiological prior guidance and provides the network with a certain degree of feature self-regulation tolerance—when a low-weight region (such as the edge of the mouth) happens to contain a weak but effective signal, the network can still utilize it to a limited extent, rather than completely discarding it as in a hard mask. This scheme is suitable for scenarios where it is desirable to retain a certain degree of network adaptability while still requiring soft constraints based on physiological priors.

[0089] S3. Downsample the spatial mask to align with the feature map resolution of the token sequence, and perform numerical mapping on the downsampled mask, wherein the effective physiological region is mapped to the first value and the invalid region is mapped to the second value; after the second value is activated by the Softmax function, the corresponding attention weight approaches zero. For example, the high-resolution binarized mask is downsampled using a pooling operation (max pooling or average pooling) to align with the feature map resolution (H / P × W / P, where P is the patch size) of the backbone network's token sequence. The scaled mask matrix is ​​flattened into a one-dimensional vector and numerically mapped: valid regions are mapped to a first value (e.g., 0), and invalid regions are mapped to a second value (e.g., -1e9). After activation by the Softmax function, the natural exponent of this second value approaches zero. This forces the corresponding attention weights to zero. Specifically: Assuming the backbone network has a token space dimension of 14×14 after patch embedding (i.e., patch size P=16, 224 / 16=14), this module performs max pooling on the aforementioned 224×224 binarized mask to reduce it to a 14×14 resolution. Specifically, max pooling with a kernel size of 16×16 and a stride of 16 is used, and the maximum value within each 16×16 pixel block (i.e., whether there is a valid physiological region) is used as the mask value for that block.

[0090] The scaled 14×14 mask matrix is ​​flattened into a one-dimensional vector (length 196) and then numerically mapped: valid regions with values ​​of 1 are mapped to the constant 0, and invalid regions with values ​​of 0 are mapped to the minimum value -1e9. The resulting mask tensor is denoted as... .

[0091] S4. Based on the spatial mask, guide attention calculation to block attention interaction between invalid regions and other tokens, and perform local sparse attention calculation only on tokens within the valid physiological region; In this embodiment, the attention calculation guided by the spatial mask includes: The token sequence is linearly transformed into a query matrix, a key matrix, and a value matrix, and the mapped spatial mask is superimposed on the scaled dot product matrix during self-attention computation.

[0092] In this embodiment, the attention weight sorting operation for the global tokens is not performed in the local sparse attention calculation; For example, the token sequence is transformed linearly to generate a query matrix Q, a key matrix K, and a value matrix V. In the self-attention computation, the spatial mask after numerical mapping is superimposed onto the scaled dot product matrix. Up (i.e.) Since the invalid regions correspond to -∞ values ​​in the mask, the attention weights in these regions are forced to zero after the Softmax activation function. Therefore, the network only needs to perform local information interaction within the effective physiological region indicated by the mask, completely eliminating the need for global sorting operations. In this step, the local sparse attention computation does not perform the operation of sorting global tokens by attention weight; the range of the effective physiological region is directly preset by the spatial mask. Specifically: The backbone network adopts an rPPG measurement network based on the Vision Transformer architecture.

[0093] First, a video sequence containing T frames (e.g., T=64) is received. A 3D convolutional kernel (5×7×7 kernel size, spanning 5 frames temporally and 7×7 pixels spatially) is used to slide slices in both the spatial and temporal dimensions, generating a series of spatiotemporal patches. Each patch is linearly projected into a token of dimension D=128, forming a token sequence. Its dimensions are (T×N, D), where N is the number of patches per frame (for a 224×224 input, a patch size of 16×16, N=14×14=196).

[0094] Will The query matrix Q, key matrix K, and value matrix V are generated through three independent linear transformations, each with dimensions (T×N, D).

[0095] In self-attention computation, the physiological prior mask generated in Example 1 is used. (Dimensions T×N) are superimposed onto a scaled dot product matrix. The mathematical formula is as follows:

[0096] in, for and The dimension (d=128). For invalid regions (background, eyes, mouth, hair, etc.), its dimension is... The corresponding value is -1e9. After the Softmax activation function, the attention weights in these regions are forced to zero. The network only needs to perform local information interaction within the effective skin area with a mask of 0, completely eliminating the need for global sorting operations. In actual testing, this scheme reduced the FLOPs of the core attention module by approximately 70%.

[0097] As an optional implementation, the attention calculation guided by the spatial mask includes: The spatial mask is convolutionally encoded and used as a query vector. The token sequence is linearly transformed and used as a key vector and a value vector. Physiologically relevant features that match the spatial mask are retrieved from the token sequence through a cross-attention mechanism.

[0098] For example, this embodiment uses a cross-attention mechanism to guide the network to focus on effective physiological regions, rather than superimposing a mask in the Softmax step of self-attention.

[0099] Specifically, the aforementioned generated physiological prior mask (14×14 dimensional) After passing through a convolutional encoding layer (3×3 kernel size, 128 output channels), it is used as the query vector. The global spatiotemporal characteristics of the backbone network (Dimensions T×N, D) After a linear transformation, they become the key vector K and value vector V. Calculated using a cross-attention mechanism:

[0100] This cross-attention layer forces the network to "retrieve" features from the global features that highly match the physiological prior mask. Since the mask has pre-identified the locations of effective physiological regions, the calculation results of cross-attention naturally focus on areas such as the forehead and cheeks, thus eliminating the need for global sorting. This approach is suitable for scenarios where it is desirable to retain the original self-attention mechanism while adding an independent module to achieve physiological guidance.

[0101] As another optional implementation, the attention calculation based on the spatial mask includes: Before the token sequence is linearly transformed into a query matrix, a key matrix, and a value matrix, the spatial mask is multiplied element-wise with the token sequence to clear the token feature values ​​corresponding to invalid regions to zero.

[0102] For example, this example moves the application of the mask forward to the token input stage.

[0103] Specifically, video image patches are mapped to token sequences. Then, before performing Q, K, V linear projection, directly apply the generated mask. (Downsampled to 14×14 and flattened into a vector of length 196) and the Token sequence Perform element-wise multiplication:

[0104] In this scheme, the mask value for valid physiological regions is 1, and the token feature value remains unchanged; the mask value for invalid regions (background, eyes, mouth, hair, etc.) is 0, and the token feature value is directly cleared to zero. Thus, in subsequent self-attention calculations, tokens with cleared values ​​will automatically lose their responsiveness, achieving the same goal of skipping the sorting and reducing computational power. This scheme is suitable for scenarios where invalid information needs to be removed in the early stages of the network.

[0105] S5. Perform spatiotemporal feature aggregation and decoding on the feature sequence after multi-layer local sparse attention calculation, and output the volumetric pulse wave waveform signal. For example, the feature sequence processed by multiple local sparse attention modules is transformed in the spatial dimension by a fully connected feedforward network (FFN), and then compressed into a one-dimensional time series signal, i.e., the predicted rPPG waveform, by a one-dimensional convolutional (1D-CNN) decoder in the temporal dimension.

[0106] S6. Determine the heart rate estimate based on the volumetric pulse wave waveform signal.

[0107] The estimated heart rate is determined based on the volumetric pulse wave waveform signal. Specifically, the heart rate value (BPM) within this time window can be calculated using bandpass filtering and peak-to-peak detection or Fast Fourier Transform (FFT).

[0108] Please see Figure 3 Compared with existing video physiological measurement methods that mainly rely on Transformer networks for global attention calculation and data-driven Top-K screening, this invention has the following significant advantages: (1) Significantly reduced computational complexity. Existing technologies, when selecting features, must first calculate the cross-correlation matrix of all tokens in the entire image (time complexity O(N²)) and then perform a global sorting operation (time complexity O(N×log₂N)). This invention extracts high-value physiological regions (ROIs) in advance using a lightweight facial keypoint model, and directly superimposes polygon masks into the attention calculation, directly "blocking" attention interactions in non-facial regions. This allows the network to extract local information only within the effective skin area, which accounts for less than 30% of the area, and completely eliminates the energy-intensive sorting step. Evaluation shows that for the core attention module, this invention can reduce the number of floating-point operations (FLOPs) by approximately 60% to 80%.

[0109] (2) Significantly improves the friendliness of hardware deployment on the edge and device sides. The dynamic sorting and dynamic memory allocation operations in the existing Top-K mechanism are extremely difficult to accelerate on the NPU of smartphones or the edge computing chips in vehicles, often becoming a serious bottleneck for network inference. This invention transforms the complex dynamic filtering operation into mask addition and mask covering operations for fixed-dimensional matrices. This static tensor operation is extremely friendly to modern mobile AI acceleration chips (NPU / DSP) and can be optimized by the underlying hardware instruction set. Therefore, this invention can significantly improve the real-time inference frame rate of mobile devices (from less than 15 frames / second to more than 30 frames / second in full-frame real-time operation on ordinary smartphones).

[0110] (3) Strong anatomical constraints are introduced to improve the model's anti-interference ability and measurement fidelity. Existing technologies use a purely data-driven "black box" screening method. When there are strong ambient light flickers, background screen flickers, or the subject blinks or talks frequently in the video, the network is easily "deceived" by these non-physiological periodic noises. This invention directly solidifies the physiological prior knowledge of rPPG in the medical field into an algorithm mask. The network is forcibly anchored to the pure skin areas such as the forehead and cheeks, which are richest in microvessels and are not easily affected by muscle deformation. This mechanism cuts off the noise input source, making the heartbeat waveform more faithful and the prediction accuracy more robust than existing technologies in challenging scenarios such as head movement, partial occlusion, or drastic changes in ambient light.

[0111] (4) Improved network training convergence speed and feature extraction purity. Existing technologies require a large number of rounds in the early stages of model training to allow the network to explore the location of the heartbeat signal, resulting in slow convergence. This invention directly provides the network with a spatial reference answer, decoupling the two major tasks of "spatial localization" and "temporal signal extraction". The backbone Transformer network no longer needs to be distracted by learning how to locate the face, but instead focuses all network parameters on how to extract extremely weak temporal micro-jumps in skin color from the target region, significantly accelerating the convergence speed during the training phase and improving the purity of the final extracted physiological features.

[0112] Example 2 Please see Figure 4 This invention discloses a non-contact heart rate estimation system based on the method described in one of the embodiments. It proposes an efficient, low-latency non-contact video heart rate estimation architecture for mobile devices and edge computing. Please refer to the appendix. Figure 5 The figure shows the overall system architecture flowchart of this invention. This system abandons the energy-intensive global attention and Top-K ranking mechanism in traditional Transformer networks, and mainly includes two core subsystems: a facial physiological prior extraction subsystem and a local sparse attention measurement subsystem.

[0113] The core task of the facial physiological prior extraction subsystem is to use extremely lightweight facial feature tracking technology to locate and extract regions rich in microvascular pulsation information from the original video frames in real time, and convert them into spatial binary / continuous masks that can be directly used by neural networks.

[0114] The local sparse attention measurement subsystem is the backbone inference network of this invention. It receives the video frame sequence and the spatial mask generated by the previous subsystem, and uses the mask to directly block tokens from irrelevant background and noise regions, strictly limiting self-attention computation to effective local physiological regions. Finally, it outputs a high-precision volumetric pulse wave (rPPG) signal and heart rate value. The internal structure and implementation methods of the two subsystems will be described in detail below.

[0115] 1. A facial physiological prior extraction subsystem is used to acquire video frames containing human faces, detect and locate key points in the facial regions of the video frames, extract interest regions located in the forehead, left cheek, right cheek and chin regions, excluding the eyes, mouth, hair and background regions, based on the localization results, generate spatial masks based on the interest regions, downsample the spatial masks to align with the feature map resolution of the backbone network, and perform numerical mapping on the downsampled masks, wherein the effective physiological regions are mapped to the first value and the invalid regions are mapped to the second value, and the second value is activated by the Softmax function to make the corresponding attention weight approach zero; Please see Figure 6 This figure details the internal functional modules and data flow of the facial physiological prior extraction subsystem. This subsystem operates with extremely low CPU / NPU overhead, providing "anatomical navigation" for subsequent attention mechanisms. Internally, it contains the following functional modules that execute sequentially: (1) Face detection and dense key point localization module Function: This module is fundamental for generating accurate physiological masks. It takes the current frame of the video stream as input, quickly locates the face, and outputs a high-density facial landmark.

[0116] Implementation method: A lightweight MediaPipe Face Mesh network (or an equivalent lightweight keypoint tracking model) is used. This model can output a keypoint set containing 468 or more 3D coordinates in real time, accurately depicting the 3D topological geometry of the face, covering the contours, eyes, mouth, and smooth skin areas.

[0117] (2) Dynamic division module of physiological high-value ROI (Region of Interest) Function: In a dense cluster of key points, filter out the region with the highest rPPG signal-to-noise ratio and remove non-skin regions that are highly susceptible to motion noise.

[0118] Implementation Method: Based on prior medical knowledge, the areas with the richest facial microcirculation and least affected by facial muscle movement are mainly concentrated on the forehead, left and right cheeks, and chin. This module extracts the boundary keypoints that constitute these four core ROIs using a predefined keypoint index table. Simultaneously, it forcibly excludes keypoints from the eyes (blinking generates extremely strong periodic noise), the mouth (speaking and breathing introduce deformation noise), hair, and the background area.

[0119] (3) Sparse attention control mask generation module Function: This module is the key technology connecting the front-end and back-end of this invention. It is responsible for downsampling and aligning irregular polygonal ROI regions to the resolution of the Transformer backbone network feature map, generating one or more spatial control mask maps to block irrelevant attention calculations.

[0120] Implementation method: 1) Polygon Filling and Binarization: Based on the ROI boundary key points extracted in step 5.1.2, a two-dimensional binarized image is generated using a polygon filling algorithm (scan line algorithm) at the original image resolution. The effective physiological region (cheek) has a pixel value of 1, while irrelevant regions (background, eyes, etc.) have a pixel value of 0.

[0121] 2) Grid alignment and pooling downsampling: Assuming that the token space dimension after patch embedding in the subsequent backbone network is H / P x W / P (where P is the size of the patch), this module performs max pooling or average pooling operations on the high-resolution binarized mask to reduce it to a resolution of H / P x W / P.

[0122] 3) Mask Tensor Construction: The scaled mask matrix is ​​flattened into a one-dimensional vector and a logical mapping is performed: valid regions with values ​​of 1 are mapped to the constant 0, and invalid regions with values ​​of 0 are mapped to the minimum value (-1e9). The generated mask tensor is denoted as... It serves as a bridge input to the local sparse attention measurement subsystem.

[0123] 2. A local sparse attention measurement subsystem is used to perform spatiotemporal block embedding on the video frame sequence to generate a token sequence, guide attention calculation based on the spatial mask to block attention interaction between invalid regions and other tokens, perform local sparse attention calculation only on tokens within the valid physiological region, and perform spatiotemporal feature aggregation and decoding on the feature sequence after multi-layer local sparse attention calculation to output a volumetric pulse wave waveform signal for determining the heart rate estimate.

[0124] Please see Figure 5 This figure details the network architecture of the locally sparse attention measurement subsystem. This invention represents a revolutionary hardware-friendly transformation of the traditional Vision Transformer architecture.

[0125] (1) Video Spatiotemporal Feature Segmentation (Patch Embedding) Module Function: Converts a continuous sequence of video frames into a sequence of word vectors (tokens) that can be processed by a neural network.

[0126] Implementation method: Receive a video sequence containing T frames, perform sliding slicing in the spatial and temporal dimensions using a 3D convolution kernel to generate a series of spatiotemporal patches, and linearly project these patches into a token sequence of dimension D, denoted as . .at this time, It contains all the information for the entire map.

[0127] (2) Mask-guided local sparse attention computation module Function: Utilizes a mask generated by the front-end system It directly implements local sparse attention computation, completely replacing the global query-key similarity computation and time-consuming Top-K sorting operation in the existing technology (RhythmFormer).

[0128] Implementation method: 1) The query matrix Q, key matrix K, and value matrix V are generated by linear transformation.

[0129] 2) Attention blocking and mask fusion: When calculating the attention weight distribution, the generated physiological prior mask is used. This is superimposed on the scaled dot product matrix of the self-attention layer. The mathematical formula is as follows:

[0130] 3) Analysis of Technical Principles and Computational Advantages: In the above formula, for non-facial areas or noise areas (background, eyes), its... The corresponding value in is After the Softmax activation function, the attention weights in these regions are forced to zero. ).

[0131] 4) Completely skipping sorting: Traditional Top-K methods require first calculating the complete N x N attention matrix, and then globally sorting the N results to select high-weight regions. In this invention, however, the network is... The "hardwired" connection reaches the effective skin area. The network only needs to perform local information exchange within the area with a mask of 0, completely eliminating the need for global sorting operations and significantly reducing the memory scheduling overhead and floating-point operations (FLOPs) of the mobile NPU / GPU.

[0132] (3) Spatiotemporal feature aggregation and heart rate regression module, which is used to perform spatiotemporal feature aggregation and decoding on the feature sequence after multi-layer local sparse attention calculation, and output volume pulse wave waveform signal to determine the heart rate estimate.

[0133] The pure features extracted by local sparse attention are aggregated in time and decoded into the final physiological signal.

[0134] Implementation method: After the feature sequence is processed by multiple layers of local sparse attention modules, the high-dimensional features are compressed into a one-dimensional time series signal, i.e., the predicted rPPG waveform, through a fully connected feedforward network (FFN) and a one-dimensional convolutional (1D-CNN) decoder in the time dimension. Finally, the heart rate value (BPM) within this time window is calculated by bandpass filtering and peak detection (Peak-to-Peak) or Fast Fourier Transform (FFT).

[0135] In order to better utilize the above method, this application proposes a non-contact heart rate estimation system. Each module corresponds to a step of the above method, and its specific principle has been described above and will not be repeated here.

[0136] Example 3 A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in one of the embodiments.

[0137] In this embodiment, to better operate and use the method described in one of the embodiments, the above method is stored in a computer-readable storage medium, and the above method is implemented using a processor. It should be noted that the principle and effect of each step have been described above and will not be elaborated further here.

[0138] The above description is only a part of the embodiments of the present invention and does not limit the scope of protection of the present invention. Any equivalent device or equivalent process transformation made based on the content of the present invention specification and drawings, or direct or indirect application in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A non-contact heart rate estimation method, characterized in that, The method includes: Obtain a video frame sequence containing a face, and perform spatiotemporal block embedding on the video frame sequence to generate a token sequence; The facial regions in the video frames are detected and key points are located. Based on the location results, regions of interest are extracted from the forehead, left cheek, right cheek and chin areas, excluding the eyes, mouth, hair and background areas. A spatial mask is generated based on the regions of interest. The spatial mask indicates whether the image area corresponding to each token in the token sequence is a valid physiological area. The spatial mask is downsampled to align with the feature map resolution of the token sequence, and the downsampled mask is numerically mapped, wherein the effective physiological region is mapped to the first value and the invalid region is mapped to the second value; after the second value is activated by the Softmax function, the corresponding attention weight approaches zero; Attention computation is guided by the spatial mask to block attention interaction between invalid regions and other tokens, and local sparse attention computation is performed only on tokens within the valid physiological region. The feature sequence after multi-layer local sparse attention computation is spatiotemporally aggregated and decoded to output a volumetric pulse wave waveform signal. The heart rate estimate is determined based on the volumetric pulse wave waveform signal.

2. The non-contact heart rate estimation method as described in claim 1, characterized in that, In the local sparse attention computation, no attention weight sorting operation is performed on the global tokens.

3. The non-contact heart rate estimation method as described in claim 1, characterized in that, The attention calculation guided by the spatial mask includes: The token sequence is linearly transformed into a query matrix, a key matrix, and a value matrix, and the mapped spatial mask is superimposed on the scaled dot product matrix during self-attention computation.

4. The non-contact heart rate estimation method as described in claim 1, characterized in that, The attention calculation guided by the spatial mask includes: The spatial mask is convolutionally encoded and used as a query vector. The token sequence is linearly transformed and used as a key vector and a value vector. Physiologically relevant features that match the spatial mask are retrieved from the token sequence through a cross-attention mechanism.

5. The non-contact heart rate estimation method as described in claim 1, characterized in that, The attention calculation guided by the spatial mask includes: Before the token sequence is linearly transformed into a query matrix, a key matrix, and a value matrix, the spatial mask is multiplied element-wise with the token sequence to clear the token feature values ​​corresponding to invalid regions to zero.

6. The non-contact heart rate estimation method as described in claim 1, characterized in that, Generating a spatial mask based on the region of interest includes: generating a binarized image using a polygon filling algorithm based on the boundary key points of the region of interest, with the effective region pixel value being 1 and the invalid region pixel value being 0, and then downsampling the binarized image through a pooling operation.

7. The non-contact heart rate estimation method as described in claim 1, characterized in that, The spatial mask is a soft mask containing continuous probability weights. The weight values ​​of each region are preset based on physiological prior knowledge. The soft mask is applied to the attention weight matrix calculated by self-attention through element-wise multiplication.

8. The non-contact heart rate estimation method as described in claim 1, characterized in that, The step of generating a spatial mask based on the region of interest includes: directly identifying facial skin regions in the video frame using an image semantic segmentation network, using the identified facial skin regions as the region of interest, and generating the spatial mask.

9. A non-contact heart rate estimation system, characterized in that, Based on the method of any one of claims 1 to 8, the system comprises: A facial physiological prior extraction subsystem is used to acquire video frames containing human faces, detect and locate key points in the facial regions of the video frames, extract regions of interest (ROIs) located in the forehead, left cheek, right cheek, and chin regions, excluding the eyes, mouth, hair, and background regions, based on the localization results, generate spatial masks based on the ROIs, downsample the spatial masks to align with the feature map resolution of the backbone network, and perform numerical mapping on the downsampled masks, wherein valid physiological regions are mapped to a first value, invalid regions are mapped to a second value, and the second value is activated by the Softmax function to make the corresponding attention weight approach zero; The local sparse attention measurement subsystem is used to perform spatiotemporal block embedding on the video frame sequence to generate a token sequence, guide attention calculation based on the spatial mask to block attention interaction between invalid regions and other tokens, perform local sparse attention calculation only on tokens within the valid physiological region, and perform spatiotemporal feature aggregation and decoding on the feature sequence after multi-layer local sparse attention calculation to output a volumetric pulse wave waveform signal for determining the heart rate estimate.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 8.

Images