Heartbeat detection method, device, equipment, medium and product based on fusion of video and millimeter wave radar

Abstract

The invention discloses a heartbeat detection method and device based on video and millimeter wave radar fusion, equipment, a medium and a product, and relates to the field of heartbeat detection. The method comprises the following steps: acquiring multi-modal sensor data; performing video modal heartbeat feature extraction according to the face video data stream based on a space attention mechanism to obtain a video heartbeat feature embedded vector; then heartbeat feature extraction is carried out, weighted fusion processing is carried out based on an attention mechanism guided by a video, and a radar heartbeat feature embedded vector is obtained; dimension normalization processing is carried out on the video heartbeat feature embedded vector and the radar heartbeat feature embedded vector, an environment feature vector is obtained through global average pooling, scene division is carried out based on a scene classifier, and a scene classification result is obtained; and after determining a multi-modal fusion weight according to the environment feature vector and a scene classification result, carrying out heartbeat signal reconstruction to obtain a heartbeat signal sequence. The objective of the invention is to improve robustness and detection precision.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of heartbeat detection, and in particular to a heartbeat detection method and device based on video and millimeter wave radar fusion, equipment, medium and product. BACKGROUND

[0002] In the field of modern medical health monitoring, heart rate as a core physiological indicator reflecting the cardiovascular function state of the human body, its accurate and real-time monitoring plays a crucial role in disease prevention, health management and emergency medical care. Traditional contact heart rate monitoring methods, such as electrocardiogram (ECG), pulse oximeter, etc., although have high detection accuracy, but need to directly contact the human skin or wear special equipment, in long-term monitoring, large-scale screening, special environment application and other scenes, there are many limitations. These limitations include low user acceptance, cross-infection risk, high equipment maintenance cost and interference to the behavior of the monitored object, etc. Therefore, developing a technical solution that can accurately obtain heartbeat signals without contacting the human body has become an urgent need in the fields of intelligent medical treatment, remote health monitoring, driver fatigue detection, emotional computing, etc.

[0003] Remote photoplethysmography (rPPG) as one of the most mature non-contact heartbeat detection technologies, its basic principle is to use ordinary camera equipment to capture video sequences of human exposed skin area (usually face), through analyzing the weak skin color fluctuations caused by the changes of hemoglobin concentration in the blood circulation process to extract heartbeat signals. Specifically, when the heart contracts, the blood volume in the arterial blood vessels increases, causing the amount of light absorption of the skin at a specific wavelength to change, and then producing periodic intensity changes in the RGB color channel. By analyzing the color changes in time or frequency domain, the physiological signals corresponding to the heart rate can be extracted.

[0004] However, the rPPG technology faces serious environmental sensitivity problems in practical applications, which mainly stems from its strong dependence on optical imaging quality. First, changes in environmental lighting conditions will directly affect the imaging effect of the camera and the signal-to-noise ratio of the color channel. In a low-illumination environment, image noise increases, and the weak color changes of the heartbeat signal are easily overwhelmed by noise; while in strong light or uneven lighting conditions, phenomena such as overexposure, shadows, etc. will cause color information distortion, making it difficult to accurately extract heartbeat features. Second, the head movement, facial expression changes, blinking, and other behaviors of the measured object will introduce serious motion artifacts, and the spectral components of these artifacts often overlap with the real heartbeat signal, making it difficult to remove them through simple filtering methods. In addition, there are significant differences in the reflection and absorption characteristics of light by people of different skin colors, resulting in unstable detection performance of the rPPG technology in specific populations and limited generalization ability of the algorithm.

[0005] The existing technical development bottlenecks mainly manifest in three aspects: first, insufficient environmental adaptability, single sensor technology is difficult to cope with complex and variable practical application environments, and the performance sharply decreases in common scenes such as changes in lighting conditions, target motion interference, etc.; second, lack of system reliability, due to the lack of effective fault tolerance mechanism, the failure of any one sensor may cause the entire detection system to fail to work normally; third, low utilization efficiency of information resources, the existing scheme fails to establish an effective multi-modal information fusion architecture, and cannot fully exert the complementary advantages and synergistic effect of different sensing technologies.

[0006] Therefore, how to improve robustness and detection accuracy is crucial. SUMMARY

[0007] The purpose of the present application is to provide a heartbeat detection method, device, equipment, medium and product based on video and millimeter wave radar fusion, which can improve robustness and detection accuracy.

[0008] To achieve the above-mentioned purpose, the present application provides the following solutions:

[0009] In a first aspect, the present application provides a heartbeat detection method based on video and millimeter wave radar fusion, comprising:

[0010] Obtaining multi-modal sensor data; the multi-modal sensor data includes face video data stream collected based on a video acquisition module and distance perception video sequence containing human chest region, and distance-time heat map and phase information matrix obtained based on a millimeter wave radar acquisition module;

[0011] Based on a spatial attention mechanism, video modal heartbeat feature extraction is performed according to the face video data stream to obtain a video heartbeat feature embedding vector;

[0012] perform heartbeat feature extraction on the distance perception video sequence containing the human chest region, the distance-time heat map and the phase information matrix, and perform weighted fusion processing based on a video guided attention mechanism to obtain a radar heartbeat feature embedding vector;

[0013] perform dimension normalization processing on the video heartbeat feature embedding vector and the radar heartbeat feature embedding vector, obtain an environment feature vector through global average pooling, and perform scene division based on a scene classifier to obtain a scene classification result;

[0014] determine a multi-modal fusion weight based on the environment feature vector and the scene classification result;

[0015] perform heartbeat signal reconstruction based on the video heartbeat feature embedding vector, the radar heartbeat feature embedding vector, the environment feature vector and the multi-modal fusion weight to obtain a heartbeat signal sequence.

[0016] In an embodiment, video modal heartbeat feature extraction is performed based on a spatial attention mechanism and based on the face video data stream to obtain a video heartbeat feature embedding vector, specifically including:

[0017] spatiotemporal feature encoding is performed on the face video data stream using a three-dimensional convolutional neural network, and color change patterns of skin regions are extracted through multi-layer convolutional operations; wherein the three-dimensional convolutional kernel of the three-dimensional convolutional neural network slides in both the time dimension and the space dimension to capture periodic color change characteristics caused by heartbeat;

[0018] based on a spatial attention mechanism of a Transformer architecture, feature weight distribution based on pixel positions is performed based on the color change patterns and attention weighting processing is performed to obtain the video heartbeat feature embedding vector;

[0019] The attention weight calculation formula corresponding to the spatial attention mechanism is:

[0020] A spatial (i,j)=sigmoid(Transformer(F spatial (i,j)));

[0021] wherein, A spatial (i,j) is the spatial attention weight of position (i,j); F spatial (i,j) is the feature vector of position (i,j); Transformer(·) is the transformation operation of the Transformer encoder; sigmoid(·) is an activation function.

[0022] In an embodiment, heartbeat feature extraction is performed on the distance perception video sequence containing the human chest region, the distance-time heat map, and the phase information matrix, and a video-guided attention mechanism is used for weighted fusion processing to obtain a radar heartbeat feature embedding vector, specifically including:

[0023] The distance perception video sequence containing the human chest region is analyzed and determined for human spatial position, and a three-dimensional convolutional network is used to extract a global scene feature vector;

[0024] A two-dimensional convolutional analysis is performed on the distance-time heat map to extract the distribution characteristics of the radar signal in the space-time domain;

[0025] A two-dimensional convolutional neural network is used to perform phase modulation pattern recognition on the phase information matrix to obtain a recognition result;

[0026] Based on the video-guided attention mechanism, the global scene feature and the distance-time feature are fused and processed by attention weighting according to the global scene feature vector, the distribution characteristics of the radar signal in the space-time domain, and the recognition result to obtain the radar heartbeat feature embedding vector;

[0027] The calculation formula of the dynamic attention weight corresponding to the video-guided attention mechanism is:

[0028] W attention (r,t)=σ(FC global (F global )+FC range (F range-time ));

[0029] Where W attention (r,t) is the attention weight of position r and time t; FC global (·) and FC range (·) are the corresponding full connection mapping functions; σ(·) is the sigmoid activation function; F global is the global scene feature vector; F range-time is the distribution characteristics of the radar signal in the space-time domain.

[0030] In an embodiment, the video heartbeat feature embedding vector and the radar heartbeat feature embedding vector are subjected to dimension normalization processing, and an environmental feature vector is obtained through global average pooling, and scene classification is performed based on a scene classifier to obtain a scene classification result, specifically including:

[0031] dimension normalization is performed on the video heartbeat feature embedding vector and the radar heartbeat feature embedding vector; wherein the video heartbeat feature embedding vector is reduced in dimension to a time sequence feature; the radar heartbeat feature embedding vector is converted from a distance-time matrix to a set standard time sequence feature;

[0032] A one-dimensional convolutional neural network is used to analyze the obtained normalized time sequence feature, extract a feature mode reflecting environmental quality, capture the change rule in the time dimension through multi-layer convolution operation, and obtain an environmental feature vector through global average pooling;

[0033] Scene classification is performed based on a scene classifier to obtain a scene classification result; the scene classifier uses a softmax function.

[0034] In an embodiment, a multi-modal fusion weight is determined according to the environmental feature vector and the scene classification result, specifically including:

[0035] In the video mode, according to the environmental feature vector and the scene classification result, the formula:

[0036] ω rPPG = softmax(FC weight ([F environment ,S category ])1);

[0037] is used to determine the corresponding fusion weight.

[0038] In the radar mode, according to the environmental feature vector and the scene classification result, the formula:

[0039] ω radar = softmax(FC weight ([F environment ,S category ])2);

[0040] is used to determine the corresponding fusion weight.

[0041] Wherein, FC weight (·) is a full connection network for weight calculation; [·,·] is a feature splicing operation; FC weight is a learnable parameter of the full connection network for weight calculation; [·,·]1 is the weight output of the video mode; [·,·]2 is the weight output of the radar mode; F environment is an environmental feature vector; S category is a scene classification result; ω rPPG is the fusion weight corresponding to the video mode; ω radar is the fusion weight corresponding to the radar mode.

[0042] In an embodiment, based on the video heartbeat feature embedding vector, the radar heartbeat feature embedding vector, the environment feature vector, and the multi-modal fusion weight, heartbeat signal reconstruction is performed to obtain a heartbeat signal sequence, specifically including:

[0043] The video heartbeat feature embedding vector and the radar heartbeat feature embedding vector are dimensionally aligned and time-synchronously synchronized, a weighted fusion strategy is adopted, the environment feature vector is extended to the time dimension as global context information, and fusion processing is performed based on the multi-modal fusion weight to obtain a fused feature; the weighted fusion strategy is adopted for fusion processing, and the corresponding expression is:

[0044] F fsion =ω rPPG ·T rPPG +ω radar ·T radar +α·F environment ;

[0045] Wherein, F fusion is the fused feature; alpha is the contribution coefficient of the environment feature; omega rPPG is the fusion weight corresponding to the video modality; omega radar is the fusion weight corresponding to the radar modality; F environment is the environment feature vector; T rPPG is the time sequence feature obtained by dimension reduction of the video heartbeat feature embedding vector; T radar is the set standard time sequence feature obtained by converting the radar heartbeat feature embedding vector from the distance-time matrix;

[0046] The up-sampling and linear interpolation method is adopted to map the fused feature into a heartbeat signal sequence of a set standard length.

[0047] In a second aspect, the present application provides a heartbeat detection device based on video and millimeter wave radar fusion, comprising:

[0048] A data acquisition module is configured to acquire multi-modal sensor data, wherein the multi-modal sensor data includes face video data stream acquired based on a video acquisition module and distance perception video sequence containing human chest region, and distance-time heat map and phase information matrix acquired based on a millimeter wave radar acquisition module;

[0049] A video modality heartbeat feature extraction module is configured to perform video modality heartbeat feature extraction based on a spatial attention mechanism according to the face video data stream to obtain a video heartbeat feature embedding vector;

[0050] a radar heartbeat feature extraction module, configured to perform heartbeat feature extraction on the distance perception video sequence containing the chest region of the human body, the distance-time heat map and the phase information matrix, and perform weighted fusion processing based on a video-guided attention mechanism to obtain a radar heartbeat feature embedding vector;

[0051] an environmental scene analysis processing module, configured to perform dimension normalization processing on the video heartbeat feature embedding vector and the radar heartbeat feature embedding vector, obtain an environmental feature vector through global average pooling, and perform scene division based on a scene classifier to obtain a scene classification result;

[0052] a multi-modal fusion weight determination module, configured to determine a multi-modal fusion weight according to the environmental feature vector and the scene classification result;

[0053] a heartbeat signal reconstruction module, configured to perform heartbeat signal reconstruction based on the video heartbeat feature embedding vector, the radar heartbeat feature embedding vector, the environmental feature vector and the multi-modal fusion weight to obtain a heartbeat signal sequence.

[0054] In a third aspect, a computer device is provided, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement the heartbeat detection method based on video and millimeter wave radar fusion.

[0055] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program executable by a processor to implement the heartbeat detection method based on video and millimeter wave radar fusion.

[0056] In a fifth aspect, a computer program product is provided, which includes a computer program executable by a processor to implement the heartbeat detection method based on video and millimeter wave radar fusion.

[0057] According to the embodiments provided in the present application, the following technical effects are disclosed:

[0058] The application provides a heartbeat detection method and device based on video and millimeter wave radar fusion, a device, a medium and a product. Based on a spatial attention mechanism, video modal heartbeat feature extraction is performed according to a face video data stream to obtain a video heartbeat feature embedding vector. Then, heartbeat feature extraction is performed, and weighted fusion processing is performed based on a video guided attention mechanism to obtain a radar heartbeat feature embedding vector. The video information, that is, the face video data stream, can be used to provide accurate spatial positioning guidance for radar detection, and the influence degree of motion interference is reduced. Dimension normalization processing is performed on the video heartbeat feature embedding vector and the radar heartbeat feature embedding vector, an environmental feature vector is obtained through global average pooling, scene classification is performed based on a scene classifier, and a scene classification result is obtained. After determining the multi-modal fusion weight according to the environmental feature vector and the scene classification result, heartbeat signal reconstruction is performed. The contribution weight of different modalities can be dynamically adjusted according to real-time environmental conditions and signal quality, so that stronger robustness and higher detection accuracy than single modal technology are realized. BRIEF DESCRIPTION OF DRAWINGS

[0059] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0060] Figure 1 Flowchart of the heartbeat detection method based on video and millimeter wave radar fusion;

[0061] Figure 2 Design flowchart of the heartbeat detection method based on video and millimeter wave radar fusion;

[0062] Figure 3 Structure diagram of the heartbeat detection device based on video and millimeter wave radar fusion;

[0063] Figure 4 Structure schematic diagram of a computer device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0064] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0065] In order to make the above objectives, characteristics and advantages of the present application more apparent, more comprehensible, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0066] In one exemplary embodiment, as shown in Figure 1 a heartbeat detection method based on video and millimeter wave radar fusion is provided, comprising:

[0067] Step 100: Obtain multi-modal sensor data. The multi-modal sensor data includes face video data stream collected based on a video acquisition module and distance perception video sequence containing human chest region, and distance-time heat map and phase information matrix obtained based on a millimeter wave radar acquisition module.

[0068] Step 200: Based on a spatial attention mechanism, video modal heartbeat feature extraction is performed according to the face video data stream to obtain a video heartbeat feature embedding vector.

[0069] Among them, based on the spatial attention mechanism, the video modal heartbeat feature extraction is performed according to the face video data stream to obtain the video heartbeat feature embedding vector, specifically including:

[0070] A three-dimensional convolutional neural network is used to encode the spatio-temporal features of the face video data stream, and the color change pattern of the skin region is extracted through multi-layer convolution operation; wherein the three-dimensional convolution kernel of the three-dimensional convolutional neural network slides in time dimension and space dimension at the same time, and captures the periodic color change characteristics caused by heartbeat.

[0071] Based on the spatial attention mechanism of the Transformer architecture, the feature weight distribution based on the pixel position is performed according to the color change pattern and the attention weighting processing is performed to obtain the video heartbeat feature embedding vector.

[0072] The attention weight calculation formula corresponding to the spatial attention mechanism is:

[0073] A spatial (i,j)=sigmoid(Transformer(F spatial (i,j)))。

[0074] Among them, A spatial (i,j) is the spatial attention weight of position (i,j); F spatial (i,j) is the feature vector of position (i,j); Transformer(·) is the transformation operation of the Transformer encoder; sigmoid(·) is the activation function.

[0075] Step 300: heartbeat feature extraction is performed on the distance perception video sequence, distance-time heat map and phase information matrix containing the chest region of the human body, and weighted fusion processing is performed based on the video guided attention mechanism to obtain a radar heartbeat feature embedding vector.

[0076] In an embodiment, heartbeat feature extraction is performed on the distance perception video sequence, distance-time heat map and phase information matrix containing the chest region of the human body, and weighted fusion processing is performed based on the video guided attention mechanism to obtain a radar heartbeat feature embedding vector, specifically including:

[0077] The distance perception video sequence containing the chest region of the human body is analyzed and determined for the spatial position of the human body, and a three-dimensional convolutional network is used to extract a global scene feature vector.

[0078] A two-dimensional convolution analysis is performed on the distance-time heat map to extract the distribution characteristics of the radar signal in the space-time domain.

[0079] A two-dimensional convolutional neural network is used to perform phase modulation pattern recognition on the phase information matrix to obtain a recognition result.

[0080] Based on the video guided attention mechanism, the global scene feature and the distance-time feature are fused and processed by attention weighting according to the global scene feature vector, the distribution characteristics of the radar signal in the space-time domain and the recognition result to obtain a radar heartbeat feature embedding vector.

[0081] The calculation formula of the dynamic attention weight corresponding to the video guided attention mechanism is:

[0082] W attention (r,t)=σ(FC global (F global )+FC range (F range-time ))。

[0083] Wherein, W attention (r,t) is the attention weight of position r and time t; FC global (·) and FC range (·) are the corresponding full connection mapping functions; σ(·) is the sigmoid activation function; F global is the global scene feature vector; F range-time is the distribution characteristics of the radar signal in the space-time domain.

[0084] Step 400: dimension normalization is performed on the video heartbeat feature embedding vector and the radar heartbeat feature embedding vector, and an environment feature vector is obtained through global average pooling, and scene classification is performed based on a scene classifier to obtain a scene classification result.

[0085] wherein, the video heartbeat feature embedding vector and the radar heartbeat feature embedding vector are subjected to dimension normalization processing, and an environment feature vector is obtained through global average pooling, and scene classification is performed based on a scene classifier to obtain a scene classification result, and the scene classification result comprises the following steps:

[0086] The video heartbeat feature embedding vector and the radar heartbeat feature embedding vector are subjected to dimension normalization processing; wherein, the video heartbeat feature embedding vector is reduced in dimension to a time sequence feature; and the radar heartbeat feature embedding vector is converted from a distance-time matrix to a set standard time sequence feature.

[0087] The normalized time sequence feature is analyzed by using a one-dimensional convolutional neural network to extract a feature pattern reflecting the environment quality, a change rule in the time dimension is captured through multi-layer convolution operation, and an environment feature vector is obtained through global average pooling.

[0088] Scene classification is performed based on a scene classifier to obtain a scene classification result; the scene classifier adopts a softmax function.

[0089] Step 500: determining a multi-modal fusion weight according to the environment feature vector and the scene classification result.

[0090] wherein, the multi-modal fusion weight is determined according to the environment feature vector and the scene classification result, and the determination comprises the following steps:

[0091] In the video mode, the corresponding fusion weight is determined according to the environment feature vector and the scene classification result by using the formula:

[0092] ω rPPG = softmax(FC weight ([F environment ,S category ])1).

[0093] In the radar mode, the corresponding fusion weight is determined according to the environment feature vector and the scene classification result by using the formula: ω radar = softmax(FC weight ([F environment ,S category ])2).

[0094] wherein, FC weight (·) is a full connection network for weight calculation; [·,·] is a feature splicing operation; FC weight is a learnable parameter of the full connection network for weight calculation; [·,·]1 is a weight output of the video mode; [·,·]2 is a weight output of the radar mode; F environment is the environment feature vector; S category is the scene classification result; and ω rPPGis a fusion weight corresponding to the video modality; ω radar is a fusion weight corresponding to the radar modality.

[0095] Step 600: Based on the video heartbeat feature embedding vector, the radar heartbeat feature embedding vector, the environment feature vector, and the multi-modal fusion weight, the heartbeat signal reconstruction is performed to obtain a heartbeat signal sequence.

[0096] In an embodiment, based on the video heartbeat feature embedding vector, the radar heartbeat feature embedding vector, the environment feature vector, and the multi-modal fusion weight, the heartbeat signal reconstruction is performed to obtain a heartbeat signal sequence, specifically comprising:

[0097] The video heartbeat feature embedding vector and the radar heartbeat feature embedding vector are dimensionally aligned and time-synchronously synchronized, and a weighted fusion strategy is adopted, the environment feature vector is expanded to the time sequence dimension as global context information, and fusion processing is performed based on the multi-modal fusion weight to obtain the fused feature; the weighted fusion strategy is adopted for fusion processing, and the corresponding expression is:

[0098] F fusion = ω rPPG · T rPPG + ω radar · T radar + α· F environment .

[0099] Wherein, F fusion is the fused feature; α is the contribution coefficient of the environment feature; ω rPPG is a fusion weight corresponding to the video modality; ω radar is a fusion weight corresponding to the radar modality; F environment is an environment feature vector; T rPPG is a time sequence feature obtained by dimension reduction of the video heartbeat feature embedding vector; T radar is a set standard time sequence feature obtained by converting the radar heartbeat feature embedding vector from a distance-time matrix.

[0100] The up-sampling and linear interpolation method are adopted to map the fused feature into a heartbeat signal sequence with a set standard length.

[0101] Millimeter-wave radar technology is based on the Doppler effect principle of electromagnetic waves. It detects the tiny physiological movements of the human body by transmitting continuous wave signals and analyzing the phase changes of the reflected echoes. When the human heart beats, the chest cavity will produce a periodic displacement of microns. This tiny movement modulates the phase of the radar reflected signal, forming a phase change pattern corresponding to the heart rate. Compared with video technology, millimeter-wave radar has significant technical advantages: it does not depend on environmental lighting conditions at all and can work normally in complete darkness; at the same time, millimeter-wave signals have certain material penetration ability and can penetrate thin clothing for detection, improving the flexibility of application scenarios; in addition, the radar detection process does not involve image acquisition, better protecting user privacy.

[0102] Despite the above advantages of millimeter-wave radar technology, it still faces many technical challenges in practical applications. The most prominent problem is motion sensitivity. When the measured object moves a large amplitude, it will produce a strong Doppler shift, which is much larger than the weak phase change caused by the heartbeat, completely masking the characteristics of the heartbeat signal. This motion interference not only includes subjective body movement, but also includes the chest fluctuation movement caused by breathing, which is usually 2-4 times lower than the heart rate, but has a larger amplitude and is easy to produce spectral aliasing with the heart rate signal. In addition, millimeter-wave radar has difficulty in accurately determining the specific position and distance of the target human body in a multi-target environment, which leads to the fact that the radar beam cannot be effectively focused on the target area, reducing the detection sensitivity of the heartbeat signal. At the same time, the spatial resolution of the radar signal is limited, making it difficult to distinguish the reflected signals from different body parts, further increasing the complexity of signal processing.

[0103] Through in-depth analysis of the advantages and limitations of existing technical solutions, the fundamental defects of single modal detection methods in complex real-world environments can be clearly identified. Although both rPPG technology and millimeter-wave radar technology have motion sensitivity problems, their failure mechanisms and impact levels differ significantly, providing an important technical basis for multi-modal fusion. Specifically, rPPG technology is relatively insensitive to small amplitude head movements, while millimeter-wave radar is completely immune to changes in light; video information can provide accurate spatial positioning and target recognition capabilities, while radar signals have stronger penetration and distance detection accuracy.

[0104] Based on the above technical status analysis, it is urgent to develop a heartbeat detection technology scheme that can organically integrate the advantages of video and millimeter wave radar modalities. This fusion scheme not only can use video information to provide accurate spatial positioning guidance for radar detection, reduce the influence degree of motion interference, but also can establish an intelligent timing signal evaluation and weighted fusion mechanism, dynamically adjust the contribution weight of different modalities according to real-time environmental conditions and signal quality, so as to realize stronger robustness and higher detection accuracy than single modality technology. This technical breakthrough will lay a solid technical foundation for the wide application of non-contact heartbeat detection in actual scenes.

[0105] The core content of the present application includes: (1) a multi-modal sensor data fusion method, which realizes effective fusion of two kinds of heterogeneous physiological signals by establishing a time domain synchronization mechanism and a frequency domain feature alignment algorithm for video rPPG signal and millimeter wave radar phase signal; (2) a radar beam focusing system guided by video, which uses computer vision technology to locate the chest region of the target human body in real time, dynamically adjusts the beam pointing and focusing range of the millimeter wave radar, and improves the acquisition accuracy of the radar heartbeat signal; (3) an adaptive weight distribution mechanism, which dynamically calculates the reliability score of each modality according to real-time environmental conditions (light intensity, motion amplitude, signal quality, etc.), and intelligently adjusts the fusion weight to optimize the detection performance. The focus of the invention is on the establishment of a cross-modal information collaborative processing mechanism, especially solving the technical bottleneck of the failure of traditional single sensor in complex environments, and significantly improving the environmental adaptability and measurement reliability of non-contact heartbeat detection through multi-sensor redundancy design and intelligent fusion algorithm.

[0106] The present application fully utilizes the complementary advantages of remote photoplethysmography (rPPG) and millimeter wave radar detection technology by establishing a multi-modal sensor collaborative working mechanism, and realizes high-precision heartbeat signal extraction in complex environmental conditions. The entire technical scheme adopts a hierarchical progressive processing architecture, including a multi-modal data acquisition layer, a feature extraction and enhancement layer, an environment adaptive evaluation layer, and an intelligent fusion decision layer, forming a complete heartbeat detection technology system. The specific design process is as shown in Figure 2 .

[0107] Step 1: Multi-modal sensor data synchronous acquisition.

[0108] The computer system simultaneously starts the video acquisition module and the millimeter wave radar acquisition module to realize multi-dimensional physiological information acquisition of the target human body. The video acquisition module captures video sequence data containing the face region through a high-resolution RGB camera, which records the subtle changes of skin color in the blood circulation process, forming a face video data stream V face(t), where t denotes the time index. Meanwhile, the video acquisition module also obtains a distance-aware video sequence V range (t) containing the chest region of the human body, which is used for subsequent radar beam positioning and spatial calibration. The millimeter-wave radar acquisition module transmits continuous wave signals and receives reflected echoes, and obtains the range-time heat map RT map (r, t) and the phase information matrix Φ(r, t), where the position r can represent the distance dimension. This step ensures the accurate alignment of the two modal data on the time axis through a hardware synchronization mechanism, laying the foundation for subsequent multi-modal fusion.

[0109] Step 2: Deep extraction of video modal heartbeat features.

[0110] The computer transmits the face video data stream V face (t) obtained in step 1 to the video heartbeat feature extraction module for deep learning processing. This module first uses a three-dimensional convolutional neural network to encode the spatio-temporal features of the input video sequence, and extracts the color change pattern of the skin region through multi-layer convolution operations. Specifically, the three-dimensional convolution kernel slides in both the time and spatial dimensions, capturing the periodic color change characteristics caused by the heartbeat. Subsequently, the system introduces a spatial attention mechanism based on the Transformer architecture, which identifies the key pixel positions in the facial region that are sensitive to the heartbeat signal through self-attention calculation, and assigns higher feature weights to these positions. The attention weight calculation formula is:

[0111] A spatial (i, j) = sigmoid(Transformer(F spatial (i, j))).

[0112] After attention weighting, the system obtains an enhanced video heartbeat feature embedding vector E rPPG , which contains rich heartbeat timing information and spatial positioning information.

[0113] Step 3: Intelligent extraction of millimeter-wave radar heartbeat signals.

[0114] The computer transmits the distance-aware video sequence V range (t), the range-time heat map RT map (r, t), and the phase information matrix Φ(r, t) collected in step 1 to the radar signal processing module for heartbeat feature extraction. This module first determines the accurate spatial position of the target human body through distance-aware video analysis, and uses a three-dimensional convolutional network to extract a global scene feature vector F global from the video sequence. At the same time, the system performs two-dimensional convolution analysis on the range-time heat map to extract the distribution features of the radar signal in the spatio-temporal domain F range-timeFor the key phase information matrix, the system adopts a two-dimensional convolutional neural network for deep feature learning, which is specially designed to identify the weak phase modulation patterns caused by heartbeats. After feature extraction is complete, the system introduces a video-guided attention mechanism to fuse the video global features and distance-time features, generating dynamic attention weights:

[0115] W attention (r,t) = σ(FC global (F global )+FC range (F range-time )).

[0116] Through the attention weighting mechanism, the system selectively enhances the phase features in space and time, ultimately obtaining the radar heartbeat feature embedding vector E radar .

[0117] Step 4: Intelligent assessment of environmental scene quality.

[0118] The video heartbeat feature embedding vector E rPPG and the radar heartbeat feature embedding vector E radar obtained in steps 2 and 3 are transmitted to the environmental scene quality assessment module for comprehensive environmental analysis. This module first performs dimension normalization on the feature vectors of the two modalities, reducing the video features from high-dimensional spatiotemporal representation to unified temporal feature representation T rPPG , and converting the radar features from distance-time matrix to standard temporal feature representation T radar . Subsequently, the system uses a one-dimensional convolutional neural network to perform deep analysis on the fused temporal features, extracting key feature patterns reflecting environmental quality. This network captures the variation of signal quality in the time dimension through multiple convolution operations and obtains the environmental feature vector F environment through global average pooling. Finally, a scene classifier is introduced to intelligently classify the current detection environment, dividing complex real-world scenarios into nine typical categories, including ideal lighting conditions, low-light environments, strong light interference, slight motion interference, moderate motion interference, strong motion interference, mixed interference, device abnormalities, and extreme environments. The scene classification result S category is output through the softmax function, representing the probability distribution of the current environment belonging to various scene categories.

[0119] Step 5: Multi-modal adaptive weight calculation.

[0120] According to the environmental feature vector F environment and the scene classification result S category, and the dynamic fusion weight of the video and radar modalities is calculated. The system first analyzes the reliability score of each modality under the current environmental conditions, and establishes the correlation between environmental characteristics and modality performance through neural network mapping. For the video modality, the weight calculation considers factors such as light quality, motion interference level, and target face clarity:

[0121] ω rPPG = softmax(FC weight ([F environment , S category ])1).

[0122] For the radar modality, the weight calculation mainly considers factors such as signal strength, multipath interference, and target distance:

[0123] ω radar = softmax(FC weight ([F environment , S category ])2).

[0124] Weight normalization ensures that ω rPPG + ω radar = 1, achieving dynamic balance of the contributions of the two modalities.

[0125] Step 6: Multi-modal feature intelligent fusion and heartbeat signal reconstruction.

[0126] The feature embedding vectors obtained in steps 2 and 3, the environmental characteristics obtained in step 4, and the fusion weight calculated in step 5 are transmitted to the multi-modal fusion module for final heartbeat signal reconstruction. This module first aligns the dimensions and synchronizes the time sequence of the video and radar features, ensuring accurate matching of the two modalities on the time axis. Then, the system adopts a weighted fusion strategy, extending the environmental characteristics as global context information to the time dimension, and deeply fusing them with the two modalities:

[0127] F fusion = ω rPPG · T rPPG + ω radar · T radar + α · F environment .

[0128] The fusion features are reconstructed by a multi-layer one-dimensional convolutional decoding network, which uses upsampling and linear interpolation techniques to map the fusion features to a standard length heartbeat signal sequence S heartbeat . The final output heartbeat signal contains complementary information from the two modalities, with stronger anti-interference ability and higher detection accuracy.

[0129] The beneficial effects of the present application are:

[0130] 1. Detection accuracy is significantly improved.

[0131] The heart rate detection accuracy of the present application is improved by 15%-25% under complex environmental conditions, and the root mean square error (RMSE) is reduced to below 2.1 bpm. This advantage mainly comes from the synergistic effect of multi-modal information fusion mechanism and adaptive weight allocation algorithm. Specifically, when a single modality (such as rPPG) is disturbed by changes in light or facial motion, the system can automatically increase the weight contribution of millimeter wave radar signals, effectively integrating complementary information to offset the measurement error of individual sensors. At the same time, the radar beam focusing technology guided by video enables the radar to accurately locate the chest heartbeat area, avoiding signal pollution from limb movement and breathing interference in traditional wide-beam detection, and improving the signal-to-noise ratio and feature purity of the radar heartbeat signal.

[0132] 2. Environmental robustness is greatly enhanced.

[0133] The present application can maintain stable detection performance in low light (<10 lux), strong light (>500 lux) and dynamic light change environment, and the environmental adaptability is improved by about 40% compared with traditional rPPG method. The breakthrough comes from the implementation of multi-level redundancy design concept, that is, when the video modality performance decreases due to deteriorating light conditions, the millimeter wave radar modality can provide stable backup detection capability. In particular, the real-time environment evaluation module can continuously monitor image quality indicators (contrast, brightness distribution, noise level, etc.) and radar signal strength, and dynamically adjust the working parameters and fusion strategy of each sensor according to the real-time changes of environmental conditions, ensuring that the system can maintain the optimal detection state under various lighting conditions.

[0134] 3. Motion interference suppression ability is significantly improved.

[0135] In the presence of moderate head movement (angle change <15°) and slight body sway, the heart rate detection success rate of the present application is more than 92%, which is improved by nearly 30% compared with single rPPG technology. This performance improvement is mainly due to the innovative design of the multi-level motion artifact suppression algorithm. This algorithm first uses facial key point tracking technology in video information to estimate head motion parameters in real time, and transmits the motion information to the radar signal processing module for motion compensation, finally realizing effective extraction of the heartbeat base frequency signal.

[0136] 4. System reliability and continuity are guaranteed.

[0137] The application has excellent fault tolerance capability, and can still maintain continuous heart rate monitoring under the condition of short-time failure (<3 seconds) of any single sensor, and the overall system availability reaches more than 98.5%. This high reliability is derived from the adoption of distributed sensing and centralized processing architecture, and the system is equipped with an intelligent fault diagnosis module, which can monitor the working state and signal quality of each sensor in real time, and immediately start the single-mode emergency working mode when an abnormality is detected. At the same time, the historical data assisted prediction mechanism uses the physiological parameter history record of the measured object to establish a personalized heart rate change model, and provides a predicted heart rate estimation value before the sensor recovers to normal, ensuring the continuity and integrity of medical monitoring.

[0138] In addition, for the core goal of multi-modal fusion, the sensor combination has the following technical alternative paths: (1) video + laser Doppler combination, using a laser Doppler vibrometer to replace the millimeter wave radar to detect the chest micro-motion caused by heartbeat, this scheme has higher displacement detection accuracy (nanometer level), but the laser power consumption is higher and the human eye safety requirement is strict; (2) video + piezoelectric sensor combination, by embedding a piezoelectric film sensor array in the seat or mattress to detect heartbeat vibration, combined with video rPPG for fusion analysis, this scheme has lower cost but requires a specific installation environment and limited application scenarios; (3) dual-band video fusion, using visible light and near-infrared cameras to obtain heartbeat signals under different spectra, and improving detection robustness through spectral domain fusion, this scheme has lower hardware complexity but limited improvement effect under strong motion interference.

[0139] At the signal fusion and processing level, the signal processing algorithm can consider the following alternative technical paths: (1) deep learning based end-to-end fusion, constructing a multi-modal convolutional neural network to directly extract heart rate features from raw video frames and radar I / Q data, avoiding the feature engineering step of traditional signal processing, but requiring a large amount of labeled data and having poor model interpretability; (2) Bayesian filtering fusion framework, using extended Kalman filter or particle filter algorithm for probability fusion of multi-modal heart rate estimation, this scheme is more rigorous in theory but has high computational complexity and may affect real-time performance; (3) frequency domain coherence analysis method, dynamically adjusting the fusion weight by calculating the coherence coefficient of different modal signals in the frequency domain, which has better noise suppression ability than time domain methods, but has higher requirements for signal preprocessing.

[0140] From the perspective of overall system design, the system architecture has the following alternative implementation paths: (1) distributed edge computing architecture, which disperses signal processing tasks to the embedded processors of each sensor node for execution, and only transmits the fused heart rate results to the central control unit. This scheme can reduce system delay and bandwidth demand, but increases hardware cost and power consumption; (2) cloud fusion processing mode, which uploads raw sensor data to a cloud server for centralized processing and analysis, and uses the powerful computing power of the cloud to implement more complex fusion algorithms, but has higher requirements for the stability and security of network connection; (3) reconfigurable hardware platform, which designs a dedicated multi-modal signal processing accelerator based on FPGA to improve the real-time performance of the system through hardware parallelization. This scheme has excellent processing speed but has a longer development cycle and relatively poor flexibility.

[0141] Although the above-mentioned alternative schemes have certain advantages in certain aspects, considering multiple factors such as detection accuracy, environmental adaptability, system complexity, cost-effectiveness, and actual deployment feasibility, the video-millimeter wave radar fusion scheme adopted by the present application still has the best overall performance under current technical conditions.

[0142] In one exemplary embodiment, as shown in Figure 3 a video-millimeter wave radar fusion-based heartbeat detection device is provided, comprising:

[0143] a data acquisition module for acquiring multi-modal sensor data, the multi-modal sensor data including face video data stream acquired based on a video acquisition module and distance perception video sequence containing human chest region, and distance-time heat map and phase information matrix acquired based on a millimeter wave radar acquisition module.

[0144] a video modal heartbeat feature extraction module for extracting video modal heartbeat features based on spatial attention mechanism according to the face video data stream to obtain a video heartbeat feature embedding vector.

[0145] a radar heartbeat feature extraction module for extracting heartbeat features from the distance perception video sequence containing the human chest region, the distance-time heat map, and the phase information matrix, and performing weighted fusion processing based on video-guided attention mechanism to obtain a radar heartbeat feature embedding vector.

[0146] an environmental scene analysis processing module for performing dimension normalization processing on the video heartbeat feature embedding vector and the radar heartbeat feature embedding vector, obtaining an environmental feature vector through global average pooling, and performing scene classification based on a scene classifier to obtain a scene classification result.

[0147] a multi-modal fusion weight determination module for determining multi-modal fusion weights according to the environmental feature vector and the scene classification result.

[0148] a heartbeat signal reconstruction module, configured to reconstruct a heartbeat signal based on the video heartbeat feature embedding vector, the radar heartbeat feature embedding vector, the environment feature vector, and the multi-modal fusion weight, to obtain a heartbeat signal sequence.

[0149] In an exemplary embodiment, a computer device is provided, which can be a server or a terminal, and an internal structure diagram thereof can be as shown in Figure 4 The computer device includes a processor, a memory, an input / output interface (I / O), and a communication interface. The processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The database of the computer device is configured to store heartbeat detection data based on video and millimeter wave radar fusion. The input / output interface of the computer device is configured to exchange information between the processor and external devices. The communication interface of the computer device is configured to communicate with external terminals through network connection. The computer program is executed by the processor to implement a heartbeat detection method based on video and millimeter wave radar fusion.

[0150] Those skilled in the art can understand that Figure 4 the structure shown in the above figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0151] In an exemplary embodiment, a computer device is provided, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0152] In an exemplary embodiment, a computer readable storage medium is provided, which stores a computer program. The computer program is executed by a processor to implement the steps in the above method embodiments.

[0153] In an exemplary embodiment, a computer program product is provided, which includes a computer program. The computer program is executed by a processor to implement the steps in the above method embodiments.

[0154] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations.

[0155] It can be understood by those skilled in the art that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing related hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments of each method. Any reference to memory, database or other medium used in each embodiment provided by the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (Read-Only Memory, ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive random access memory (Resistive Random Access Memory, ReRAM), magnetoresistive random access memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric memory (Ferroelectric Random Access Memory, FRAM), phase change memory (Phase Change Memory, PCM), graphene memory, etc. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), etc.

[0156] The database involved in each embodiment provided by the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in each embodiment provided by the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0157] Any technical features in the above embodiments can be combined, and for the sake of brevity, not all possible combinations are described above, however, it should be understood that the application encompasses all possible combinations of the technical features described above.

[0158] The principles and implementation manners of the present application are described herein by using specific examples, and the above embodiments are only used to help understand the method of the present application and its core idea; meanwhile, according to the idea of the present application, the specific implementation manners and application scopes will be changed by those skilled in the art. In conclusion, the content of the present specification should not be understood as a limitation of the present application.

Claims

1. A heartbeat detection method based on the fusion of video and millimeter-wave radar, characterized in that, include: Acquire multimodal sensor data; The multimodal sensor data includes: a facial video data stream acquired by the video acquisition module and a distance-sensing video sequence containing the human chest region, as well as a distance-time heatmap and phase information matrix acquired by the millimeter-wave radar acquisition module; Based on the spatial attention mechanism, video modal heartbeat features are extracted from the face video data stream to obtain video heartbeat feature embedding vectors. Heartbeat features are extracted from the distance-sensing video sequence containing the human chest region, the distance-time heatmap, and the phase information matrix, and weighted fusion is performed based on a video-guided attention mechanism to obtain a radar heartbeat feature embedding vector. The video heartbeat feature embedding vector and the radar heartbeat feature embedding vector are subjected to dimensionality normalization, and environmental feature vectors are obtained by global average pooling. Scenes are then segmented based on a scene classifier to obtain scene classification results. The multimodal fusion weights are determined based on the environmental feature vector and the scene classification result; Based on the video heartbeat feature embedding vector, the radar heartbeat feature embedding vector, the environmental feature vector, and the multimodal fusion weights, the heartbeat signal is reconstructed to obtain a heartbeat signal sequence.

2. The heartbeat detection method based on video and millimeter-wave radar fusion according to claim 1, characterized in that, Based on the spatial attention mechanism, video modality heartbeat features are extracted from the face video data stream to obtain a video heartbeat feature embedding vector, specifically including: A three-dimensional convolutional neural network is used to encode the spatiotemporal features of the face video data stream, and the color change pattern of the skin region is extracted through multi-layer convolution operations; wherein, the three-dimensional convolution kernel of the three-dimensional convolutional neural network slides simultaneously in the time and space dimensions to capture the periodic color change characteristics caused by heartbeat; Based on the spatial attention mechanism of the Transformer architecture, feature weights are assigned based on pixel position according to the color change pattern and attention weighting is performed to obtain the video heartbeat feature embedding vector. The formula for calculating attention weights corresponding to spatial attention mechanisms is: A spatial (i,j)=sigmoid(Transformer(F spatial (i,j))); Among them, A spatial (i,j) represents the spatial attention weight at position (i,j); F spatial (i,j) is the feature vector at position (i,j); Transformer(·) is the transformation operation of the Transformer encoder; sigmoid(·) is the activation function.

3. The heartbeat detection method based on video and millimeter-wave radar fusion according to claim 1, characterized in that, Heartbeat features are extracted from the distance-sensing video sequence containing the human chest region, the distance-time heatmap, and the phase information matrix. A weighted fusion process based on a video-guided attention mechanism is then performed to obtain a radar heartbeat feature embedding vector, specifically including: The spatial position of the human body is determined by analyzing the distance-sensing video sequence containing the chest region of the human body, and a three-dimensional convolutional network is used to extract the global scene feature vector. Two-dimensional convolutional analysis is performed on the range-time heatmap to extract the distribution characteristics of radar signals in the spatiotemporal domain; A two-dimensional convolutional neural network is used to perform phase modulation pattern recognition on the phase information matrix to obtain the recognition result. Based on the video-guided attention mechanism, the global video features and distance-time features are fused according to the global scene feature vector, the distribution characteristics of radar signals in the spatiotemporal domain, and the recognition results, and the radar heartbeat feature embedding vector is obtained through attention weighting. The formula for calculating the dynamic attention weights corresponding to the video-guided attention mechanism is as follows: W attention (r,t)=σ(FC global (F global )+FC range (F range-time )); Among them, W attention (r,t) represents the attention weights at position r and time t; FC global (·) and FC range (·) represent the corresponding fully connected mapping functions; σ(·) is the sigmoid activation function; F global F is the global scene feature vector; range-time This represents the distribution characteristics of radar signals in the spatiotemporal domain.

4. The heartbeat detection method based on video and millimeter-wave radar fusion according to claim 1, characterized in that, The video heartbeat feature embedding vector and the radar heartbeat feature embedding vector are subjected to dimensionality normalization, and environmental feature vectors are obtained through global average pooling. Scene classification results are then obtained by segmenting the scene based on a scene classifier. Specifically, this includes: The video heartbeat feature embedding vector and the radar heartbeat feature embedding vector are subjected to dimensionality normalization processing; wherein, the video heartbeat feature embedding vector is reduced to temporal features; and the radar heartbeat feature embedding vector is converted from a distance-time matrix into a set standard temporal features. A one-dimensional convolutional neural network is used to analyze the obtained normalized time-series features, extract feature patterns that reflect environmental quality, capture the change patterns in the time dimension through multi-layer convolution operations, and obtain environmental feature vectors through global average pooling. Scenes are segmented based on a scene classifier to obtain scene classification results; the scene classifier uses the softmax function.

5. The heartbeat detection method based on video and millimeter-wave radar fusion according to claim 1, characterized in that, The multimodal fusion weights are determined based on the environmental feature vector and the scene classification result, specifically including: In the video modality, based on the environmental feature vector and the scene classification result, the following formula is used: ω rPPG =softmax(FC weight ([F environment ,S category ])1); Determine the corresponding fusion weights; In radar mode, based on the environmental feature vector and the scene classification result, the following formula is used: ω radar =softmax(FC weight ([F environment ,S category ])2); Determine the corresponding fusion weights; Among them, FC weight (·) represents a fully connected network for weight calculation; [·,·] represents the feature concatenation operation; FC weight F represents the learnable parameters of the fully connected network used for weight calculation; [·,·]1 represents the weight output of the video modality; [·,·]2 represents the weight output of the radar modality; F environment S is the environmental feature vector; category For scene classification results; ω rPPG ω represents the fusion weights corresponding to the video modalities. radar The fusion weights are the radar modes.

6. The heartbeat detection method based on video and millimeter-wave radar fusion according to claim 1, characterized in that, Based on the video heartbeat feature embedding vector, the radar heartbeat feature embedding vector, the environmental feature vector, and the multimodal fusion weights, heartbeat signals are reconstructed to obtain a heartbeat signal sequence, specifically including: The video heartbeat feature embedding vector and the radar heartbeat feature embedding vector are dimensionally aligned and temporally synchronized. A weighted fusion strategy is then employed to extend the environmental feature vector as global context information to the temporal dimension, and fusion processing is performed based on the multimodal fusion weights to obtain the fused features. The corresponding expression for the weighted fusion strategy is as follows: F fsion =ω rPPG ·T rPPG +oh radar ·T radar +α·F environment ; Among them, F fusion The fused features are represented by α, which is the contribution coefficient of the environmental features; ω is the fused feature coefficient. rPPG ω represents the fusion weights corresponding to the video modalities. radar F represents the fusion weights corresponding to the radar modes. environment T represents the environmental feature vector. rPPG The temporal features obtained by embedding vector dimensionality reduction into the video heartbeat features; T radar The radar heartbeat feature embedding vector is obtained by transforming the range-time matrix to a set standard time-series feature; The fused features are mapped to a heartbeat signal sequence of a set standard length using upsampling and linear interpolation methods.

7. A heartbeat detection device based on the fusion of video and millimeter-wave radar, characterized in that, include: The data acquisition module is used to acquire data from multimodal sensors; The multimodal sensor data includes: a facial video data stream acquired by the video acquisition module and a distance-sensing video sequence containing the human chest region, as well as a distance-time heatmap and phase information matrix acquired by the millimeter-wave radar acquisition module; The video modal heartbeat feature extraction module is used to extract video modal heartbeat features based on the face video data stream using a spatial attention mechanism, and to obtain a video heartbeat feature embedding vector. The radar heartbeat feature extraction module is used to extract heartbeat features from the distance-sensing video sequence containing the human chest region, the distance-time heatmap, and the phase information matrix, and to perform weighted fusion processing based on a video-guided attention mechanism to obtain a radar heartbeat feature embedding vector. The environmental scene analysis and processing module is used to perform dimensionality normalization processing on the video heartbeat feature embedding vector and the radar heartbeat feature embedding vector, obtain the environmental feature vector through global average pooling, and perform scene segmentation based on the scene classifier to obtain the scene classification result. A multimodal fusion weight determination module is used to determine multimodal fusion weights based on the environmental feature vector and the scene classification result; The heartbeat signal reconstruction module is used to reconstruct the heartbeat signal based on the video heartbeat feature embedding vector, the radar heartbeat feature embedding vector, the environmental feature vector, and the multimodal fusion weights to obtain a heartbeat signal sequence.

8. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the heartbeat detection method based on video and millimeter-wave radar fusion as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the heartbeat detection method based on video and millimeter-wave radar fusion as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the heartbeat detection method based on video and millimeter-wave radar fusion as described in any one of claims 1-6.

Images