Non-contact breathing and heartbeat monitoring method and device combining visual auxiliary positioning and millimeter wave radar dynamic tracking

Abstract

The invention provides a non-contact breathing and heartbeat monitoring method and device combining visual auxiliary positioning and millimeter wave radar dynamic tracking, and belongs to the technical field of vital sign monitoring and millimeter wave radar. The system comprises a depth camera module, an edge calculation module and a millimeter wave radar module. The depth camera module is used for collecting human body image data; the edge calculation module is used for identifying skeleton key points according to the human body image data, positioning a chest position and predicting a motion track of a target to obtain prediction information; the millimeter wave radar module is used for dynamically adjusting wave beams according to prediction information and tracking and collecting signals of a chest area; the edge calculation module is used for separating the signals of the chest area and extracting respiration and heartbeat information to obtain non-inductive monitoring data of human body vital signs. According to the invention, accurate positioning and dynamic and non-contact vital sign monitoring of the human chest can be realized, and the monitoring accuracy and the anti-interference capability are effectively improved.

Description

Technical Field

[0001] This application relates to the fields of vital sign monitoring and millimeter-wave radar technology, and in particular to a non-contact respiratory and heart rate monitoring method and device that combines visual-assisted positioning with dynamic tracking by millimeter-wave radar. Background Technology

[0002] In fields such as healthcare, smart elderly care, and remote monitoring, the monitoring of vital signs such as respiration and heartbeat is receiving increasing attention. Traditional methods of vital sign monitoring mainly rely on contact sensors, such as electrocardiogram electrodes and chest strap respiratory sensors. While these devices can achieve high-precision monitoring, prolonged wear can cause discomfort for users, limiting their application in everyday environments, and making them particularly unsuitable for special populations such as the elderly, infants, or patients with severe burns. Furthermore, contact sensors also have limitations in scenarios involving multiple people and long-distance monitoring, making it difficult to meet the needs of smart homes and public spaces for non-intrusive monitoring.

[0003] In recent years, non-contact vital sign monitoring technologies have gradually developed, such as camera-based visible light detection methods and millimeter-wave radar detection methods. Camera-based technologies are easily affected by factors such as light and obstruction, and privacy protection issues are prominent. Although millimeter-wave radar technology has advantages such as penetrating clothing and non-intrusive detection, a single radar system has difficulty accurately locking onto the target's chest, especially when the monitored person is moving or there are multiple people present, as it is easily affected by body movement and environmental interference, resulting in poor signal stability and decreased accuracy.

[0004] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the Invention

[0005] The main purpose of this application is to propose a non-contact respiratory and heartbeat monitoring method and device that combines visual-assisted positioning with millimeter-wave radar dynamic tracking. This device can dynamically track and accurately lock the target's chest position, and achieve highly reliable, imperceptible non-contact monitoring of vital signs in everyday life environments.

[0006] To achieve the above objectives, one aspect of this application proposes a non-contact respiratory and heart rate monitoring device that combines visual-assisted positioning with millimeter-wave radar dynamic tracking. The device includes a depth camera module, an edge computing module, and a millimeter-wave radar module.

[0007] The depth camera module is used to collect human image data;

[0008] The edge computing module is used to identify key skeletal points based on the human image data, locate the chest position, and predict the target's movement trajectory to obtain prediction information;

[0009] The millimeter-wave radar module is used to dynamically adjust the beam according to the prediction information, and to track and collect signals from the chest region.

[0010] The edge computing module separates the signals from the chest area and extracts breathing and heartbeat information to obtain non-invasive monitoring data of human vital signs.

[0011] In some embodiments, the depth camera module is a structured light depth camera, a time-of-flight depth camera, or a binocular vision depth camera.

[0012] In some embodiments, the edge computing module includes a human skeleton point recognition deep learning model and a motion trajectory prediction unit;

[0013] The human skeleton point recognition deep learning model is used to identify the chest region of the human body, obtain chest position information, and jointly calculate the spatial coordinates of the chest region with the depth camera module; the motion trajectory prediction unit is used to calculate the chest movement velocity and acceleration based on the chest position information, and predict the future position trajectory of the chest.

[0014] In some embodiments, the millimeter-wave radar module adopts a multi-transmitter, multi-receiver antenna array structure and operates in the frequency range of 60 GHz to 81 GHz.

[0015] In some embodiments, the millimeter-wave radar module focuses the transmitted radar signal onto the predicted future location of the human chest by controlling the phase of each transmitting antenna; the millimeter-wave radar module adjusts the weighting coefficients of each receiving antenna using beamforming technology at the receiving end so that the reflected signal in the direction of the predicted location maintains a distortion-free response.

[0016] In some embodiments, the millimeter-wave radar module transmits a linear frequency modulated continuous wave signal, using the following formula:

[0017] S tx (t)=A·cos[2πf c t+πKt 2 +φ0];

[0018] Among them, S tx (t) represents the linear frequency modulated continuous wave signal; A represents the amplitude of the transmitted linear frequency modulated continuous wave signal; f c φ0 is the carrier frequency; K is the tuning frequency; φ0 is the initial phase; t is the transmission time.

[0019] In some embodiments, the signal processing function within the edge computing module includes extracting phase information of the reflected signal, processing the phase information, and separating the respiratory signal and heartbeat signal respectively using digital filtering technology.

[0020] In some embodiments, the digital filtering technology employs a bandpass filtering method, wherein the filtering frequency range of the respiratory signal is 0.1Hz to 0.6Hz, and the filtering frequency range of the heartbeat signal is 0.8Hz to 2Hz.

[0021] To achieve the above objectives, another aspect of this application proposes a non-contact respiratory and heart rate monitoring method combining visual-assisted positioning and millimeter-wave radar dynamic tracking, which is implemented using the device described above. The method includes the following steps:

[0022] Use a depth camera module to collect human image data;

[0023] The edge computing module is used to identify key skeletal points based on the human image data, locate the chest position, and predict the target's movement trajectory to obtain prediction information;

[0024] The millimeter-wave radar module dynamically adjusts the beam based on the predicted information to track and acquire signals from the chest region.

[0025] The edge computing module is used to separate the signals in the chest area and extract breathing and heartbeat information to obtain non-invasive monitoring data of human vital signs.

[0026] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer program product, including a computer program that, when executed by a processor, implements the aforementioned method.

[0027] This application provides at least the following beneficial effects: It offers a non-contact respiratory and heartbeat monitoring method, device, and program product that combines visual-assisted positioning with millimeter-wave radar dynamic tracking. The solution includes: a depth camera module, an edge computing module, and a millimeter-wave radar module. The depth camera module is used to acquire human image data; the edge computing module is used to identify key skeletal points based on the human image data, locate the chest position, and predict the target's movement trajectory to obtain prediction information; the millimeter-wave radar module is used to dynamically adjust the beam based on the prediction information, track and acquire signals from the chest region; the edge computing module separates the signals from the chest region and extracts respiratory and heartbeat information to obtain non-contact monitoring data of human vital signs. This application enables precise positioning of the human chest and dynamic, non-contact monitoring of vital signs, effectively improving monitoring accuracy and anti-interference capabilities. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of the non-contact respiratory and heart rate monitoring device that combines visual-assisted positioning with millimeter-wave radar dynamic tracking provided in this application;

[0029] Figure 2This is a schematic diagram of a module composition embodiment of the non-contact respiratory and heart rate monitoring device that combines visual-assisted positioning and millimeter-wave radar dynamic tracking provided in this application;

[0030] Figure 3 This is a flowchart illustrating the non-contact respiratory and heart rate monitoring method combining visual-assisted positioning and millimeter-wave radar dynamic tracking provided in this application.

[0031] Figure 4 This is a schematic diagram illustrating the monitoring principle of an embodiment of the depth camera target localization method provided in this application. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0033] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to a determination” as used herein may be interpreted as “when…” or “when…” or “in response to a determination.”

[0034] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0035] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0036] Before providing a detailed description of the embodiments of this application, some of the nouns and terms involved in the embodiments of this application will be explained first. The nouns and terms involved in the embodiments of this application are subject to the following interpretations.

[0037] 1) GHz, gigahertz, a unit of frequency;

[0038] 2) Hz, Hertz, a unit of frequency;

[0039] 3) FMCW (Frequency Modulated Continuous Wave): A type of continuous wave signal whose transmission frequency varies linearly.

[0040] 4) MVDR (Minimum Variance Distortionless Response), a type of adaptive beamforming technology;

[0041] 5) FFT (Fast Fourier Transform);

[0042] 6) ResNet18, a deep residual network model;

[0043] 7) D455, a depth camera in the Intel RealSense D400 series, designed for machine vision applications;

[0044] 8) Jetson AGX Orin, NVIDIA's edge AI supercomputing module;

[0045] 9) TI IWR6843ISK-ODS, a 60GHz millimeter-wave radar evaluation kit from Texas Instruments (TI).

[0046] In related technologies, vital sign monitoring primarily relies on contact sensors, such as electrocardiogram electrodes and chest strap respiratory sensors. While these devices can achieve high-precision monitoring, prolonged wear can cause discomfort for users, limiting their application in everyday environments, and making them particularly unsuitable for special populations such as the elderly, infants, or patients with severe burns. Furthermore, contact sensors also have limitations in scenarios involving multiple people and long-distance monitoring, making it difficult to meet the needs of smart homes and public spaces for seamless monitoring.

[0047] In recent years, non-contact vital sign monitoring technologies have gradually developed, such as camera-based visible light detection methods and millimeter-wave radar detection methods. Camera-based technologies are easily affected by factors such as light and obstruction, and privacy protection issues are prominent. Although millimeter-wave radar technology has advantages such as penetrating clothing and non-intrusive detection, a single radar system has difficulty accurately locking onto the target's chest, especially when the monitored person is moving or there are multiple people present, as it is easily affected by body movement and environmental interference, resulting in poor signal stability and decreased accuracy.

[0048] In view of this, this application provides a non-contact respiratory and heart rate monitoring method and device that combines visual-assisted positioning with millimeter-wave radar dynamic tracking, in order to solve the problems mentioned in the background art, such as the inconvenience of contact monitoring, the susceptibility of pure radar or pure visual monitoring to interference, and the difficulty in stable monitoring under human activity.

[0049] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require the acquisition of sensitive personal information of users, separate permission or consent from the user is obtained. Only after explicitly obtaining the user's separate permission or consent is the necessary user-related data acquired to enable the embodiments of this application to function properly.

[0050] The embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0051] On one hand, embodiments of the present invention provide a non-contact respiratory and heart rate monitoring device combining visual-assisted positioning and millimeter-wave radar dynamic tracking, hereinafter referred to as the device, with reference to Figure 1 The device includes a depth camera module, an edge computing module, and a millimeter-wave radar module;

[0052] The depth camera module is used to acquire human image data;

[0053] The edge computing module is used to identify key skeletal points based on human image data, locate the chest position, and predict the target's motion trajectory to obtain prediction information;

[0054] The millimeter-wave radar module is used to dynamically adjust the beam based on predictive information to track and acquire signals in the chest region;

[0055] The edge computing module separates the signals from the chest area and extracts breathing and heartbeat information to obtain non-invasive monitoring data of human vital signs.

[0056] Furthermore, the non-contact respiratory and heart rate monitoring device combining visual-assisted positioning and millimeter-wave radar dynamic tracking in this embodiment of the invention can achieve unobtrusive vital sign monitoring without requiring the wearer to wear any sensors, making it suitable for scenarios such as family health management, smart elderly care, and medical assistance.

[0057] Furthermore, this invention fully utilizes the spatial positioning capabilities of a vision system and the micro-motion detection advantages of millimeter-wave radar to achieve high-precision, non-contact vital sign monitoring. This invention also provides data visualization technology, enabling users to view historical data via local LED displays and a remote app, supporting data export and analysis, and providing a basis for health management.

[0058] In some embodiments, the depth camera module disclosed in this invention is a structured light depth camera, a time-of-flight depth camera, or a binocular vision depth camera.

[0059] In some embodiments, the edge computing module disclosed in this invention includes a human skeleton point recognition deep learning model and a motion trajectory prediction unit.

[0060] A deep learning model for human skeleton point recognition is used to identify the chest region of the human body, obtain chest position information, and calculate the spatial coordinates of the chest region together with a depth camera module; a motion trajectory prediction unit is used to calculate the chest movement velocity and acceleration based on the chest position information, and predict the future position trajectory of the chest.

[0061] In some embodiments, the millimeter-wave radar module disclosed in this invention adopts a multi-transmitter, multi-receiver antenna array structure, with an operating frequency range of 60 GHz to 81 GHz.

[0062] In some embodiments, the millimeter-wave radar module disclosed in this invention focuses the transmitted radar signal on the predicted future location of the human chest by controlling the phase of each transmitting antenna; the millimeter-wave radar module adjusts the weighting coefficients of each receiving antenna by beamforming technology at the receiving end so that the reflected signal in the direction of the predicted location maintains a distortion-free response.

[0063] Furthermore, the millimeter-wave radar module of this embodiment of the invention uses beamforming technology at the receiver end to adjust the weighting coefficients of each receiving antenna, so that the reflected signal in the direction of the predicted future chest position of the human body remains undistorted, while minimizing interference and noise energy from other directions.

[0064] In some embodiments, the millimeter-wave radar module disclosed in this invention transmits a linear frequency modulated continuous wave signal, and the formula used includes:

[0065] S tx (t)=A·cos[2πf c t+πKt 2 +φ0];

[0066] Among them, S tx (t) represents the linear frequency modulated continuous wave signal; A represents the amplitude of the transmitted linear frequency modulated continuous wave signal; f cφ0 is the carrier frequency; K is the tuning frequency; φ0 is the initial phase; t is the transmission time.

[0067] In some embodiments, the signal processing function within the edge computing module disclosed in this invention includes extracting phase information of the reflected signal, processing the phase information, and separating the respiratory signal and heartbeat signal respectively using digital filtering technology.

[0068] Furthermore, in this embodiment of the invention, the phase information is processed, and finally the respiratory signal and heartbeat signal are separated by digital filtering technology.

[0069] Specifically, a bandpass filter is used to process the phase sequence to obtain respiratory and heartbeat signals. This filter can be expressed using a difference equation:

[0070]

[0071] Where x[n] is the input signal, y[n] is the output signal, and {a k ,b k} are the filter coefficients, which can be obtained using filter design tools.

[0072] In some embodiments, the digital filtering technology disclosed in this invention employs a bandpass filtering method, with the filtering frequency range for respiratory signals being 0.1Hz to 0.6Hz and the filtering frequency range for heartbeat signals being 0.8Hz to 2Hz.

[0073] On the other hand, embodiments of the present invention also provide a non-contact respiratory and heart rate monitoring method combining visual-assisted positioning and millimeter-wave radar dynamic tracking, which is implemented by the aforementioned device, and the method includes the following steps:

[0074] Step S100: Use a depth camera module to acquire human image data;

[0075] Step S200: Use the edge computing module to identify key skeletal points based on human image data, locate the chest position, and predict the target's motion trajectory to obtain prediction information;

[0076] Step S300: Use the millimeter-wave radar module to dynamically adjust the beam based on the prediction information, track and collect signals from the chest region;

[0077] Step S400: Use the edge computing module to separate the signal in the chest area and extract breathing and heartbeat information to obtain non-invasive monitoring data of human vital signs.

[0078] As an optional implementation method, refer to Figure 2 , Figure 3 and Figure 4The present invention proposes a non-contact respiratory and heartbeat monitoring device that combines visual-assisted positioning and millimeter-wave radar dynamic tracking, comprising: a depth camera module, an edge computing module, and a millimeter-wave radar module;

[0079] The depth camera module is used to collect human image data and send it back to the edge computing module;

[0080] The edge computing module is used to identify key points on the skeleton, locate the chest position, and predict its movement trajectory;

[0081] The millimeter-wave radar module dynamically adjusts the beam based on prediction information, continuously tracking and acquiring signals from the chest region.

[0082] Furthermore, the edge computing module also has signal processing capabilities, separating and extracting breathing and heartbeat information to achieve seamless monitoring of human vital signs.

[0083] Specifically, the depth camera continuously captures three frames of images at fixed time intervals, records this time interval as Δt, and quickly transmits these images to the edge computing module via wired communication.

[0084] Furthermore, the edge computing module uses a deep learning model for human skeleton point recognition to find the location of the human heart in the three frames of images, denoted as (x1, y1, z1), (x2, y2, z2), and (x3, y3, z3).

[0085] Optionally, the deep learning model for human skeleton point recognition uses the ResNet18 model.

[0086] Specifically, Figure 4 This demonstrates the ranging principle of a binocular depth camera. It shows how to measure distances when the baseline length *b* between the left and right cameras, the focal length *f* from the depth camera to the imaging plane, and the position coordinates (x, y) of the two cameras are obtained. L ,y L ),(x R ,y R After that, the position coordinates of the target at the time of shooting can be calculated:

[0087]

[0088] Optionally, the edge computing module estimates the velocity (v) of the human chest at this moment using the following formula. x ,v y ,v z ):

[0089]

[0090] Optionally, the edge computing module estimates the acceleration of the human chest at this moment using the following formula (a x ,a y ,a z ):

[0091]

[0092] Optionally, the edge computing module estimates the position (x′, y′, z′) of the human chest at the next moment using the following formula:

[0093]

[0094] Furthermore, considering the positional difference (Δx, Δy, Δz) between the camera and the millimeter-wave radar, the predicted position needs to be compensated and corrected.

[0095]

[0096] Furthermore, the edge computing module transforms the coordinates in the spatial coordinate system into the azimuth angle θ and elevation angle φ of the radar array (with the center of the millimeter-wave radar array as the origin):

[0097]

[0098] Furthermore, the edge computing module inputs the obtained azimuth angle θ and elevation angle φ into the millimeter-wave radar module via wired communication.

[0099] Furthermore, before using millimeter-wave radar modules to track the human body, interference from static environmental noise should be eliminated.

[0100] Specifically, it is only necessary to record the signal received when there is no target being measured, denoted as . Subtracting this signal during subsequent monitoring will eliminate static noise interference. Let the received signal be Y. t Then the signal Y after eliminating static interference t * It can be obtained in the following ways:

[0101]

[0102] Optionally, the millimeter-wave radar transmits a linear frequency modulated continuous wave (FMCW) signal, the baseband form of which can be expressed as:

[0103] S tx (t)=A·cos[2πf c t+πKt 2 +φ0],

[0104] Where A represents the amplitude of the transmitted signal, f cThe carrier frequency is represented by K, the tuning frequency (Hz / s) is represented by φ0, the initial phase is represented by t, and the transmission time is represented by t.

[0105] Furthermore, assuming c is the speed of light, the propagation delay experienced by the reflected signal is:

[0106]

[0107] Furthermore, the signal received by the millimeter-wave radar should be:

[0108] S rx (t)=α·A·cos[2πf c (t-τ)+πK(t-τ) 2 +φ0],

[0109] Where α is the target reflectance coefficient, which includes path loss and human body reflectance characteristics.

[0110] Furthermore, the transmitted and received signals undergo frequency mixing due to coherent demodulation by the local oscillator, resulting in the following mixed signal:

[0111] S IF (t)=β·cos[2πf IF (t)t+φ IF (t)],

[0112] Among them, S IF (t) is a mid-frequency complex signal, f IF (t) is the frequency of the intermediate frequency signal, φ IF (t) represents the phase change including the initial phase and the phase changes caused by breathing and heartbeat disturbances.

[0113] Furthermore, the millimeter-wave radar module controls the phase of each transmitting antenna to focus the transmitted radar signal on the predicted future location of the human chest.

[0114] Specifically, for the nth transmitting antenna (n = 0, 1, ..., N-1), the phase shift of its excitation signal is:

[0115]

[0116] The distance between each antenna is d, and the operating wavelength is λ.

[0117] Furthermore, its complex weights (phase compensation factors) are:

[0118]

[0119] Furthermore, the signal from each transmitting antenna is multiplied by w. n This allows the main lobe of the transmitting antenna beam to be pointed at the predicted future location of the human chest.

[0120] Furthermore, the receiver can also set weights and adjust the weighting coefficients of each receiving antenna to ensure that the reflected signal in the direction of the predicted future chest position of the human body remains undistorted, while minimizing interference and noise energy from other directions.

[0121] Furthermore, beamforming at the receiver can be achieved using a minimum noise variance beamformer (MVDR), which can be expressed as solving the following optimization problem:

[0122]

[0123] Where w is the weighting coefficient, a0 is the steering vector of the desired signal, and R is the sample covariance matrix.

[0124] Furthermore, the millimeter-wave radar transmits the received echo signal (intermediate frequency signal) to the edge computing module via wired communication for further signal processing.

[0125] Furthermore, the millimeter-wave radar processes the intermediate frequency signal and extracts the phase information of the echo signal using the arctangent function:

[0126]

[0127] Among them, Im[S] IF [(t)] represents the imaginary part of the signal, Re[S] IF [(t)] represents the real part of the signal.

[0128] Furthermore, to eliminate phase abrupt changes, a continuous phase change sequence obtained through phase unwinding is used, and the signal phase after continuous unwinding... It can be represented as:

[0129]

[0130] Here, n(t) is an integer counting function used to eliminate 2π jumps.

[0131] Furthermore, respiratory signals and heartbeat signals are separated using digital filtering technology.

[0132] Optionally, a sixth-order Butterworth bandpass filter is used to process the phase sequence to obtain respiratory and heartbeat signals. This filter can be expressed using a difference equation:

[0133]

[0134] Where x[n] is the input signal, y[n] is the output signal, and {a k ,b k} are the coefficients of a sixth-order bandpass filter, which can be automatically generated by filter design tools for the target frequency band.

[0135] Specifically, the filtering frequency range for respiratory signals is 0.1Hz to 0.6Hz, and the filtering frequency range for heartbeat signals is 0.8Hz to 2Hz.

[0136] Furthermore, Fast Fourier Transform (FFT) is performed on the respiratory signal and heartbeat signal respectively, and the components corresponding to the main peak frequency are the respiratory rate and heartbeat rate.

[0137] Furthermore, as the human body moves or changes posture, the edge computing module continuously updates the heart coordinates, while the millimeter-wave radar adjusts its beam direction in real time to achieve continuous tracking of the heart region. Through this beamforming and dynamic tracking mechanism, the system can stably acquire high signal-to-noise ratio radar signals reflecting the subtle movements of the heart, providing a solid foundation for the high-quality extraction of subsequent respiratory and heartbeat signals.

[0138] Alternatively, the depth camera module uses the Intel RealSense camera D455, which is essentially a binocular vision depth camera.

[0139] Optionally, the edge computing module uses NVIDIA Jetson AGX Orin.

[0140] Optionally, the millimeter-wave radar module uses the TI IWR6843ISK-ODS.

[0141] The beneficial effects of this invention include:

[0142] (1) Visual and radar fusion monitoring: This invention combines a depth camera and millimeter-wave radar to achieve precise positioning of the human chest and dynamic, non-contact vital sign monitoring, effectively improving monitoring accuracy and anti-interference capability.

[0143] (2) Acceleration prediction and dynamic tracking: The chest movement acceleration is calculated through visual data to predict its real-time position, and the beam is dynamically adjusted in combination with millimeter-wave radar to ensure that high-quality breathing and heartbeat signals can be stably obtained even when the human body is moving.

[0144] (3) Seamless and highly adaptable: The system does not require the human body to wear sensors throughout the entire process, does not affect daily behavior, and can be applied to various scenarios such as family health management, smart elderly care and medical assistance, improving user comfort and application scope.

[0145] (4) Lightweight design and real-time performance: The device has a simple structure and lightweight algorithms, which are easy to deploy in real time on edge devices and meet the high reliability requirements of continuous monitoring in daily life.

[0146] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0147] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0148] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0149] The non-contact respiratory and heartbeat monitoring method, device, and program product combining visual-assisted positioning and millimeter-wave radar dynamic tracking provided in this application includes a depth camera module, an edge computing module, and a millimeter-wave radar module. The depth camera module collects human image data and transmits it back to the edge computing module. The edge computing module identifies key skeletal points, locates the chest position, and predicts its movement trajectory. The millimeter-wave radar module dynamically adjusts its beam based on the predicted information, continuously tracking and collecting signals from the chest region. Furthermore, the edge computing module also has signal processing capabilities, separating and extracting respiratory and heartbeat information to achieve non-contact monitoring of human vital signs. This invention is applicable to scenarios such as home health, smart elderly care, and medical assistance.

[0150] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0151] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0152] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0153] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0154] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0155] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0156] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0157] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0158] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0159] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0160] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A non-contact respiratory and heart rate monitoring device combining visual-assisted positioning and millimeter-wave radar dynamic tracking, characterized in that, The device includes a depth camera module, an edge computing module, and a millimeter-wave radar module; The depth camera module is used to collect human image data; The edge computing module is used to identify key skeletal points based on the human image data, locate the chest position, and predict the target's movement trajectory to obtain prediction information; The millimeter-wave radar module is used to dynamically adjust the beam according to the prediction information, and to track and collect signals from the chest region. The edge computing module separates the signals from the chest area and extracts breathing and heartbeat information to obtain non-invasive monitoring data of human vital signs.

2. The apparatus according to claim 1, characterized in that, The depth camera module is a structured light depth camera, a time-of-flight depth camera, or a binocular vision depth camera.

3. The apparatus according to claim 1, characterized in that, The edge computing module includes a deep learning model for human skeleton point recognition and a motion trajectory prediction unit; The human skeleton point recognition deep learning model is used to identify the chest region of the human body, obtain chest position information, and jointly calculate the spatial coordinates of the chest region with the depth camera module; the motion trajectory prediction unit is used to calculate the chest movement velocity and acceleration based on the chest position information, and predict the future position trajectory of the chest.

4. The apparatus according to claim 1, characterized in that, The millimeter-wave radar module adopts a multi-transmitter, multi-receiver antenna array structure and operates in the frequency range of 60 GHz to 81 GHz.

5. The apparatus according to claim 1, characterized in that, The millimeter-wave radar module controls the phase of each transmitting antenna to focus the transmitted radar signal on the predicted future location of the human chest; the millimeter-wave radar module uses beamforming technology at the receiving end to adjust the weighting coefficients of each receiving antenna so that the reflected signal in the direction of the predicted location maintains a distortion-free response.

6. The apparatus according to claim 1, characterized in that, The millimeter-wave radar module transmits a linear frequency modulated continuous wave signal, using the following formulas: S tx (t)=A·cos[2πf c t+πKt 2 +φ0]; Among them, S tx (t) represents the linear frequency modulated continuous wave signal; A represents the amplitude of the transmitted linear frequency modulated continuous wave signal; f c φ0 is the carrier frequency; K is the tuning frequency; φ0 is the initial phase; t is the transmission time.

7. The apparatus according to claim 1, characterized in that, The signal processing function within the edge computing module includes extracting the phase information of the reflected signal, processing the phase information, and separating the respiratory signal and heartbeat signal respectively using digital filtering technology.

8. The apparatus according to claim 7, characterized in that, The digital filtering technology employs a bandpass filtering method, with the filtering frequency range for the respiratory signal being 0.1Hz to 0.6Hz and the filtering frequency range for the heartbeat signal being 0.8Hz to 2Hz.

9. A non-contact respiratory and heart rate monitoring method combining visual-assisted positioning and millimeter-wave radar dynamic tracking, implemented by the device as described in any one of claims 1 to 8, characterized in that, The method includes the following steps: Use a depth camera module to collect human image data; The edge computing module is used to identify key skeletal points based on the human image data, locate the chest position, and predict the target's movement trajectory to obtain prediction information; The millimeter-wave radar module dynamically adjusts the beam based on the predicted information to track and acquire signals from the chest region. The edge computing module is used to separate the signals in the chest area and extract breathing and heartbeat information to obtain non-invasive monitoring data of human vital signs.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of claim 9.

Images