Vital sign detection method based on millimeter wave radar and camera fusion

Abstract

The present invention relates to the technical field of image processing, and provides a vital sign detection method based on the fusion of millimeter-wave radar and camera, comprising: performing posture detection based on a human body image of a target person to obtain a chest area of ​​the target person, performing frequency analysis on a millimeter-wave radar signal to obtain a distance angle heat map; performing human body positioning based on the chest area and the distance angle heat map to obtain a human body position; performing phase extraction on a millimeter-wave radar signal based on the human body position to obtain a vital sign signal of the target person, and performing variational modal decomposition on the vital sign signal to obtain vital sign parameters of the target person, thereby overcoming the defects that current detection is easily affected by human skin conditions, inaccurate detection results, and limitations in special populations, continuous monitoring, and remote monitoring, thereby realizing non-invasive detection, and the detection results are more accurate and reliable, more beneficial to special groups, and providing support for home medical care and remote health monitoring.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a vital sign detection method based on the fusion of millimeter wave radar and camera. Background Art

[0002] With the continuous development of medical technology, vital sign detection has become an indispensable part of clinical diagnosis and treatment, rehabilitation care, and daily health monitoring. However, current vital sign detection mainly relies on contact sensors, such as electrocardiographs, respiratory belts, and sphygmomanometers. These devices directly contact human skin to collect key physiological parameters such as heart rate, respiratory rate, and blood pressure. Although they are easy to operate and low in cost, they face many challenges in practical applications.

[0003] First, the contact detection method is particularly inconvenient for special groups of people. For example, burn patients have damaged skin, and the use of contact sensors for detection will not only increase the patient's pain, but may also cause complications such as infection. Secondly, since the contact detection method requires direct contact with human skin, it is easily affected by skin oils, sweat, etc., resulting in inaccurate detection results. In addition, since it requires real-time contact with human skin, contact sensors cannot achieve long-term continuous monitoring, nor can they meet the monitoring needs in scenarios such as telemedicine and home care. They have limitations in continuous monitoring and remote monitoring. Summary of the invention

[0004] The present invention provides a vital sign detection method based on the fusion of millimeter-wave radar and camera, which is used to overcome the defects of existing contact detection technology that is invasive and has limited application to specific groups of people, and to achieve accurate positioning of the human body and vital sign detection in complex scenarios.

[0005] The present invention provides a vital sign detection method based on the fusion of millimeter wave radar and camera, comprising:

[0006] Acquire the human body image and millimeter wave radar signal of the target person;

[0007] Performing posture detection based on the human body image to obtain the chest area of ​​the target person, and performing frequency analysis on the millimeter wave radar signal to obtain a distance angle heat map;

[0008] Performing human body positioning based on the chest area and the distance angle heat map to obtain a human body position;

[0009] Based on the human body position, the millimeter wave radar signal is phase extracted to obtain the vital sign signal of the target person, and the vital sign signal is subjected to variational mode decomposition to obtain the vital sign parameters of the target person.

[0010] According to a vital sign detection method based on millimeter wave radar and camera fusion provided by the present invention, the human body positioning is performed based on the chest area and the distance angle heat map to obtain the human body position, including:

[0011] Performing field of view angle matching on the chest area and the distance angle heat map;

[0012] Based on the chest area and distance angle heat map after field of view angle matching, the minimum variance distortion-free response beamformer is used to locate the human body and obtain the human body position;

[0013] The minimum variance distortionless response beamformer satisfies the following conditions:

[0014] ;

[0015] in, is the beamforming weight vector of the minimum variance distortionless response beamformer, A guiding vector indicating the direction of the target person, Indicates the direction of the target person, represents the conjugate transpose, represents the covariance matrix of the minimum variance distortionless response beamformer.

[0016] According to a vital sign detection method based on millimeter wave radar and camera fusion provided by the present invention, the phase extraction of the millimeter wave radar signal based on the human body position to obtain the vital sign signal of the target person includes:

[0017] Based on the human body position, performing phase extraction on the millimeter wave radar signal to obtain an initial vital sign signal of the target person;

[0018] The initial vital sign signal is subjected to phase difference processing to obtain a vital sign signal.

[0019] According to a vital sign detection method based on millimeter wave radar and camera fusion provided by the present invention, the phase difference processing of the initial vital sign signal to obtain the vital sign signal includes:

[0020] Performing phase difference processing on the initial vital sign signal to obtain a target vital sign signal;

[0021] filtering the target vital sign signal to remove environmental noise and obtain a vital sign signal;

[0022] The phase difference processing of the initial vital sign signal is implemented by the following formula:

[0023] ;

[0024] in, represents the target vital sign signal, represents the initial vital sign signal, Indicates that the initial vital sign signal is at time point The value of .

[0025] According to a vital sign detection method based on millimeter wave radar and camera fusion provided by the present invention, the vital sign signal is subjected to variational mode decomposition to obtain the vital sign parameters of the target person, including:

[0026] Performing variational mode decomposition on the vital sign signal to obtain vital sign signal components of the target person;

[0027] Performing Hilbert transform on the vital sign signal component to obtain the instantaneous frequency and marginal spectrum of the vital sign signal component;

[0028] Peak value extraction is performed based on the instantaneous frequency and the marginal spectrum to obtain vital sign parameters of the target person, where the vital sign parameters include heart rate and / or respiratory rate.

[0029] According to a vital sign detection method based on millimeter wave radar and camera fusion provided by the present invention, the posture detection is performed based on the human body image to obtain the chest area of ​​the target person, including:

[0030] Performing posture detection based on the human body image to obtain key points of the human body of the target person;

[0031] Determine a middle area based on the shoulder key points and the hip key points among the human body key points;

[0032] The chest area of ​​the target person is determined based on the center point of the middle area, the middle point of the line connecting the left shoulder key point and the right shoulder key point among the shoulder key points, and the middle point of the line connecting the right shoulder key point and the right hip key point among the hip key points.

[0033] The present invention also provides a vital sign detection device based on the fusion of millimeter wave radar and camera, comprising:

[0034] An acquisition unit, used for acquiring a human body image and a millimeter-wave radar signal of a target person;

[0035] An analysis unit, configured to perform posture detection based on the human body image to obtain a chest area of ​​the target person, and perform frequency analysis on the millimeter wave radar signal to obtain a distance angle heat map;

[0036] A positioning unit, used for positioning a human body based on the chest area and the distance angle heat map to obtain a human body position;

[0037] The detection unit is used to perform phase extraction on the millimeter wave radar signal based on the human body position to obtain the vital sign signal of the target person; and perform variational mode decomposition on the vital sign signal to obtain the vital sign parameters of the target person.

[0038] The present invention also provides a vital sign detection system based on the fusion of millimeter wave radar and camera, comprising a millimeter wave radar sensor, a camera, and a processor;

[0039] The processor is used to obtain a human body image of the target person captured by the camera and a millimeter-wave radar signal of the target person captured by the millimeter-wave radar sensor, and perform posture detection based on the human body image to obtain a chest area of ​​the target person, and perform frequency analysis on the millimeter-wave radar signal to obtain a distance angle heat map; perform human body positioning based on the chest area and the distance angle heat map to obtain a human body position, and based on the human body position, perform phase extraction on the millimeter-wave radar signal to obtain a vital sign signal of the target person; perform variational mode decomposition on the vital sign signal to obtain vital sign parameters of the target person.

[0040] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the processor executes the computer program, it implements any of the above-described vital sign detection methods based on the fusion of millimeter-wave radar and camera.

[0041] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described vital sign detection methods based on the fusion of millimeter-wave radar and camera.

[0042] The vital sign detection method based on the fusion of millimeter-wave radar and camera provided by the present invention performs posture detection based on the human body image of the target person to obtain the chest area of ​​the target person, performs frequency analysis on the millimeter-wave radar signal to obtain a distance angle heat map; performs human body positioning based on the chest area and the distance angle heat map to obtain the human body position; based on the human body position, performs phase extraction on the millimeter-wave radar signal to obtain the vital sign signal of the target person, and performs variational modal decomposition on the vital sign signal to obtain the vital sign parameters of the target person. The method overcomes the defects in the traditional scheme that the detection is easily affected by the human skin state, resulting in inaccurate detection results, and has obvious limitations in special populations, continuous monitoring and remote monitoring, realizes non-invasive detection, and the detection results are more accurate and reliable, which is more beneficial to special groups, and provides support for home medical care and remote health monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0044] Figure 1 It is a flow chart of a vital sign detection method based on millimeter wave radar and camera fusion provided by the present invention;

[0045] Figure 2 is an example diagram of key points of the human body provided by the present invention;

[0046] Figure 3 It is an overall framework diagram of the vital signs detection method based on millimeter wave radar and camera fusion provided by the present invention;

[0047] Figure 4 It is a structural schematic diagram of a vital sign detection device based on millimeter wave radar and camera fusion provided by the present invention;

[0048] Figure 5 It is a structural schematic diagram of a vital sign detection system based on millimeter wave radar and camera fusion provided by the present invention;

[0049] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] In recent years, the rapid development of non-contact radar detection technology has shown great potential in medical and monitoring applications, especially in breathing and heart rate detection. This technology has achieved a major breakthrough in long-term and long-distance detection. Compared with traditional contact detection methods, non-contact radar detection technology is highly concealed and non-invasive. It does not require wearing sensors or electrodes that directly contact the skin, reducing possible discomfort or skin irritation. It can continuously monitor the user's vital signs without interfering with the user, which is especially important for burn patients and the elderly.

[0052] Based on this, the present invention provides a vital signs detection method based on the fusion of millimeter-wave radar and camera, which aims to perform non-contact vital signs detection on a target person through the body image and millimeter-wave radar signal of the target person to obtain the vital signs parameters of the target person, thereby realizing fast and accurate vital signs detection, ensuring the accuracy and reliability of the vital signs parameters, avoiding the problems of inaccurate detection results and limited application in special populations, continuous monitoring and remote monitoring in traditional contact detection methods, and improving the accuracy, reliability and scope of application of vital signs detection.

[0053] Figure 1 : is a flow chart of a vital sign detection method based on millimeter wave radar and camera fusion provided by the present invention, such as Figure 1 As shown, the method includes:

[0054] Step 110, acquiring a human body image and a millimeter wave radar signal of a target person;

[0055] Step 120, performing posture detection based on the human body image to obtain the chest area of ​​the target person, performing frequency analysis on the millimeter wave radar signal to obtain a distance angle heat map;

[0056] Step 130, performing human body positioning based on the chest area and distance angle heat map to obtain the human body position;

[0057] Step 140, based on the human body position, phase extraction is performed on the millimeter wave radar signal to obtain the vital sign signal of the target person, and variational mode decomposition is performed on the vital sign signal to obtain the vital sign parameters of the target person.

[0058] Specifically, considering that the contact detection method is easily affected by the human skin condition, resulting in inaccurate detection results, and has obvious limitations in special populations, continuous monitoring and remote monitoring. In the embodiment of the present invention, it is proposed to detect vital signs based on a non-contact detection method. Compared with the contact detection method, the non-contact detection uses vision, radar and other technologies to capture the physiological signals of the human body without contacting the human body, thereby realizing real-time monitoring of the vital signs of the target person. It not only avoids the inconvenience and discomfort of wearing electrodes or other sensing devices during monitoring, but is also suitable for people and scenes where traditional detection methods cannot be used, and has a very wide range of applications.

[0059] It is understandable that before performing vital signs detection, the first thing that needs to be determined is the person who needs to undergo vital signs detection, that is, the target person, and the data used for the detection, that is, the vital signs data of the target person, needs to be obtained. The vital signs data here include the human body image and millimeter wave radar signal of the target person, both of which are collected in a non-contact manner. Specifically, the target person can be imaged by a camera to obtain an image containing the torso of the target person, that is, the human body image of the target person. The camera used here can be an ordinary camera, a depth camera, or a camera of other types and models, and the embodiment of the present invention does not specifically limit this. After the human body image is obtained by collecting the camera, considering that the camera only has an advantage in angular resolution, but lacks in depth, in the embodiment of the present invention, in order to achieve more accurate vital signs detection, it is also necessary to obtain a radar signal with a clear advantage in depth resolution.

[0060] It is worth noting that non-contact detection methods have significant advantages over traditional contact detection. First, non-contact detection methods avoid the discomfort that may be caused by contact sensing equipment and improve the user's monitoring compliance, which is especially valuable in long-term monitoring. Secondly, non-contact detection methods are more suitable for various special populations, such as newborns, burn patients, skin allergies, infectious disease patients, etc., avoiding the risk of secondary injury or infection that may be caused by traditional detection methods. Finally, non-contact detection methods are continuous and can continuously monitor the user's health status. For patients with chronic diseases such as cardiovascular disease and respiratory problems, they can track changes in their condition in a timely manner, detect abnormalities in a timely manner and respond.

[0061] Millimeter wave radar sensor, as one of the non-contact detection devices, is used in the electromagnetic wave frequency of the millimeter wave band, and can emit electromagnetic waves to the target and receive electromagnetic waves reflected by the target. When these electromagnetic waves encounter a moving target, the frequency changes due to the Doppler effect. By analyzing the Doppler frequency shift of the electromagnetic waves emitted by the millimeter wave radar sensor and the reflected electromagnetic waves, the speed and direction of movement of the target can be identified and determined. When the target is a human body, it can also detect the subtle vibrations of the chest wall in the human chest cavity. These vibrations are closely related to the chest wall fluctuations caused by the heartbeat and breathing of the human body. Therefore, the millimeter wave radar signal scanned by it can accurately reflect the vital signs of the human body, and can be used as a better choice of radar signals in the embodiments of the present invention.

[0062] After obtaining the human body image and millimeter-wave radar signal of the target person, in an embodiment of the present invention, vital signs detection can be performed based on the two. Specifically, human body area detection can be first performed using the human body image to determine the various areas in the target person's body. Since the human body's vital signs information is mainly presented through the movement and state of the heart and lungs in the chest cavity, when performing human body area detection here, the chest area of ​​the target person is mainly determined, that is, posture detection is performed based on the human body image to judge the various areas in the human body according to the posture of the human body, thereby locating the chest position and obtaining the chest area of ​​the target person.

[0063] At the same time, the millimeter-wave radar signal can be processed to generate a distance angle heat map, which can reflect the dynamic changes of the target person's chest wall movement. Based on this and the chest area of ​​the target person obtained by posture detection, the target person can be positioned to obtain an accurate and reliable human body position.

[0064] Specifically, when using millimeter-wave radar to scan a target person, it works by emitting electromagnetic waves and capturing the signals reflected by these electromagnetic waves from objects. These reflected waves contain important information about the position and speed of the object. Therefore, in order to extract this information, after receiving the millimeter-wave radar signal, a series of signal preprocessing can be performed on it, such as amplifying the signal and using filters to reduce noise to remove background noise, so as to ensure the clarity and accuracy of the signal; then, the preprocessed signal can be subjected to frequency analysis using fast Fourier transform to analyze its frequency components, so as to construct a distance angle heat map. That is, since the time delay of the radar signal (the time it takes for the signal to be emitted to be captured by the receiver) is a key factor in calculating the distance, and because of the Doppler effect, the movement of the object will cause the frequency of the reflected signal to change, and this frequency change can be used to calculate the speed. Once the signal is processed and converted into data, these data can be used to construct a distance angle heat map, which can be used to intuitively observe and analyze the distribution and dynamic changes of objects in space.

[0065] Furthermore, after obtaining the distance angle heat map and the chest area of ​​the target person, in an embodiment of the present invention, the distance angle heat map and the chest area can be combined to perform human body positioning to determine the position of the target person, that is, the human body position. Here, specifically, the chest area of ​​the target person obtained by posture detection can be used to assist the distance angle heat map in human body positioning, so as to determine the human body position in the spatial area through millimeter wave radar beamforming, thereby obtaining the human body position of the target person, and then facilitating the detection of human chest wall movement. Here, the advantages of the high angular resolution of the camera and the high depth resolution of the millimeter wave radar are integrated to perform human body positioning, which can achieve precise positioning of the target person and obtain accurate and reliable human body position.

[0066] After that, the millimeter-wave radar signal can be processed according to the position of the human body to extract signals that can characterize the vital signs of the target person, that is, the vital signs signal, and decompose the vital signals to achieve the extraction of effective vital signs information and obtain the vital signs information of the target person. This information can be represented by specific parameters, such as heart rate, respiratory rate, etc. Therefore, the vital signs information extracted here can also be called the vital signs parameters of the target person.

[0067] Specifically, here, the phase of the millimeter-wave radar signal can be extracted by using the human body position to obtain the vital sign signal of the target person. In this process, by analyzing the phase change of the corresponding reflected wave, the movement change of the human chest wall can be more accurately measured, which helps to accurately detect the vital signs of the target person. Then, the extracted vital sign signal is subjected to variational mode decomposition (VMD) to decompose the vital sign signal, thereby obtaining the vital sign information of the target person.

[0068] In the embodiments of the present invention, non-contact vital signs detection is achieved through radar and camera. Compared with the traditional contact detection method, it has the advantages of being non-invasive, non-interference, continuous monitoring, and remote monitoring. That is, there is no need to wear sensors or electrodes that contact the skin, which reduces the possible discomfort or skin irritation, and can continuously monitor vital signs without interfering with the target person; and, thanks to the high angular resolution of the camera, closely arranged targets can be identified, thereby achieving accurate detection of complex scenes and better adapting to special populations and diversified scenes.

[0069] The vital sign detection method based on the fusion of millimeter-wave radar and camera provided by the present invention performs posture detection based on the human body image of the target person to obtain the chest area of ​​the target person, performs frequency analysis on the millimeter-wave radar signal to obtain a distance angle heat map; performs human body positioning based on the chest area and the distance angle heat map to obtain the human body position; based on the human body position, performs phase extraction on the millimeter-wave radar signal to obtain the vital sign signal of the target person, and performs variational modal decomposition on the vital sign signal to obtain the vital sign parameters of the target person. The method overcomes the defects in the traditional scheme that the detection is easily affected by the human skin state, resulting in inaccurate detection results, and has obvious limitations in special populations, continuous monitoring and remote monitoring, realizes non-invasive detection, and the detection results are more accurate and reliable, which is more beneficial to special groups, and provides support for home medical care and remote health monitoring.

[0070] Based on the above embodiment, step 130 includes:

[0071] Match the field of view angle between the chest area and the distance angle heat map;

[0072] Based on the chest area and distance angle heat map after field of view angle matching, the human body is located through the minimum variance distortion-free response beamformer to obtain the human body position.

[0073] Specifically, in step 130, the process of locating the human body according to the chest area and the distance angle heat map to obtain the human body position may specifically include:

[0074] After determining the distance angle heat map for the target person, the position of the target person can be located by fusing different contours and peaks in the distance heat map. However, in this process, considering the different field angles of view (FOV) of different devices, the field angle of the camera is usually in the range of -60 degrees to +60 degrees, which means that the camera can capture image information within this range, while the field angle of the millimeter wave radar is usually wider, that is, -90 degrees to +90 degrees, and it can perceive a wider area.

[0075] However, in actual experiments, the main lobe of the radar transmitter (i.e. the area with the strongest signal) is usually only focused within a range of about ±60 degrees. This means that although it can theoretically cover a wider angle, the most effective and accurate monitoring area in practical applications is within the range of ±60 degrees. The beam gain in other directions is significantly lower than the gain of the main lobe, resulting in its detection in other directions not having high reliability. The reason is that the signal strength is reduced, which increases errors and noise.

[0076] In view of this, in an embodiment of the present invention, when using the information obtained by the camera and the heat map obtained by the millimeter wave radar to locate the human body, the field of view angle can be adjusted first to achieve the matching of the field of view angle, and then the human body can be located through the minimum variance distortion-free response beamformer based on the chest area and distance angle heat map after the field of view angle matching to obtain the human body position.

[0077] Specifically, in order to achieve the best field of view match between the camera and the radar, the search range of the millimeter-wave radar's angle of arrival (AoA) can be set, that is, the search range of the millimeter-wave radar can be set to be consistent with the field of view of the camera, thereby achieving an accurate match between the chest area and the distance angle heat map. Here, specifically, the search range of the millimeter-wave radar's angle of arrival (AoA) can be set to ±60 degrees, and the angle grid can be set to ,in, is the angle step, is the number of angular units; when the given search range is ±60 degrees and the total resolution needs to be angle unit, there is In this way, the field of view of the camera and the millimeter-wave radar can be matched to the maximum extent, that is, the field of view of the chest area and the distance angle heat map can be matched to ensure that the two are highly consistent within the coverage range, so that effective data fusion can be achieved between the camera and the millimeter-wave radar, thereby improving the accuracy of human body positioning.

[0078] Furthermore, in order to solve the problem of low gain in the non-main lobe direction mentioned above, in an embodiment of the present invention, a minimum variance distortionless response beamformer MVDR (Minimum Variance Distortionless Response) is introduced to enhance effective information and suppress noise through MVDR, thereby improving the detection capability in the non-main lobe direction. Specifically, MVDR can form a specific beam by controlling the transmitter, and dynamically adjust the beam shape according to the interference and noise conditions in the environment, thereby minimizing the impact of interference and noise while maintaining high-resolution positioning of the target person. In this process, the phase of the transmitted signal on each antenna is precisely controlled to change the direction of the main beam lobe to point to the area of ​​interest, and the position of the target person is calculated, and the beam direction is adjusted accordingly. In order to accurately point the beam in the target direction , the target direction refers to the direction where the target person is located, and the transmit beamforming weight vector can be defined first ,This weight vector is calculated based on the direction of the target person and the physical distance between the antennas. Each element in the weight vector represents the phase adjustment value of the signal on the corresponding antenna.

[0079] Here, the transmit beamforming weight vector It can be expressed by the following formula:

[0080] ;

[0081] In the formula, represents the imaginary unit, , used for complex number operations; Represents the distance between adjacent antennas in the antenna array of millimeter-wave radar, that is, the antenna array element spacing; Represents the working wavelength of the millimeter-wave radar, that is, the signal wavelength of the millimeter-wave radar signal ( , With signal frequency and propagation speed related).

[0082] Then, the original transmitted signal can be Multiply , to obtain the beamformed signal. Since the goal of MVDR is to minimize the output power while ensuring no distortion in the target direction, thereby maximizing the suppression of noise and interference, when beamforming, given the covariance matrix of the received signal and the steering vector in the target direction, it needs to meet the following conditions:

[0083] ;

[0084] In the formula, is the beamforming weight vector of the minimum variance distortionless response beamformer, The steering vector representing the target direction, Indicates the target direction, represents the conjugate transpose, Represents the covariance matrix of the minimum variance distortionless response beamformer.

[0085] Here, Ensures the direction of the target The signal has no distortion response (gain is 1), and at the same time, through the inverse covariance matrix Noise and interference suppression can be achieved.

[0086] Among them, the original transmission signal can be expressed by the following formula:

[0087] ;

[0088] ;

[0089] In the formula, and Respectively represent the complex signal forms of the original transmitted signal at the 0th and 1st antenna elements of the antenna array; Represents the amplitude of the original transmitted signal (amplitude, usually a positive real number, determines the signal strength); Indicates the carrier frequency of the original transmitted signal (Carrier Frequency), in Hz; represents the time variable; Represents the bandwidth of the original transmitted signal, usually associated with a linear frequency modulation signal; Indicates the time period or modulation duration of the original transmitted signal; Indicates the initial phase of the original transmitted signal, which is used to specify the starting phase angle of the original transmitted signal.

[0090] Through the above process, the main beam can be concentrated in a specific direction, thereby increasing the signal gain in that direction and reducing signal leakage in other directions. Specifically, it can achieve distortion-free positioning in a specific direction, and effectively enhance the signal in the non-main lobe direction to improve the detection capability in the non-main lobe direction; at the same time, it can suppress noise and interference, reduce the impact of multipath and clutter on positioning accuracy, and improve positioning precision and accuracy. Among them, the phase adjustment of each antenna element depends not only on the angular position of the target person, but also on the antenna spacing, beam width and required beam shape. The combination of multiple factors can ensure that the beam can accurately and effectively point to the target direction.

[0091] Based on the above embodiment, in step 140, based on the position of the human body, the phase of the millimeter wave radar signal is extracted to obtain the vital sign signal of the target person, including:

[0092] Based on the position of the human body, the phase of the millimeter-wave radar signal is extracted to obtain the initial vital sign signal of the target person;

[0093] The initial vital sign signal is subjected to phase difference processing to obtain a vital sign signal.

[0094] Specifically, in step 140, the process of performing phase extraction on the millimeter wave radar signal according to the human body position to obtain the initial vital sign signal of the target person may specifically include:

[0095] First, the position of the human body can be used to extract the phase of the millimeter-wave radar signal. Specifically, after determining the position of the human body, the phase change of the millimeter-wave radar signal can be analyzed based on this, so as to more accurately measure the movement of the chest wall. Since the chest displacement movement is very large relative to the wavelength of the millimeter wave, the extended differential and cross multiplication DACM (differentiate and cross multiply) can be used here for phase unwrapping analysis to avoid phase ambiguity caused by phase discontinuity and jump, resulting in signal quality degradation or information loss, thereby improving the signal quality, extracting phase information more accurately, and obtaining the initial vital sign signal.

[0096] Next, considering that the heartbeat frequency used to focus on characterizing vital signs is usually very small, therefore, in an embodiment of the present invention, in order to subsequently decompose the vital sign signal and obtain accurate and effective vital sign parameters, after obtaining the initial vital sign signal, the initial vital sign signal can be phase-differentially processed, that is, the difference between adjacent continuous phase values ​​is implemented to enhance the heartbeat signal, thereby obtaining an enhanced initial vital sign signal, that is, a vital sign signal.

[0097] In an embodiment of the present invention, phase information is extracted from the millimeter-wave radar signal based on the position of the human body to obtain an initial vital sign signal. In order to remove the fixed phase offset introduced by the environment and system errors, the initial vital sign signal is further subjected to phase differential processing to effectively eliminate the static background and obtain a pure vital sign signal.

[0098] Based on the above embodiment, the initial vital sign signal is subjected to phase difference processing to obtain a vital sign signal, including:

[0099] Performing phase difference processing on the initial vital sign signal to obtain the target vital sign signal;

[0100] The target vital sign signal is subjected to environmental noise filtering to obtain a vital sign signal.

[0101] Specifically, the process of performing phase differential processing on the initial vital sign signal to obtain the vital sign signal includes the following steps:

[0102] First, the initial vital sign signal can be subjected to phase differential processing, where the phase differential processing means that the difference between adjacent continuous phase values ​​is implemented to enhance the heartbeat signal, thereby obtaining an enhanced initial vital sign signal; however, since the signal obtained at this time may also contain some environmental noise, such as wind noise, background conversation noise, noise generated by other equipment, etc., therefore, in order to ensure the accuracy of subsequent detection, after the phase differential enhancement, it is necessary to further process it in the embodiment of the present invention, and the signal obtained at this time is called the target vital sign signal.

[0103] Then, the target vital sign signal can be filtered to remove environmental noise, thereby obtaining a vital sign signal; here, specifically, an adaptive filter can be used to filter the target vital sign signal to remove environmental noise. Since the adaptive filter can automatically adjust its performance according to the input signal, filtering based on it can effectively reduce the impact of environmental noise on the signal and obtain an accurate vital sign signal. Here, the adaptive filter can maintain high efficiency in a dynamic or unstable environment due to its environmental adaptability, that is, it can automatically adjust according to environmental changes.

[0104] Here, the phase difference processing of the initial vital sign signal is realized by the following formula:

[0105] ;

[0106] in, Indicates the target vital sign signal, Indicates the initial sign signal, Indicates the initial vital sign signal at time point The value of Represents a time difference (usually used to define a signal sampling interval or time delay).

[0107] Based on the above embodiment, in step 140, the vital sign signal is subjected to variational mode decomposition to obtain the vital sign parameters of the target person, including:

[0108] Perform variational mode decomposition on the vital sign signal to obtain the vital sign signal components of the target person;

[0109] Performing Hilbert transformation on the vital sign signal component to obtain the instantaneous frequency and marginal spectrum of the vital sign signal component;

[0110] Peak value extraction is performed based on the instantaneous frequency and the marginal spectrum to obtain the vital sign parameters of the target person, where the vital sign parameters include heart rate and / or respiratory rate.

[0111] Specifically, the process of performing variational mode decomposition on the vital sign signal to obtain the vital sign parameters of the target person may specifically include:

[0112] First, the vital sign signal can be subjected to variational mode decomposition to obtain the vital sign signal components of the target person; here, variational mode decomposition VMD can be introduced to decompose the vital sign signal to decompose the complex signal into a series of intrinsic mode functions (IMFs), thereby obtaining multiple vital sign signal components of the target person. By analyzing these decomposed frequency components, the respiratory and heartbeat signal components can be accurately identified according to the frequency range, thereby realizing the extraction of heartbeat and respiratory signals.

[0113] Here, the signal is decomposed by variational mode decomposition, which can effectively separate the effective components and noise in the signal, thereby helping to improve the clarity and quality of the signal, and is very suitable for the analysis of non-stationary and nonlinear signals.

[0114] Then, the vital sign signal components can be Hilbert transformed to obtain the instantaneous frequency and marginal spectrum of the vital sign signal components. Specifically, the respiratory signal component and the heartbeat signal component in the vital sign signal components are Hilbert transformed respectively to obtain the instantaneous frequency and calculate the marginal spectra corresponding to the respiration and heartbeat.

[0115] Afterwards, the instantaneous frequency and marginal spectrum obtained in the previous step can be subjected to peak extraction, and the extracted peak is the final result of vital sign detection, that is, the vital sign parameters of the target person, specifically including the heart rate and / or respiratory rate of the target person.

[0116] In the examples of the present invention, the useful signal and noise in the radar signal are effectively separated through variational mode decomposition processing, the clarity and quality of the radar signal are improved, and further in complex scenes, it is possible to distinguish multiple overlapping or close targets, improve the resolution and target recognition capabilities, and extract subtle signal changes caused by micro-motion.

[0117] Based on the above embodiment, the process of performing Hilbert transform on the vital sign signal component can be expressed by the following formula:

[0118] ;

[0119] In the formula, Represents the vital sign signal component (Intrinsic Mode Function, IMF); It represents Hilbert transform. Hilbert transform expands real signal into complex signal and analyzes its instantaneous frequency and phase characteristics. The result of the transformation is a complex signal. represents the complex envelope signal after Hilbert transform, is the amplitude part, is the phase factor, which indicates the phase information of the signal; Represents the imaginary unit, in the complex field ; Represents the instantaneous phase, which is the complex number (real part) obtained after the Hilbert transform of the signal +Imaginary part ) is used to reflect the phase information of the signal changing with time, which can be expressed as:

[0120] ;

[0121] In an embodiment of the present invention, by performing variational modal decomposition on the vital sign signal, performing Hilbert transform on the decomposed vital sign signal components, extracting the instantaneous frequency and marginal spectrum, and analyzing the peak value to obtain the heart rate and / or respiratory rate, high-precision vital sign parameters are obtained, thereby ensuring accurate and reliable vital sign detection.

[0122] Based on the above embodiment, in step 120, posture detection is performed based on the human body image to obtain the chest area of ​​the target person, including:

[0123] Perform posture detection based on human body images to obtain the key points of the target person;

[0124] Based on the shoulder key points and the hip key points in the human body key points, determine the middle area;

[0125] The chest area of ​​the target person is determined based on the center point of the middle area, the middle point of the line connecting the left shoulder key point and the right shoulder key point in the shoulder key points, and the middle point of the line connecting the right hip key point in the right shoulder key point and hip key points.

[0126] Specifically, in step 120, the process of performing posture detection according to the human body image to obtain the chest area of ​​the target person may specifically include:

[0127] First, posture detection can be performed on the human body image of the target person to obtain the human body key points of the target person; here, posture detection is implemented with the help of YOLOv8 (You Only Look Once version 8), that is, the human body image can be input into the posture detection model constructed on the basis of YOLOv8, so that the posture detection model performs posture detection on the input human body image and outputs the human body posture and key point positions.

[0128] Here, it should be noted that compared with YOLOv5 (You Only Look Once version 5), YOLOv8 has significantly improved efficiency and flexibility. At the same time, it also maintains low resource consumption, versatility and strong task adaptability. It can perform well in posture detection tasks and obtain accurate human key points.

[0129] Figure 2 is an example diagram of the key points of the human body provided by the present invention, such as Figure 2 As shown, multiple human key points can be obtained through posture detection, and each human key point corresponds to a human body part, such as key point 1 corresponds to the left eye of the human body, key point 2 corresponds to the right eye of the human body, key point 3 corresponds to the left ear of the human body, key point 5 corresponds to the left shoulder of the human body, key point 7 corresponds to the left elbow of the human body, key point 9 corresponds to the left wrist of the human body, key point 11 corresponds to the left hip of the human body, etc. After obtaining multiple human key points, in the embodiment of the present invention, the human torso, that is, the middle area of ​​the human body, can be determined based on the shoulder key points and hip key points among the human key points. Figure 2 It can be seen that the middle area of ​​the human body can be accurately located according to the left shoulder key point 5, the right shoulder key point 6, the left hip key point 11 and the right hip key point 12.

[0130] Then, the chest can be located based on the center point of the middle area, the middle point of the line connecting the left shoulder key point and the right shoulder key point in the shoulder key points, and the middle point of the line connecting the right shoulder key point and the right hip key point, so as to obtain the chest area of ​​the target person. That is, the area is divided with the center point, the middle point of the line connecting the left shoulder key point and the right shoulder key point, and the middle point of the line connecting the right shoulder key point and the right hip key point as the vertices, so as to delineate a square area in the middle area, which is the chest area, that is, Figure 2 The red area in.

[0131] Figure 3 is the overall framework diagram of the vital signs detection method based on millimeter wave radar and camera fusion provided by the present invention, such as Figure 3 As shown, the method specifically comprises the following steps:

[0132] First, the human body image and millimeter-wave radar signal of the target person are acquired.

[0133] Next, posture detection is performed based on the human body image to obtain the human body key points of the target person. The middle area is determined based on the shoulder key points and hip key points among the human body key points, and the chest area of ​​the target person is determined based on the center point of the middle area, the middle point of the line connecting the left shoulder key point and the right shoulder key point among the shoulder key points, and the middle point of the line connecting the right hip key point among the right shoulder key points and hip key points.

[0134] At the same time, the frequency analysis of the millimeter-wave radar signal is performed to obtain a distance angle heat map.

[0135] Then, the field of view angle of the chest area and the distance angle heat map are matched, and based on the chest area and the distance angle heat map after the field of view angle matching, the human body is located through the minimum variance distortion-free response beamformer to obtain the human body position.

[0136] Afterwards, based on the position of the human body, the phase of the millimeter-wave radar signal is extracted to obtain the initial vital sign signal of the target person, and the initial vital sign signal is phase-differentially processed to obtain the target vital sign signal. The environmental noise of the target vital sign signal is filtered out to obtain the vital sign signal.

[0137] Finally, the vital sign signal is subjected to variational mode decomposition to obtain the vital sign signal components of the target person; the vital sign signal components are subjected to Hilbert transform to obtain the instantaneous frequency and marginal spectrum of the vital sign signal components; peak extraction is performed based on the instantaneous frequency and marginal spectrum to obtain the vital sign parameters of the target person, which include heart rate and / or respiratory rate.

[0138] The method provided by the embodiment of the present invention performs posture detection on the human body image of the target person to obtain the chest area of ​​the target person, combines it with the distance angle heat map for the target person determined by the millimeter wave radar signal, and uses the minimum variance distortionless response beamformer to locate the human body, which can effectively solve the problems of surrounding environmental influence and multi-person interference. On this basis, phase extraction is performed using extended differentiation and cross multiplication, and vital sign signal decomposition is performed through variational mode decomposition, which can achieve fast and accurate vital sign detection and obtain accurate and reliable vital sign parameters.

[0139] The following is a description of the vital signs detection device based on the fusion of millimeter-wave radar and camera provided by the present invention. The vital signs detection device based on the fusion of millimeter-wave radar and camera described below and the vital signs detection method based on the fusion of millimeter-wave radar and camera described above can be referenced to each other.

[0140] Figure 4 is a structural schematic diagram of a vital sign detection device based on millimeter wave radar and camera fusion provided by the present invention, such as Figure 4 As shown, the device comprises:

[0141] An acquisition unit 410 is used to acquire a human body image and a millimeter-wave radar signal of a target person;

[0142] An analysis unit 420 is used to perform posture detection based on the human body image to obtain a chest area of ​​the target person, and perform frequency analysis on the millimeter wave radar signal to obtain a distance angle heat map;

[0143] A positioning unit 430, configured to perform human body positioning based on the chest area and the distance angle heat map to obtain a human body position;

[0144] The detection unit 440 is used to perform phase extraction on the millimeter-wave radar signal based on the human body position to obtain the vital sign signal of the target person; and perform variational mode decomposition on the vital sign signal to obtain the vital sign parameters of the target person.

[0145] The vital sign detection device based on the fusion of millimeter-wave radar and camera provided by the present invention performs posture detection based on the human body image of the target person to obtain the chest area of ​​the target person, performs frequency analysis on the millimeter-wave radar signal to obtain a distance angle heat map; performs human body positioning based on the chest area and the distance angle heat map to obtain the human body position; based on the human body position, performs phase extraction on the millimeter-wave radar signal to obtain the vital sign signal of the target person, and performs variational modal decomposition on the vital sign signal to obtain the vital sign parameters of the target person, thereby overcoming the defects in the traditional scheme that the detection is easily affected by the human skin state, resulting in inaccurate detection results, and having obvious limitations in special populations, continuous monitoring and remote monitoring, thereby realizing non-invasive detection, and the detection results are more accurate and reliable, more beneficial to special groups, and providing support for home medical care and remote health monitoring.

[0146] Based on the above embodiment, the positioning unit 430 is used for:

[0147] Performing field of view angle matching on the chest area and the distance angle heat map;

[0148] Based on the chest area and distance angle heat map after field of view angle matching, the minimum variance distortion-free response beamformer is used to locate the human body and obtain the human body position;

[0149] The minimum variance distortionless response beamformer satisfies the following conditions:

[0150] ;

[0151] in, is the beamforming weight vector of the minimum variance distortionless response beamformer, A guiding vector indicating the direction of the target person, Indicates the direction of the target person, represents the conjugate transpose, represents the covariance matrix of the minimum variance distortionless response beamformer.

[0152] Based on the above embodiment, the detection unit 440 is used for:

[0153] Based on the human body position, performing phase extraction on the millimeter wave radar signal to obtain an initial vital sign signal of the target person;

[0154] The initial vital sign signal is subjected to phase difference processing to obtain a vital sign signal.

[0155] Based on the above embodiment, the detection unit 440 is used for:

[0156] Performing phase difference processing on the initial vital sign signal to obtain a target vital sign signal;

[0157] filtering the target vital sign signal to remove environmental noise and obtain a vital sign signal;

[0158] The phase difference processing of the initial vital sign signal is implemented by the following formula:

[0159] ;

[0160] in, represents the target vital sign signal, represents the initial vital sign signal, Indicates that the initial vital sign signal is at time point The value of .

[0161] Based on the above embodiment, the detection unit 440 is used for:

[0162] Performing variational mode decomposition on the vital sign signal to obtain vital sign signal components of the target person;

[0163] Performing Hilbert transform on the vital sign signal component to obtain the instantaneous frequency and marginal spectrum of the vital sign signal component;

[0164] Peak value extraction is performed based on the instantaneous frequency and the marginal spectrum to obtain vital sign parameters of the target person, where the vital sign parameters include heart rate and / or respiratory rate.

[0165] Based on the above embodiment, the analysis unit 420 is used for:

[0166] Performing posture detection based on the human body image to obtain key points of the human body of the target person;

[0167] Determine a middle area based on the shoulder key points and the hip key points among the human body key points;

[0168] The chest area of ​​the target person is determined based on the center point of the middle area, the middle point of the line connecting the left shoulder key point and the right shoulder key point among the shoulder key points, and the middle point of the line connecting the right shoulder key point and the right hip key point among the hip key points.

[0169] Figure 5 : is a structural schematic diagram of a vital sign detection system based on millimeter wave radar and camera fusion provided by the present invention, such as Figure 5 As shown, the system includes a millimeter wave radar sensor 510, a camera 520, and a processor 530;

[0170] The processor 530 is used to obtain the human body image of the target person collected by the camera 520 and the millimeter-wave radar signal of the target person collected by the millimeter-wave radar sensor 510, and perform posture detection based on the human body image to obtain the chest area of ​​the target person, perform frequency analysis on the millimeter-wave radar signal to obtain a distance angle heat map; perform human body positioning based on the chest area and the distance angle heat map to obtain the human body position, and based on the human body position, perform phase extraction on the millimeter-wave radar signal to obtain the vital sign signal of the target person; perform variational mode decomposition on the vital sign signal to obtain the vital sign parameters of the target person.

[0171] The vital sign detection system based on the fusion of millimeter-wave radar and camera provided by the present invention comprises a millimeter-wave radar sensor, a camera and a processor, wherein the processor is used to obtain a human body image of a target person collected by the camera and a millimeter-wave radar signal of the target person collected by the millimeter-wave radar sensor, and perform posture detection based on the human body image to obtain the chest area of ​​the target person, perform frequency analysis on the millimeter-wave radar signal to obtain a distance angle heat map; perform human body positioning based on the chest area and the distance angle heat map to obtain the human body position, and perform phase extraction on the millimeter-wave radar signal based on the human body position to obtain the vital sign signal of the target person; perform variational mode decomposition on the vital sign signal to obtain the vital sign parameters of the target person, thereby overcoming the defects in the traditional scheme that the detection is easily affected by the human skin state, resulting in inaccurate detection results, and having obvious limitations in special populations, continuous monitoring and remote monitoring, realizing non-invasive detection, and the detection results are more accurate and reliable, more beneficial to special groups, and providing support for home medical care and remote health monitoring.

[0172] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute a vital sign detection method based on the fusion of millimeter wave radar and camera, the method comprising: acquiring a human body image and a millimeter wave radar signal of a target person; performing posture detection based on the human body image to obtain a chest area of ​​the target person, performing frequency analysis on the millimeter wave radar signal to obtain a distance angle heat map; performing human body positioning based on the chest area and the distance angle heat map to obtain a human body position; performing phase extraction on the millimeter wave radar signal based on the human body position to obtain a vital sign signal of the target person, and performing variational mode decomposition on the vital sign signal to obtain a vital sign parameter of the target person.

[0173] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0174] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the vital signs detection method based on millimeter-wave radar and camera fusion provided by the above-mentioned methods, and the method includes: acquiring a human body image and a millimeter-wave radar signal of a target person; performing posture detection based on the human body image to obtain a chest area of ​​the target person, performing frequency analysis on the millimeter-wave radar signal to obtain a distance angle heat map; performing human body positioning based on the chest area and the distance angle heat map to obtain a human body position; based on the human body position, performing phase extraction on the millimeter-wave radar signal to obtain a vital signs signal of the target person, and performing variational mode decomposition on the vital signs signal to obtain vital signs parameters of the target person.

[0175] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the vital signs detection method based on millimeter-wave radar and camera fusion provided by the above-mentioned methods, the method comprising: acquiring a human body image and a millimeter-wave radar signal of a target person; performing posture detection based on the human body image to obtain a chest area of ​​the target person, performing frequency analysis on the millimeter-wave radar signal to obtain a distance angle heat map; performing human body positioning based on the chest area and the distance angle heat map to obtain a human body position; performing phase extraction on the millimeter-wave radar signal based on the human body position to obtain a vital signs signal of the target person, and performing variational mode decomposition on the vital signs signal to obtain vital signs parameters of the target person.

[0176] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0177] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0178] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A vital sign detection method based on millimeter wave radar and camera fusion, characterized in that: include: Acquire the human body image and millimeter wave radar signal of the target person; Performing posture detection based on the human body image to obtain the chest area of ​​the target person, and performing frequency analysis on the millimeter wave radar signal to obtain a distance angle heat map; Performing human body positioning based on the chest area and the distance angle heat map to obtain a human body position; Based on the human body position, performing phase extraction on the millimeter wave radar signal to obtain a vital sign signal of the target person, and performing variational mode decomposition on the vital sign signal to obtain a vital sign parameter of the target person; The performing posture detection based on the human body image to obtain the chest area of ​​the target person includes: Performing posture detection based on the human body image to obtain key points of the human body of the target person; Determine a middle area based on the shoulder key points and the hip key points among the human body key points; The chest area of ​​the target person is determined based on the center point of the middle area, the middle point of the line connecting the left shoulder key point and the right shoulder key point among the shoulder key points, and the middle point of the line connecting the right shoulder key point and the right hip key point among the hip key points.

2. The vital sign detection method based on millimeter wave radar and camera fusion according to claim 1 is characterized in that: The method of performing human body positioning based on the chest area and the distance angle heat map to obtain the human body position includes: Performing field of view angle matching on the chest area and the distance angle heat map; Based on the chest area and distance angle heat map after field of view angle matching, the minimum variance distortion-free response beamformer is used to locate the human body and obtain the human body position; The minimum variance distortionless response beamformer satisfies the following conditions: ； in, is the beamforming weight vector of the minimum variance distortionless response beamformer, A guiding vector indicating the direction of the target person, Indicates the direction of the target person, represents the conjugate transpose, represents the covariance matrix of the minimum variance distortionless response beamformer.

3. The vital sign detection method based on millimeter wave radar and camera fusion according to claim 1 is characterized in that: The method of performing phase extraction on the millimeter wave radar signal based on the human body position to obtain the vital sign signal of the target person includes: Based on the human body position, performing phase extraction on the millimeter wave radar signal to obtain an initial vital sign signal of the target person; The initial vital sign signal is subjected to phase difference processing to obtain a vital sign signal.

4. The vital sign detection method based on millimeter wave radar and camera fusion according to claim 3 is characterized in that: The performing phase difference processing on the initial vital sign signal to obtain a vital sign signal includes: Performing phase difference processing on the initial vital sign signal to obtain a target vital sign signal; filtering the target vital sign signal to remove environmental noise and obtain a vital sign signal; The phase difference processing of the initial vital sign signal is implemented by the following formula: ； in, represents the target vital sign signal, represents the initial vital sign signal, Indicates that the initial vital sign signal is at time point The value of .

5. The vital sign detection method based on millimeter wave radar and camera fusion according to any one of claims 1 to 4, characterized in that: The performing variational modal decomposition on the vital sign signal to obtain the vital sign parameters of the target person includes: Performing variational mode decomposition on the vital sign signal to obtain vital sign signal components of the target person; Performing Hilbert transform on the vital sign signal component to obtain the instantaneous frequency and marginal spectrum of the vital sign signal component; Peak value extraction is performed based on the instantaneous frequency and the marginal spectrum to obtain vital sign parameters of the target person, where the vital sign parameters include heart rate and / or respiratory rate.

6. A vital sign detection device based on the fusion of millimeter wave radar and camera, characterized in that: include: An acquisition unit, used for acquiring a human body image and a millimeter-wave radar signal of a target person; An analysis unit, configured to perform posture detection based on the human body image to obtain a chest area of ​​the target person, and perform frequency analysis on the millimeter wave radar signal to obtain a distance angle heat map; A positioning unit, used for positioning a human body based on the chest area and the distance angle heat map to obtain a human body position; A detection unit, configured to perform phase extraction on the millimeter-wave radar signal based on the human body position to obtain a vital sign signal of the target person; and perform variational mode decomposition on the vital sign signal to obtain a vital sign parameter of the target person; The analysis unit is specifically used for: Performing posture detection based on the human body image to obtain key points of the human body of the target person; Determine a middle area based on the shoulder key points and the hip key points among the human body key points; The chest area of ​​the target person is determined based on the center point of the middle area, the middle point of the line connecting the left shoulder key point and the right shoulder key point among the shoulder key points, and the middle point of the line connecting the right shoulder key point and the right hip key point among the hip key points.

7. A vital sign detection system based on millimeter wave radar and camera fusion, characterized in that: Includes millimeter-wave radar sensors, cameras, and processors; The processor is used to obtain a human body image of the target person collected by the camera and a millimeter-wave radar signal of the target person collected by the millimeter-wave radar sensor, and perform posture detection based on the human body image to obtain a chest area of ​​the target person, and perform frequency analysis on the millimeter-wave radar signal to obtain a distance angle heat map; perform human body positioning based on the chest area and the distance angle heat map to obtain a human body position, and perform phase extraction on the millimeter-wave radar signal based on the human body position to obtain a vital sign signal of the target person; perform variational mode decomposition on the vital sign signal to obtain a vital sign parameter of the target person; The performing posture detection based on the human body image to obtain the chest area of ​​the target person includes: Performing posture detection based on the human body image to obtain key points of the human body of the target person; Determine a middle area based on the shoulder key points and the hip key points among the human body key points; The chest area of ​​the target person is determined based on the center point of the middle area, the middle point of the line connecting the left shoulder key point and the right shoulder key point among the shoulder key points, and the middle point of the line connecting the right shoulder key point and the right hip key point among the hip key points.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the vital sign detection method based on millimeter wave radar and camera fusion as described in any one of claims 1 to 5 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for detecting vital signs based on the fusion of millimeter-wave radar and camera as described in any one of claims 1 to 5 is implemented.

Images