A heart rate detection method based on depth camera and millimeter wave radar
Through the combination of depth camera and millimeter wave radar, deep image segmentation and signal processing technology are used to solve the problem of low accuracy of heart rate measurement of single millimeter wave radar, improving the accuracy of heart rate detection and anti-interference ability, and achieving more Accurate heartbeat and breathing frequency measurements.
Patent Information
- Application Number
- CN202111183181.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-10-11
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2041-10-11
AI Technical Summary
In the prior art, the accuracy of the heart rate measured by a single millimeter-wave radar is low, and the anti-interference ability is weak, and the accuracy is low when the detection range is large.
Combining the depth camera and millimeter wave radar, the chest area is determined through depth image segmentation processing, the average value of pixel values in the chest area is calculated to determine the physiological signal distance range, the chest vibration signal, the heart rate and breathing signal are obtained, and the accuracy of heart rate detection is improved. .
By combining image processing of depth cameras and signal processing of millimeter wave radars, the detection range of the radar is narrowed, the accuracy of heart rate detection and anti-interference ability are improved, and the resulting heartbeat frequency and breathing frequency are more accurate.
Smart Images

Figure CN114041767B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of heart rate detection, and more specifically, to a heart rate detection method based on a depth camera and a millimeter wave radar. Background Art
[0002] With the popularization and widespread use of automobiles, people are paying more and more attention to the safety and intelligence of driving. At present, there are many safety accidents caused by drivers driving in a fatigued state. The driver's driving state can be judged according to the changes in his heart rate. Therefore, by monitoring the heart rate and other characteristics of the life in the car, safety accidents caused by factors such as fatigue driving can be reduced; heart rate detection can also be used to determine whether the driver's seat is alive or fake, which plays an anti-counterfeiting role.
[0003] In the existing technology, heart rate detection is mostly based on a single method such as millimeter wave radar. Since the measurement of heart rate is based on the distance difference caused by the tiny movements of heart contraction and relaxation, and then the phase change converted, when the human body swings greatly, its accuracy will be affected. In addition, the power of the radar is fixed. The larger the search range, the lower the detection accuracy and the weaker the anti-interference ability, resulting in a low accuracy rate of heart rate values measured by a single method such as millimeter wave radar.
[0004] Therefore, how to solve the problem of low accuracy of heart rate values measured by a single method such as millimeter wave radar in the prior art has become an important technical problem to be solved by those skilled in the art. Summary of the invention
[0005] The purpose of the present invention is to provide a heart rate detection method based on a depth camera and a millimeter wave radar, so as to solve the problem of low accuracy of the heart rate value measured by a single method of a millimeter wave radar in the prior art. The preferred technical solutions among the many technical solutions provided by the present invention can produce many technical effects as described below.
[0006] To achieve the above object, the present invention provides the following technical solutions:
[0007] The present invention provides a heart rate detection method based on a depth camera and a millimeter wave radar, comprising the following steps:
[0008] Obtaining an original depth image of a human body by a depth camera, and performing segmentation processing based on the original depth image to obtain a chest region of the human body;
[0009] Calculating an average value of pixel values in the chest region on the original depth image to obtain a chest region distance value;
[0010] The radar determines the physiological signal distance range according to the chest area distance value, and obtains the chest vibration signal within the physiological signal distance range, separates the heart rate signal and the breathing signal from the chest vibration signal, obtains the heart rate and the breathing rate, and outputs the relationship between the heart rate frequency and time, and the relationship between the breathing frequency and time, respectively.
[0011] Preferably, the segmentation process includes:
[0012] Performing mask segmentation and connected domain analysis on the original depth image in sequence to determine the human body area;
[0013] Eliminate the arm portion of the human body region and determine the upper body region in the retained portion;
[0014] The thoracic region is determined in the upper body region.
[0015] Preferably, the connected domain analysis includes: based on the image segmented by the mask, finding and marking each connected domain in the image, wherein the area with the largest area among the connected domains is the human body region.
[0016] Preferably, the head width is obtained from the human body area, and two vertical lines whose horizontal distance from the center of the head width is equal to 1-1.2 times of the head width are cutting lines, and the part of the human body area located on the side of the cutting line away from the center of the head width is the arm part.
[0017] Preferably, the human body area is framed by a rectangular frame, the maximum width and the maximum height of the human body area are used as the length and width of the rectangular frame respectively, and within the height range of 9-11 pixels from the upper edge of the rectangular frame, the largest connected domain is selected as the head width representative domain, and the maximum length of the head width representative domain is the head width.
[0018] Preferably, the width of the chest region is equal to 1.8-2.2 times the width of the head, and the height of the chest region is equal to 0.3-0.6 times the height of the upper body region.
[0019] Preferably, the average value of each row of pixels in the retained part is traversed row by row from top to bottom, and then the difference between the average values of the current row and the previous row is calculated, and the straight line corresponding to the maximum value of the difference is the dividing line, and the upper body area is located below the dividing line.
[0020] Preferably, data enhancement processing is also included between the mask segmentation and the connected domain analysis to obtain an enhanced depth image, and the connected domain analysis is based on the enhanced depth image; the data enhancement processing includes median filtering, closing operation and binarization of the image segmented by the mask.
[0021] Preferably, before the radar determines the distance range of the physiological signal, the method further includes: the radar acquires ADC data, and performs Fourier transform on the ADC data to obtain a distance curve.
[0022] Preferably, after acquiring the chest vibration signal within the distance range of the physiological signal, phase unwrapping and phase difference calculation are performed to enhance the physiological signal and eliminate phase drift, and bandpass filter calculation is performed to separate the heart rate signal and the breathing signal.
[0023] In the technical solution provided by the present invention, the heart rate detection method based on the depth camera and the millimeter wave radar finds the chest area of the human body in the original depth image of the human body, calculates the average value of the pixel values in the chest area on the original depth image as the distance value of the chest area, and then determines the physiological signal distance range through the radar, obtains the chest vibration signal within the physiological signal distance range, obtains the heart rate and breathing rate, and outputs the relationship between the heart rate and time, and the breathing rate and time. In this way, the chest area is obtained by segmentation processing based on the original depth image, which is convenient for more accurate determination of the position of the chest area. The radar obtains the heart rate and breathing rate based on the chest vibration signal within the physiological signal distance range. The detection range of the radar is reduced by combining the original depth image segmentation processing with the millimeter wave radar detection technology, and its detection accuracy is improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0025] Figure 1 is a schematic diagram of an original depth image in an embodiment of the present invention;
[0026] Figure 2 is a schematic diagram of an image obtained after mask segmentation in an embodiment of the present invention;
[0027] Figure 3 is a schematic diagram of enhancing a depth image in an embodiment of the present invention;
[0028] Figure 4 is a schematic diagram of a human body region and a head width representative domain framed by a rectangular frame in an embodiment of the present invention;
[0029] Figure 5 is a schematic diagram of a human body region after removing the arm portion in an embodiment of the present invention;
[0030] Figure 6 is a schematic diagram of the position of the dividing line in an embodiment of the present invention;
[0031] Figure 7 is a schematic diagram of an upper body area and a head area framed by a rectangular frame in an embodiment of the present invention;
[0032] Figure 8 is a schematic diagram of the chest region of a human body in an embodiment of the present invention;
[0033] Fig. 9 It is a schematic diagram of the average output result of the pixel values in the chest area on the original depth image in an embodiment of the present invention. DETAILED DESCRIPTION
[0034] To make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be described in detail below. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other implementation methods obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present invention.
[0035] The purpose of this specific implementation is to provide a heart rate detection method based on a depth camera and a millimeter-wave radar, so as to solve the problem of low accuracy of the heart rate value measured by a single method of millimeter-wave radar in the prior art.
[0036] The following is an explanation of the embodiments with reference to the accompanying drawings. In addition, the embodiments shown below do not limit the invention described in the claims. In addition, the entire contents of the configurations shown in the following embodiments are not limited to the solutions required as the invention described in the claims.
[0037] See also Figure 1-Figure 9 The present invention provides a heart rate detection method based on a depth camera and a millimeter wave radar, comprising the following steps:
[0038] The original depth image of the human body is obtained by a depth camera, and the original depth image is segmented to obtain the chest area of the human body.
[0039] In a specific embodiment, the depth camera adopts a TOF (Time offlight) camera, and the original depth image of the human body can be obtained based on the SDK (Software Development Kit) of the TOF camera, such as Figure 1 As shown in the figure, since the driver is in a sitting position while driving, the original depth image does not contain the lower limbs of the human body.
[0040] In a specific embodiment, the segmentation process includes: first, performing mask segmentation, data enhancement processing, and connected domain analysis on the original depth image in sequence to obtain a human body region; then, removing the arm portion of the human body region, and determining the upper body region from the retained portion of the human body region after removing the arm portion; and then determining the chest region in the upper body region.
[0041] Specifically, the principle of mask segmentation is to set the pixel values of a specific area of the original depth image to 0 to obtain a segmented depth image. The purpose is to detect only the central area of the original depth image (that is, the human activity area) to reduce the interference of the external environment on the image. Figure 2 As shown, the segmented depth image is obtained after the original depth image is segmented by mask.
[0042] Data enhancement is performed on the mask segmented image to eliminate the influence of the camera's own noise on the image and obtain an enhanced depth image, such as Figure 3 As shown in Figure 1, the data enhancement process includes median filtering, closing operation and binarization of the segmented depth image. Median filtering, closing operation and binarization are some common methods of image processing.
[0043] The object of connected domain analysis is the enhanced depth image. A connected domain refers to an image region composed of pixels with the same pixel value and adjacent positions in the image. Connected domain analysis is based on the image obtained after mask segmentation and data enhancement processing. Each connected domain in the enhanced depth image is found and marked. The connected domain with the largest area in each connected domain is determined to be the human body region. In this scheme, the Two-Pass algorithm is used to search and mark the connected domain. The Two-Pass algorithm can find and mark all connected domains in the enhanced depth image by scanning the enhanced depth image twice. The principle is: a label is assigned to each pixel position during the first scan. During the scanning process, one or more different labels may be assigned to the pixel set in the same connected domain. Therefore, it is necessary to merge these labels belonging to the same connected domain but with different values, that is, to record the connectivity relationship between them; the second scan is to classify the pixels with connectivity into a connected domain and assign the same label.
[0044] When removing the arm part, the arm part needs to be determined first. The basis for determining the arm part is the head width. The head width is determined by: using a rectangular frame to frame the human body area in the enhanced depth image. The length of the rectangular frame is equal to the maximum width of the human body area, and the width of the rectangular frame is equal to the maximum width of the human body area. In the height range of 9-11 pixels from the upper edge of the rectangular frame, the largest connected domain is determined as the head width representative domain by the connected domain analysis method. The same method as the human body region is used to frame the head width representative domain with a rectangular frame, such as Figure 4The maximum length of the head width representative domain, that is, the length of the rectangular frame that selects the head width representative domain, is the head width. In a preferred embodiment, the head width representative domain is selected within a height range of 10 pixels from the upper edge of the rectangular frame.
[0045] The basis for determining the arm part is as follows: According to experience, the width of a person's shoulders is approximately 2-2.4 times the width of the head, so the head width is used as a standard and expanded to the left and right by 0.5-0.7 times each, which is the width of the shoulders, and the part outside the shoulder width is the arm part. The method for determining the arm part is as follows: a cutting line is determined on each side of the head width, and the two vertical lines whose horizontal distance from the center of the head width is equal to 1-1.2 times the head width are the cutting lines. The part of the human body area that is located on the side of the cutting line away from the center of the head width is the arm part. The method for determining the arm part can also be understood as follows: using the head width as a standard, expand it to the left and right by 0.5-0.7 times each, which is the width of the shoulders, and the part to the left and right of the shoulder width is the arm part. After determining the arm part, remove the arm part from the human body area. Figure 5 That is, it is a schematic diagram of the human body region after removing the arm part. In a specific embodiment, twice the width of the head can be used as the width of the shoulder, but due to individual differences, the multiple relationship between the width of the head and the width of the shoulder is slightly different.
[0046] The dividing line between the upper body area and the head area is determined in the retained part after removing the arm part of the human body area. The upper body area is the part below the dividing line, and the upper body area is framed by a rectangular frame. The dividing line is determined by traversing the average value of each row of pixels in the retained part from top to bottom, and then calculating the difference between the average value of the current row and the previous row. The straight line corresponding to the maximum value of the difference is the dividing line. Figure 7 As shown in the figure, the width of the rectangular box used to select the upper body area is the height of the upper body area. The principle of determining the dividing line is that for the upper body of the human body, the width of the head and neck is obviously smaller than that of the shoulders, so there is an obvious change in pixel size at the connection between the neck and shoulders.
[0047] The part above the dividing line is the area where the head of the human body is located. Because external factors such as clothes and scarves may cause the height of the area where the head is located to be greater than the actual head height, the height of the area where the head is located is reduced to 0.6-0.8 times, and then the head area is framed with a rectangular frame. In this way, the framed head area reduces the influence of the external environment. In a specific embodiment, the height of the area where the head is located can be reduced to 0.7 times, and then the head area is framed.
[0048] The chest area is determined based on the head width and the upper body height. The chest area is 1.8-2.2 times the head width and 0.3-0.6 times the upper body height. The chest area is framed with a rectangular frame, as shown in Figure 8 As shown. Determining the width and height of the chest region also determines the position of the chest region, because the rectangular frame that frames the chest region is actually made based on the coordinates. In a specific embodiment, twice the width of the head can be used as the width of the chest region, and 0.4 times the height of the upper body region can be used as the height of the chest region.
[0049] After obtaining the chest area of the human body, the average value of the pixel values in the chest area on the original depth image is calculated to obtain the chest area distance value, such as Fig. 9 As shown in the figure, since each pixel value in the original depth image of the depth camera is the vertical distance value between the current point and the camera lens, the chest area distance value represents the distance between the depth camera and the chest area.
[0050] When the radar detects the distance between itself and the depth camera, it can obtain the distance between the chest area and the radar, and then determine the physiological signal distance range to be detected. Before the radar determines the physiological signal distance range, it is also necessary to obtain ADC (analog to digital converter) data, perform Fourier transform on the ADC data to obtain the distance curve, and the radar obtains the chest vibration signal within the physiological signal distance range based on the distance curve. After obtaining the chest vibration signal, the DSP (Digital Signal Process) is used to process the signal range corresponding to breathing and heartbeat, and the physiological signal is enhanced and the phase drift is eliminated through phase unwrapping and phase difference calculation. The heart rate signal and the breathing signal are separated by a bandpass filter to obtain the heart rate and breathing frequency, and then the relationship between the heart rate and time, and the breathing frequency and time are output respectively. According to the change relationship between the heart rate and time, it can be judged whether the driver is in a fatigue driving state. If so, a reminder signal is issued to reduce the occurrence of traffic accidents.
[0051] In a specific embodiment, according to the difference between the heart rate and the respiratory rate, the phase value is filtered using a bandpass filter to distinguish the heart rate signal from the respiratory signal, and FFT (Fast Fourier Transform) is performed on the phase signal. According to the peak value and its harmonic characteristics, the corresponding respiratory frequency and heart rate within N Frame times are obtained, and then the frequency at this time is judged according to different confidence indicators, and the relationship between the respiratory frequency and heart rate over time is output.
[0052] The technical solution provided by this scheme is based on the combination of segmentation processing of the original depth image and millimeter wave radar technology, which makes the radar more accurate in positioning the chest area, reduces the radar's detection area, and is conducive to improving its detection accuracy, and the obtained heart rate and respiratory rate values are more accurate.
[0053] The following content gives the specific steps of heart rate detection according to a heart rate detection method based on a depth camera and millimeter wave radar in this solution:
[0054] S01: The SDK of the TOF camera obtains the original depth image of the human body, and performs mask segmentation on the original depth image to obtain a segmented depth image;
[0055] S02: Performing graphic enhancement processing on the segmented depth image to obtain an enhanced depth image;
[0056] S03: Based on the enhanced depth image, a connected domain analysis is performed to obtain the human body region and the head width representative domain and frame them respectively, and the maximum length of the head width representative domain is the head width;
[0057] S04: determining a cutting line according to the width of the head, and removing the arm portion located on the side of the cutting line away from the center of the head width;
[0058] S05: determining a segmentation line according to the obvious pixel changes at the connection between the neck and the shoulders, and framing the head area and the upper body area above and below the segmentation line respectively;
[0059] S06: Identify the thoracic region within the upper body region;
[0060] S07: Calculate the average value of the pixel values in the chest area on the original depth image to obtain the chest area distance value;
[0061] S08: The millimeter wave radar works and obtains ADC data, and performs Fourier transform on the ADC data to obtain a distance curve;
[0062] S09: The millimeter wave radar determines the physiological signal distance range based on the chest area distance value;
[0063] S10: The millimeter wave radar obtains the chest vibration signal within the physiological signal distance range according to the distance curve;
[0064] S11: Processing the signal range corresponding to breathing and heartbeat through DSP, performing phase unwrapping, phase difference calculation, and bandpass filter calculation to distinguish the heart rate signal and the breathing signal and obtain the heart rate and breathing frequency;
[0065] S12: Output the relationship between the heart rate and the breathing rate changing with time.
[0066] It is understood that the same or similar parts in the above embodiments can refer to each other, and the content not described in detail in some embodiments can refer to the same or similar content in other embodiments. The multiple solutions provided by the present invention include the basic solution itself, which are independent of each other and do not restrict each other, but they can also be combined with each other without conflict to achieve multiple effects together.
[0067] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.
Claims
1. A heart rate detection method based on a depth camera and a millimeter wave radar, characterized in that: The following steps are involved: Obtaining an original depth image of a human body by a depth camera, and performing segmentation processing based on the original depth image to obtain a chest region of the human body; calculating an average value of pixel values in the chest region on the original depth image to obtain a chest region distance value; The radar determines the physiological signal distance range according to the chest area distance value, and obtains the chest vibration signal within the physiological signal distance range, separates the chest vibration signal to obtain the heart rate signal and the breathing signal, obtains the heart rate and the breathing rate, and outputs the relationship between the heart rate and time, and the relationship between the breathing rate and time respectively; The segmentation process includes: performing mask segmentation and connected domain analysis on the original depth image in sequence to determine a human body region; removing the arm portion of the human body region and determining an upper body region from the retained portion; and determining the chest region from the upper body region; The connected domain analysis includes: based on the image segmented by the mask, finding and marking each connected domain in the image, wherein the largest area among the connected domains is the human body region; The head width is obtained from the human body region, and two vertical lines whose horizontal distance from the center of the head width is equal to 1-1.2 times of the head width are cutting lines, and the part of the human body region located on the side of the cutting line away from the center of the head width is the arm part; The human body region is framed with a rectangular frame, the maximum width and the maximum height of the human body region are respectively used as the length and the width of the rectangular frame, and within a height range of 9-11 pixels from the upper edge of the rectangular frame, the largest connected domain is selected as the head width representative domain, and the maximum length of the head width representative domain is the head width; The width of the chest region is equal to 1.8-2.2 times the width of the head, and the height of the chest region is equal to 0.3-0.6 times the height of the upper body region; Traversing the average value of each row of pixels in the retained part row by row from top to bottom, and then calculating the difference between the average values of the current row and the previous row, the straight line corresponding to the maximum value of the difference is the dividing line, and the upper body area is located below the dividing line; A data enhancement process is also included between the mask segmentation and the connected domain analysis to obtain an enhanced depth image, and the connected domain analysis is based on the enhanced depth image; the data enhancement process includes median filtering, closing operation and binarization of the image segmented by the mask; Before the radar determines the distance range of the physiological signal, the method further includes: the radar acquires ADC data, and performs Fourier transform on the ADC data to obtain a distance curve; After acquiring the chest vibration signal within the distance range of the physiological signal, the physiological signal is enhanced and the phase drift is eliminated by phase unwrapping and phase difference calculation, and the heart rate signal and the breathing signal are separated by bandpass filter calculation.
