Non-contact heart rate and respiration rate detection method and device

Abstract

The invention discloses a non-contact heart rate and respiration rate detection method, which is characterized in that video information acquisition is carried out on a preset area of a detector, and the forehead of the detector in a video is identified based on the video information, so that the cost of equipment is greatly reduced, a common detector does not repel a camera, and the adaptability to the detector is improved; the first forehead information related to the forehead of the detector is extracted according to the identified forehead, and then the heart rate value and the breathing rate value of the detector are obtained through the first forehead information of the detector, so that whether the detector wears the mask does not affect the detection result of the heart rate value and the breathing rate value, the requirement on the detector is reduced, and the detection efficiency is improved. And the adaptability to a detector is further improved.

Description

Technical Field

[0001] The present application belongs to the field of information processing technology, and in particular, relates to a non-contact heart rate and respiratory rate detection method and device. Background Art

[0002] Traditional vital sign monitoring is generally achieved through non-contact pulse wave detection technology using laser and electromagnetic detection. Laser detection technology uses laser irradiation and optical sensors to collect reflected signals to achieve high-precision, non-invasive and real-time pulse wave and heart rate monitoring, but the equipment cost is high, the adaptability to some people is poor, and the detection distance and accuracy are limited. At the same time, the non-contact pulse wave technology based on electromagnetic detection uses low-frequency electromagnetic wave radiation and inductive sensors for monitoring, but it also has the problems of high equipment cost and poor adaptability to people who are sensitive to electromagnetic waves. Summary of the invention

[0003] The purpose of this application is to provide a non-contact heart rate and respiratory rate detection method, aiming to solve the problems of high cost and poor patient adaptability in traditional vital signs monitoring.

[0004] A first aspect of an embodiment of the present application provides a non-contact heart rate and respiratory rate detection method, comprising:

[0005] Collecting video information of a preset area of ​​a subject, wherein the preset area includes the face of the subject;

[0006] Identify the forehead of the subject based on the video information and extract first forehead information;

[0007] Preprocessing the first forehead information to obtain second forehead information, and extracting preset channel information from the second forehead information to obtain third forehead information; and

[0008] The third forehead information is subjected to noise reduction processing to obtain fourth forehead information, and the fourth forehead information is transformed to obtain a frequency domain signal, and a heart rate value and a respiratory rate value are obtained according to the frequency signal.

[0009] In one optional implementation manner, the step of identifying the forehead of the subject based on the video information and extracting first forehead information includes:

[0010] Extracting facial information of the subject from the video information through three cascaded networks;

[0011] The forehead of the subject is recognized based on the facial information of the subject and the first forehead information is extracted.

[0012] In one optional implementation, the three cascade networks include a candidate region generation network, a refinement network, and an output network;

[0013] The extracting of the facial information of the subject from the video information through three cascaded networks includes:

[0014] The candidate region generation network performs preliminary feature extraction based on the video information to generate a first facial candidate window;

[0015] The refinement network filters and selects the first candidate facial window to obtain a second candidate facial window;

[0016] The output network performs deep feature extraction based on the second candidate window to obtain the target facial bounding box and facial key point information of the subject;

[0017] Based on the target facial bounding box and the facial key point information, an image of a facial region containing a subject being detected is cropped from the video information and used as the facial information of the subject being detected.

[0018] In one optional implementation manner, the step of identifying the forehead of the subject based on the facial information of the subject and extracting the first forehead information includes:

[0019] Identifying the forehead of the subject by using edge features, line features, and four-corner-rectangle features of the face region image of the subject;

[0020] First forehead information of a forehead area of ​​the subject is extracted based on the forehead of the subject.

[0021] In one optional implementation manner, the preprocessing of the first forehead information to obtain the second forehead information includes:

[0022] Collect images upward on the first forehead information through a Gaussian pyramid to obtain an image sequence, call back the Gaussian pyramid, and subtract the first forehead information from the Gaussian pyramid to obtain a Laplacian pyramid;

[0023] The Laplacian pyramid is low-pass filtered and down-sampled, and decomposed to obtain a low-pass approximate image, the low-pass approximate image is interpolated and filtered, and the interpolation of the low-pass approximate image and the image corresponding to the first forehead information is calculated to obtain a decomposed band-pass component, and then the low-pass approximate image is decomposed again at the next level to iteratively obtain a multi-scale decomposed forehead image, and then the Gaussian pyramid is used to amplify the color of the forehead area in the multi-scale decomposed forehead image to obtain the second forehead information.

[0024] In one optional implementation manner, the extracting the preset channel information in the second forehead information to obtain the third forehead information includes:

[0025] Separating the three channels of R, G and B in the second forehead information;

[0026] The G channel in the second forehead information is extracted as the third forehead information.

[0027] In one optional implementation, the third forehead information is subjected to noise reduction processing to obtain fourth forehead information, and the fourth forehead information is transformed to obtain a frequency domain signal, and a heart rate value and a respiratory rate value are obtained according to the frequency domain signal, including:

[0028] Performing noise reduction on the third frontal information by using a fourth-order Butterworth filter to obtain the fourth frontal information;

[0029] Performing fast Fourier transform on the fourth forehead information to obtain the frequency domain signal;

[0030] The frequency domain signal is filtered for non-heart rate frequency intervals, and frequency domain values ​​are obtained through a two-layer Laplace pyramid detection result, wherein the heart rate frequency interval is 1-2 Hz, and the respiratory rate frequency interval is 0.2-0.4 Hz, and a heart rate frequency domain graph and a respiratory rate frequency domain graph are obtained through band-pass filtering;

[0031] Calculate the heart rate and respiratory rate values, specifically:

[0032] V hr =60*f hr ,

[0033] Vhr——heart rate value; fhr——frequency value corresponding to the peak value in the frequency domain graph;

[0034] V rr =60*f rr ,

[0035] V rr ——Respiration rate value; f rr ——The frequency value corresponding to the peak in the frequency domain diagram.

[0036] The second aspect of the present application also provides a non-contact heart rate and respiratory rate detection device, comprising:

[0037] A shooting module, used for collecting video information of a preset area of ​​a subject, wherein the preset area includes the face of the subject; and

[0038] A processing module is used to identify the forehead of the tester based on the video information and extract the first forehead information; pre-process the first forehead information to obtain the second forehead information, and extract the preset channel information in the second forehead information to obtain the third forehead information; and perform noise reduction processing on the third forehead information to obtain the fourth forehead information, and transform the fourth forehead information to obtain a frequency domain signal, and obtain the heart rate value and the respiratory rate value according to the frequency signal.

[0039] The third aspect of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method as described in any one of the above items is implemented.

[0040] The fourth aspect of the present application also provides a server, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the methods listed above when executing the computer program.

[0041] Compared with the prior art, the embodiments of the present invention have the following beneficial effects: by collecting video information of a preset area of ​​the tester and identifying the forehead of the tester in the video based on the video information, the cost of the equipment is greatly reduced, and generally the tester will not be averse to the camera, thereby improving the adaptability to the tester; and based on the identified forehead, the first forehead information related to the forehead of the tester is extracted, and then the heart rate and respiratory rate of the tester are obtained through the first forehead information of the tester, so that whether the tester wears a mask will not affect the detection results of the heart rate and respiratory rate, thereby reducing the requirements for the tester and further improving the adaptability to the tester. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A schematic flowchart of a non-contact heart rate and respiratory rate detection method provided in an embodiment of the present application;

[0043] Figure 2 Schematic diagram of heart rate (left) and respiratory rate (right) signals of a first-order Butterworth filter according to an embodiment of the present application;

[0044] Figure 3 Schematic diagram of heart rate (left) and respiratory rate (right) signals of a second-order Butterworth filter according to an embodiment of the present application;

[0045] Figure 4 Schematic diagram of heart rate (left) and respiratory rate (right) signals of a third-order Butterworth filter according to an embodiment of the present application;

[0046] Figure 5 Schematic diagram of heart rate (left) and respiratory rate (right) signals of a fourth-order Butterworth filter according to an embodiment of the present application;

[0047] Figure 6 Schematic diagram of the frequency domain diagram of the heart rate (left) and respiratory rate (right) of the first layer of the Laplace pyramid of the embodiment of the present application;

[0048] Figure 7 Schematic diagram of the frequency domain diagram of the heart rate (left) and respiratory rate (right) of the second layer of the Laplace pyramid of the embodiment of the present application;

[0049] Figure 8 Schematic diagram of the frequency domain diagram of the heart rate (left) and the respiratory rate (right) of the third layer of the Laplace pyramid of the embodiment of the present application;

[0050] Fig. 9 Schematic diagrams of three test results of rate and respiratory rate in different time periods of an embodiment of the present application;

[0051] Fig.10 A schematic diagram of the structure of a non-contact heart rate and respiratory rate detection device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0053] Figure 1 The first exemplary principle block diagram provided by the embodiment of the present application is shown. For the convenience of explanation, only the part related to the present embodiment is shown, which is described in detail as follows:

[0054] A non-contact heart rate and respiratory rate detection method includes S100-S400.

[0055] S100: Collecting video information of a preset area of ​​the subject, wherein the preset area includes the face of the subject. Video information is obtained by shooting a video of the preset area of ​​the subject through a camera, wherein the preset area refers to an area including the face of the subject.

[0056] Video shooting does not require any external light source or filter. The video of the subject’s face can be shot under conventional indoor fluorescent lighting and natural light conditions. The video information of the subject is obtained through the camera, which greatly reduces the cost of the equipment. In addition, the subject generally does not reject the camera, so it has high adaptability.

[0057] In a specific implementation, a binocular camera may be used to capture video of the subject, and specific parameters of the binocular camera may be a resolution of 8 million pixels, a field of view of 83 / 73 / 50 degrees (diagonal / horizontal / vertical), and a distortion of <1%. The binocular camera may be specifically selected as the model IMX219-83 Stereo CameraIMX219-83 Stereo Camera.

[0058] S200: Identify the forehead of the subject based on the video information and extract first forehead information.

[0059] Based on the video information, the forehead of the testee in the video is identified, and the first forehead information related to the forehead of the testee is extracted based on the identified forehead. Subsequently, the heart rate and respiratory rate values ​​of the testee are obtained through the first forehead information of the testee. Therefore, whether the testee wears a mask will not affect the detection results of the heart rate and respiratory rate values, which reduces the requirements for the testee and further improves the adaptability to the testee.

[0060] In one of the actual application scenarios, such as during ward inspections, patients (testers) sometimes wear masks, which will cover a large area of ​​the face. By identifying the forehead of the tester, the practicality of this method can be further improved.

[0061] S300: Preprocessing the first forehead information to obtain second forehead information, and extracting preset channel information from the second forehead information to obtain third forehead information.

[0062] The second forehead information obtained by preprocessing the first forehead information has smaller noise than the first forehead information; then the preset information in the second forehead information is extracted to obtain the third forehead information, which further reduces the noise.

[0063] S400: Perform noise reduction processing on the third forehead information to obtain fourth forehead information, transform the fourth forehead information to obtain a frequency domain signal, and obtain a heart rate value and a respiratory rate value according to the frequency signal.

[0064] The third frontal information is further subjected to noise reduction processing to obtain fourth frontal information, and the fourth frontal information is transformed to obtain a frequency domain signal carrying information related to the heart rate value and the respiratory rate value, thereby obtaining the heart rate value and the respiratory rate value through the frequency domain signal.

[0065] In this embodiment, video information is collected from a preset area of ​​the tester, and the forehead of the tester in the video is identified based on the video information, which greatly reduces the cost of the equipment. In addition, the tester generally does not reject the camera, which improves the adaptability to the tester. The first forehead information related to the tester's forehead is extracted based on the identified forehead, and the heart rate and respiratory rate of the tester are subsequently obtained through the first forehead information of the tester, so that whether the tester wears a mask will not affect the detection results of the heart rate and respiratory rate, thereby reducing the requirements for the tester and further improving the adaptability to the tester.

[0066] In one optional implementation manner, the step of identifying the forehead of the subject based on the video information and extracting first forehead information includes:

[0067] The facial information of the detector is extracted from the video information through three cascaded networks.

[0068] In a specific implementation, the three cascaded networks are multi-task cascaded convolutional networks (MTCNN) algorithms developed in python language, which belong to multi-convolutional neural networks. It would be slow to run the MTCNN program with only the central processing unit (CPU), so TensorRT is selected as a high-performance deep learning reasoning optimizer, combining TensorRT with the graphics processing unit (GPU) of NVIDIA's Jetson Nano embedded board to provide low-latency, high-throughput deployment reasoning for the overall processing framework. Among them, TensorRT is a high-performance deep learning reasoning engine provided by NVIDIA; the Jetson Nano embedded board is an AI embedded development board provided by NVIDIA.

[0069] The forehead of the subject is recognized based on the facial information of the subject and the first forehead information is extracted.

[0070] In a specific implementation, a Haar cascade classifier is used to complete forehead detection of the subject based on the facial information of the subject, the forehead of the subject is selected, and the forehead of the subject is extracted separately to obtain first forehead information, thereby filtering more environmental noise.

[0071] In one optional implementation, the three cascade networks include a candidate region generation network P-Net, a refinement network R-Net and an output network O-Net.

[0072] The extracting of the facial information of the subject from the video information through three cascaded networks includes:

[0073] The candidate region generation network performs preliminary feature extraction based on the video information to generate a first facial candidate window.

[0074] The refinement network filters and selects the first candidate facial window to obtain a second candidate facial window.

[0075] The output network performs deep feature extraction based on the second candidate window to obtain the target facial bounding box and facial key point information of the subject.

[0076] Based on the target facial bounding box and the facial key point information, an image of a facial region containing a subject being detected is cropped from the video information and used as the facial information of the subject being detected.

[0077] The three-cascade network design allows the network structure to be adjusted dynamically according to the training data or learning process, such as automatically adding new layers or neurons to adapt to more complex pattern recognition tasks; and the learning speed can be accelerated through incremental learning or local training, and only the newly added parts are trained and optimized. And in some cases, the cascade network can adopt a non-traditional weight update mechanism, and there is no need to perform a global error back propagation algorithm on the entire network. The use of this three-level hierarchical structure helps to improve the robustness of the system, and errors in a single level may be corrected in subsequent levels.

[0078] In one optional implementation manner, the step of identifying the forehead of the subject based on the facial information of the subject and extracting the first forehead information includes:

[0079] The forehead of the subject being detected is identified by edge features, line features, and four-corner-rectangle features of the face region image of the subject being detected.

[0080] First forehead information of a forehead area of ​​the subject is extracted based on the forehead of the subject.

[0081] In this embodiment, a Haar cascade classifier is used to complete forehead detection. The Haar cascade detects edge features, line features, and four-corner-rectangle features, which can better select the forehead of the person being detected, and extract the forehead of the person being detected separately to obtain the first forehead information. The first forehead information obtained in this way has filtered out most of the environmental noise.

[0082] In one optional implementation manner, the preprocessing of the first forehead information to obtain the second forehead information includes:

[0083] The first forehead information is upwardly collected through the Gaussian pyramid to obtain an image sequence, and the Gaussian pyramid is called back to subtract the first forehead information from the Gaussian pyramid to obtain a Laplacian pyramid. The first forehead information is image information. The Gaussian pyramid is a multi-scale representation of the signal, that is, the same signal or image is Gaussian blurred multiple times and downsampled to generate multiple groups of signals or images at different scales for subsequent processing, and the Gaussian kernel is the only linear kernel, and other noise will not be introduced into the image blur.

[0084] Specifically, a Gaussian pyramid is constructed by using a custom first function, the input value of which is the first forehead information and the number of layers of the Gaussian pyramid. The first function returns a series of images with gradually decreasing sizes, which are obtained by continuously applying Gaussian blur and downsampling. An empty list is created to store the layers of the Laplacian pyramid, and then a loop is used to iterate from the second layer of the Gaussian pyramid. In each iteration, the next layer of the Gaussian pyramid is upsampled using the second function, the height and width of the upsampled image are obtained, and the current Gaussian pyramid layer (i.e., the image of smaller size) is adjusted to the size of the upsampled image using the third function. This is to ensure that the two images have the same size for subtraction operations, and the difference (residual) between the two images is calculated using the fourth function, and the result is added to the Laplacian pyramid list, and the last layer of the Gaussian pyramid is added to the Laplacian pyramid list, thereby obtaining the Laplacian pyramid. Among them, the second function, the third function, and the fourth function are functions for image upsampling, resizing, and image subtraction operations, respectively.

[0085] The Laplacian pyramid is low-pass filtered and down-sampled, and decomposed to obtain a low-pass approximate image, the low-pass approximate image is interpolated and filtered, and the interpolation of the low-pass approximate image and the image corresponding to the first forehead information is calculated to obtain a decomposed band-pass component, and then the low-pass approximate image is decomposed again at the next level to iteratively obtain a multi-scale decomposed forehead image, and then the Gaussian pyramid is used to amplify the color of the forehead area in the multi-scale decomposed forehead image to obtain the second forehead information.

[0086] In one optional implementation manner, the extracting the preset channel information in the second forehead information to obtain the third forehead information includes:

[0087] Separate the three channels of R, G, and B in the second frontal information.

[0088] The G channel in the second forehead information is extracted as the third forehead information; the G channel in the second forehead information is the most sensitive to the heart rate and has relatively small noise.

[0089] In one optional implementation, the third forehead information is subjected to noise reduction processing to obtain fourth forehead information, and the fourth forehead information is transformed to obtain a frequency domain signal, and a heart rate value and a respiratory rate value are obtained according to the frequency domain signal, including:

[0090] The fourth frontal information is obtained by performing noise reduction on the third frontal information through a fourth-order Butterworth filter.

[0091] like Figure 2-5 As shown in the figure, after passing through the Butterworth filter, it can be seen that the second-order respiration rate signal tends to be stable, while the heart rate signal tends to be stable only when it reaches the fourth order. Selecting the fourth-order Butterworth filter for signal noise reduction can improve the detection accuracy and reduce the amount of program calculation.

[0092] Performing a fast Fourier transform on the fourth forehead information to obtain the frequency domain signal;

[0093] The frequency domain signal is filtered for non-heart rate frequency intervals, and the frequency domain value is obtained through the two-layer Laplace pyramid detection result, wherein the heart rate frequency interval is 1-2Hz, the respiratory rate frequency interval is 0.2-0.4Hz, and the heart rate frequency domain map and the respiratory rate frequency domain map are obtained through bandpass filtering. Generally speaking, the lower the spatial frequency, the higher the signal-to-noise ratio, the fewer the number of image pixels, and the shorter the detection time. Too low a spatial frequency will cause distortion, and the information contained in different Laplace layers will also be different. Figure 6-Figure 8 The frequency domain diagram showing the detection results of Laplace pyramids with different numbers of layers shows that, by comparing the experimental values ​​and true values ​​of heart rate and respiratory rate, it is better to use the second layer of the Laplace pyramid for processing heart rate and respiratory rate.

[0094] Calculate the heart rate and respiratory rate values, specifically:

[0095] V hr =60*f hr ,

[0096] Vhr——heart rate value; fhr——frequency value corresponding to the peak value in the frequency domain graph;

[0097] V rr =60*f rr ,

[0098] V rr ——Respiration rate value; f rr ——The frequency value corresponding to the peak in the frequency domain diagram.

[0099] The heart rate and respiratory rate detection results obtained by this embodiment have only one point outside the consistency limit, but more than 95% of the data are within the consistency limit. This detection method has good consistency with the traditional contact gold standard detection method.

[0100] Fig. 9 The results of heart rate and respiratory rate detection in different time periods are shown. It can be seen that if the detection time is too short, the sampling points will be reduced. Due to the non-stationarity of heart rate and respiratory rate, accurate results cannot be obtained, and the detection accuracy decreases accordingly. As the video length gradually increases, the number of sampling points gradually increases. It can be seen that the detection result accuracy is highest at 60 seconds, so the duration of the video shooting of the tester is generally at least 60 seconds. At the same time, too short a detection time will lead to inaccurate detection results, especially for respiratory rate detection; too long a detection time will increase the amount of calculation and increase the operation time; the detection result accuracy is highest at 60 seconds, and further increasing the detection time will increase the calculation cost. In order to balance the effect and efficiency of the detection, the shooting time is at least 60 seconds, and the detection time of the subsequent algorithm is preferably 60 seconds.

[0101] like Fig.10 As shown, the present application also provides a non-contact heart rate and respiratory rate detection device, including a shooting module 510 and a processing module 520.

[0102] The shooting module 510 is used to collect video information of a preset area of ​​the tester, where the preset area includes the face of the tester.

[0103] Processing module 520 is used to identify the forehead of the tester based on the video information and extract the first forehead information; pre-process the first forehead information to obtain the second forehead information, and extract the preset channel information in the second forehead information to obtain the third forehead information; and perform noise reduction processing on the third forehead information to obtain the fourth forehead information, and transform the fourth forehead information to obtain a frequency domain signal, and obtain the heart rate value and the respiratory rate value according to the frequency signal.

[0104] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0105] An embodiment of the present application further provides a server, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the methods listed above when executing the computer program.

[0106] An embodiment of the present invention further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the step features included in the above-mentioned systems or devices that can be implemented by the computer program are implemented.

[0107] The processor referred to in this application may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0108] If the above integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or circuit that can carry the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0109] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned functional elements and circuits is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional elements and circuits as needed, that is, the internal structure of the circuit is divided into different functional elements or circuits to complete all or part of the functions described above. The functional elements and circuits in the embodiments can be integrated into a processing element, or each element can exist physically alone, or two or more elements can be integrated into one element. The above-mentioned integrated elements can be implemented in the form of hardware or in the form of software functional elements. In addition, the specific names of the functional elements and circuits are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the elements and circuits in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0110] Those of ordinary skill in the art will appreciate that the elements and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0111] References to "various embodiments", "in an embodiment", "one embodiment", or "an embodiment" throughout the specification, etc., mean that a particular feature, structure, or characteristic described with respect to an embodiment is included in at least one embodiment. Therefore, the appearance of the phrases "in various embodiments", "in some embodiments", "in an embodiment", or "in an embodiment", etc., in appropriate places throughout the specification, do not necessarily refer to the same embodiment. In addition, particular features, structures, or characteristics may be combined in any appropriate manner in one or more embodiments. Therefore, a particular feature, structure, or characteristic shown or described with respect to one embodiment may be combined in whole or in part with features, structures, or characteristics of one or more other embodiments without assuming that such a combination is not an illogical or non-functional limitation. Any directional references (e.g., plus, minus, upper, lower, up, down, left, right, leftward, rightward, top, bottom, above, below, vertical, horizontal, clockwise, and counterclockwise) are used for identification purposes to help the reader understand the present disclosure, and do not create limitations, particularly with respect to the location, orientation, or use of the embodiments.

[0112] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0113] Those of ordinary skill in the art will appreciate that the elements and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0114] In the embodiments provided by the present invention, it should be understood that the disclosed circuits / terminal devices and methods can be implemented in other ways. For example, the circuit / terminal device embodiments described above are only schematic. For example, the division of the circuits or components is only a logical function division. There may be other division methods in actual implementation, such as multiple components or circuits can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of circuits or components, which can be electrical, mechanical or other forms.

[0115] The components described as separate components may or may not be physically separated, and the components shown as components may or may not be physical components, that is, they may be located in one place or distributed on multiple network components. Some or all of the components may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0116] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A non-contact heart rate and respiratory rate detection method, characterized in that: include: Collecting video information of a preset area of ​​a subject, wherein the preset area includes the face of the subject; Identify the forehead of the subject based on the video information and extract first forehead information; Preprocessing the first forehead information to obtain second forehead information, and extracting preset channel information from the second forehead information to obtain third forehead information; and The third forehead information is subjected to noise reduction processing to obtain fourth forehead information, and the fourth forehead information is transformed to obtain a frequency domain signal, and a heart rate value and a respiratory rate value are obtained according to the frequency signal.

2. The method according to claim 1, characterized in that The step of identifying the forehead of the subject based on the video information and extracting first forehead information includes: Extracting facial information of the subject from the video information through three cascaded networks; The forehead of the subject is recognized based on the facial information of the subject and the first forehead information is extracted.

3. The method according to claim 2, characterized in that The three cascade networks include a candidate region generation network, a refinement network and an output network; The extracting of the facial information of the subject from the video information through three cascaded networks includes: The candidate region generation network performs preliminary feature extraction based on the video information to generate a first facial candidate window; The refinement network filters and selects the first candidate facial window to obtain a second candidate facial window; The output network performs deep feature extraction based on the second candidate window to obtain the target facial bounding box and facial key point information of the subject; Based on the target facial bounding box and the facial key point information, an image of a facial region containing a subject being detected is cropped from the video information and used as the facial information of the subject being detected.

4. The method according to claim 3, characterized in that The step of identifying the forehead of the subject based on the facial information of the subject and extracting the first forehead information comprises: Identifying the forehead of the subject by using edge features, line features, and four-corner-rectangle features of the face region image of the subject; First forehead information of a forehead area of ​​the subject is extracted based on the forehead of the subject.

5. The method according to claim 1, characterized in that The preprocessing of the first forehead information to obtain the second forehead information includes: Collect images upward on the first forehead information through a Gaussian pyramid to obtain an image sequence, call back the Gaussian pyramid, and subtract the first forehead information from the Gaussian pyramid to obtain a Laplacian pyramid; The Laplacian pyramid is low-pass filtered and down-sampled, and decomposed to obtain a low-pass approximate image, the low-pass approximate image is interpolated and filtered, and the interpolation of the low-pass approximate image and the image corresponding to the first forehead information is calculated to obtain a decomposed band-pass component, and then the low-pass approximate image is decomposed again at the next level to iteratively obtain a multi-scale decomposed forehead image, and then the Gaussian pyramid is used to amplify the color of the forehead area in the multi-scale decomposed forehead image to obtain the second forehead information.

6. The method according to claim 5, characterized in that The extracting the preset channel information from the second forehead information to obtain the third forehead information includes: Separating the three channels of R, G and B in the second forehead information; The G channel in the second forehead information is extracted as the third forehead information.

7. The method according to claim 1, characterized in that Performing noise reduction processing on the third forehead information to obtain fourth forehead information, transforming the fourth forehead information to obtain a frequency domain signal, and obtaining a heart rate value and a respiratory rate value according to the frequency domain signal, including: Performing noise reduction on the third frontal information by using a fourth-order Butterworth filter to obtain the fourth frontal information; Performing fast Fourier transform on the fourth forehead information to obtain the frequency domain signal; The frequency domain signal is filtered for non-heart rate frequency intervals, and frequency domain values ​​are obtained through a two-layer Laplace pyramid detection result, wherein the heart rate frequency interval is 1-2 Hz, and the respiratory rate frequency interval is 0.2-0.4 Hz, and a heart rate frequency domain graph and a respiratory rate frequency domain graph are obtained through band-pass filtering; Calculate the heart rate and respiratory rate values, specifically: V hr =60*f hr , Vhr——heart rate value; fhr——frequency value corresponding to the peak value in the frequency domain graph; V rr =60*f rr , V rr ——Respiration rate value; f rr ——The frequency value corresponding to the peak in the frequency domain diagram.

8. A non-contact heart rate and respiratory rate detection device, characterized in that: include: A shooting module, used for collecting video information of a preset area of ​​the subject, wherein the preset area includes the face of the subject; and A processing module is used to identify the forehead of the tester based on the video information and extract the first forehead information; pre-process the first forehead information to obtain the second forehead information, and extract the preset channel information in the second forehead information to obtain the third forehead information; and perform noise reduction processing on the third forehead information to obtain the fourth forehead information, and transform the fourth forehead information to obtain a frequency domain signal, and obtain the heart rate value and the respiratory rate value according to the frequency signal.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. A server comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

Images