Respiration rate measuring method and system fusing thoracic motion and non-contact signal

Abstract

The invention discloses a respiratory rate measuring method and system fusing thoracic motion and non-contact signals, and the method comprises the steps: carrying out the feature point recognition of a video frame, connecting the positions of feature points of left and right shoulders, and calculating the distance from the position of a nose feature point to the connecting line; the distance change serves as a thoracic motion signal, the signal is filtered, and a breathing signal based on thoracic motion is obtained; obtaining a region of interest based on the facial feature point positions; calculating an RGB signal in the region of interest, and extracting a non-contact PPG signal from the RGB signal; extracting a breathing mode signal from the non-contact PPG signal; performing fusion filtering on the breathing mode signal to obtain a breathing signal based on the non-contact PPG signal; and respectively extracting supplementary information of the signals from the breathing signal based on thoracic movement and the breathing signal based on the non-contact PPG signal, and simultaneously inputting the extracted supplementary information and the two signals into the trained neural network to obtain the breathing rate.

Description

Technical Field

[0001] The present invention relates to the technical field of respiratory rate measurement, and in particular to a respiratory rate measurement method and system integrating thoracic motion and non-contact signals. Background Art

[0002] As one of the basic life activities of the human body, breathing maintains the normal metabolism of the human body and the acid-base balance of the internal environment. In clinical practice, respiratory rate is an important monitoring physiological indicator. Certain diseases or health problems may show abnormal breathing patterns, such as lung infection, cardiac arrest, etc. At the same time, in emergency scenarios, monitoring of breathing can help medical staff assess the patient's condition and determine whether relevant emergency measures need to be taken.

[0003] At present, respiratory rate detection is often not taken seriously in clinical practice, because the most commonly used respiratory rate detection method is still the visual counting method. Medical staff estimate the respiratory rate by directly observing the rise and fall of the patient's chest and abdomen, counting the number of breaths the patient takes in one minute. Although this method is relatively simple, it takes time and manpower, and requires medical staff to have certain experience and professional ability. At the same time, it cannot achieve continuous real-time monitoring.

[0004] To solve this inconvenience, many devices use contact sensors to perform continuous respiratory monitoring, such as multi-parameter monitors and respiratory belts. Multi-parameter physiological monitors usually use the change in the relative distance between ECG electrodes to measure the rise and fall of the chest or detect changes in chest impedance to calculate the respiratory rate. This method requires sticking electrodes on the patient, and still requires experienced medical staff to assist in the detection, which affects the patient's normal actions. It is usually used in ward monitoring scenarios.

[0005] The breathing belt uses a strap to fix the sensor on the patient's chest, and measures the expansion and contraction of the chest to measure the respiratory rate, which may give the patient a certain sense of restraint. Although both contact methods can achieve real-time monitoring of the respiratory rate, both require the sensor to be in direct contact with the patient's body, which is not suitable for detection in daily life scenarios or when the patient has open wounds on the body.

[0006] There are many non-contact respiratory rate detection technologies currently available, which are usually based on thermal imaging cameras, chest movement detection or non-contact photoplethysmography (PPG) technology.

[0007] The technology based on thermal imaging cameras uses infrared thermal imaging cameras to detect the temperature changes in the nostril area when the subject breathes to detect the breathing rate. The detection accuracy is relatively high, but this technology has high requirements on the accuracy of the infrared thermal imaging cameras used. High-precision thermal imaging cameras are expensive and are not suitable for daily home use.

[0008] Methods based on chest movement detection usually use equipment such as bio-radar or depth cameras to detect the ups and downs of the chest caused by breathing. This technology is easy to implement, but it is easily interfered by the movement of other limbs of the subject. The subject's other large limb movements may mask the smaller chest movement, so this technology is more suitable for sleep breathing monitoring scenarios.

[0009] The technology based on non-contact PPG signals detects changes in blood perfusion of human skin through a visible light camera to achieve non-contact PPG signal detection, and then indirectly detects the respiratory rate from the PPG signal through a respiratory signal extraction algorithm. This method only requires the use of a low-cost visible light camera, and is more robust to motion than the method based on chest movement detection, making it more suitable for daily life scenarios. However, due to the complex mechanism of the impact of breathing on PPG signals, how to accurately extract respiratory signals from non-contact PPG signals is the focus of current research. Summary of the invention

[0010] In order to solve the deficiencies of the prior art, the present invention provides a respiratory rate measurement method and system that integrates chest movement and non-contact signals;

[0011] In one aspect, a method for measuring respiratory rate by fusing chest motion and non-contact signals is provided, comprising:

[0012] Obtain a video of the target to be measured; perform feature point recognition on the video frame to identify the location of the nose feature point, the left shoulder feature point, the right shoulder feature point, and the facial feature point;

[0013] Connect the feature point position of the left shoulder with the feature point position of the right shoulder, and calculate the distance from the feature point position of the nose to the connecting line; use the distance that changes over time as a chest movement signal, filter the chest movement signal, and obtain a breathing signal based on the chest movement;

[0014] Based on the positions of facial feature points, a region of interest is obtained; RGB signals in the region of interest are calculated, and a non-contact PPG signal is extracted from the RGB signal; several respiratory modality signals are extracted from the non-contact PPG signal; several respiratory modality signals are fused, and the fused signal is filtered to obtain a respiratory signal based on the non-contact PPG signal;

[0015] Supplementary information of the signals is extracted from the respiratory signal based on the chest movement and the respiratory signal based on the non-contact PPG signal, respectively, and the extracted supplementary information and the two signals are simultaneously input into the trained neural network to obtain the respiratory rate.

[0016] In another aspect, a respiratory rate measurement system integrating chest motion and non-contact signals is provided, comprising:

[0017] An acquisition module is configured to: acquire a video of a target to be measured; perform feature point recognition on the video frame to identify the location of a nose feature point, a left shoulder feature point, a right shoulder feature point, and a facial feature point;

[0018] The breathing signal extraction module based on thoracic movement is configured as follows: connecting the left shoulder feature point position and the right shoulder feature point position, calculating the distance from the nose feature point position to the connecting line; using the distance that changes over time as a thoracic movement signal, filtering the thoracic movement signal, and obtaining a breathing signal based on thoracic movement;

[0019] A respiratory signal extraction module based on a non-contact PPG signal is configured to: obtain a region of interest based on the position of facial feature points; calculate an RGB signal in the region of interest, and extract a non-contact PPG signal from the RGB signal; extract several respiratory modality signals from the non-contact PPG signal; fuse the several respiratory modality signals, filter the fused signal, and obtain a respiratory signal based on the non-contact PPG signal;

[0020] The respiratory rate prediction module is configured to extract supplementary information of the signal from the respiratory signal based on the chest movement and the respiratory signal based on the non-contact PPG signal, respectively, and input the extracted supplementary information and the two signals into the trained neural network at the same time to obtain the respiratory rate.

[0021] On the other hand, there is also provided an electronic device, comprising:

[0022] a memory for non-transitory storage of computer-readable instructions; and

[0023] a processor for executing the computer readable instructions,

[0024] When the computer-readable instructions are executed by the processor, the method described in the first aspect is executed.

[0025] On the other hand, a storage medium is provided, which non-temporarily stores computer-readable instructions, wherein when the non-temporary computer-readable instructions are executed by a computer, the instructions of the method described in the first aspect are executed.

[0026] On the other hand, a computer program product is provided, comprising a computer program, wherein the computer program is used to implement the method described in the first aspect when running on one or more processors.

[0027] The above technical solution has the following advantages or beneficial effects:

[0028] A visible light camera is used as a detection device, combined with a method based on chest movement and a method based on non-contact PPG signals, and the motion signal of the chest movement and the respiratory signal extracted from the non-contact PPG signal are collected at the same time, and finally the respiratory rate is calculated together through a signal fusion algorithm. The present invention combines the two methods to compensate for the sensitivity of the method based on chest movement to limb movement. At the same time, according to the influence of breathing on blood perfusion, three respiratory modal signals are extracted from the non-contact PPG signal, and the respiratory rate is extracted together with the motion signal. The robustness of the detection algorithm is improved without increasing the detection equipment, which is more suitable for daily life detection scenarios and saves medical and labor costs. The non-contact method can be used to measure the respiratory rate, which can be used for patients who cannot wear contact sensors due to open wounds on the body. At the same time, there is no need to wear complex sensors, and it can be used for daily respiratory rate detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0030] Figure 1 Schematic diagram of posture feature points and regions of interest in Example 1;

[0031] Figure 2 This is a work flow chart of Example 1;

[0032] Figure 3 It is a schematic diagram of a signal extraction method of a non-contact PPG signal according to Embodiment 1;

[0033] Figure 4 Schematic diagram of a neural network of Embodiment 1;

[0034] Among them, 1 is the nose feature point, 2 is the left shoulder feature point, 3 is the right shoulder feature point, and 4 is the facial area of ​​interest. DETAILED DESCRIPTION

[0035] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0036] Embodiment 1

[0037] This embodiment provides a method for measuring respiratory rate by fusing chest movement and non-contact signals;

[0038] like Figure 2 As shown, a respiratory rate measurement method integrating chest motion and non-contact signal comprises:

[0039] S101: Acquire a video of the target to be measured; perform feature point recognition on the video frame to identify the position of nose feature point 1, left shoulder feature point 2, right shoulder feature point 3 and facial feature point positions;

[0040] S102: Connect the left shoulder feature point position and the right shoulder feature point position, and calculate the distance from the nose feature point position to the connection line; use the distance that changes over time as a thorax motion signal, filter the thorax motion signal, and obtain a breathing signal based on thorax motion;

[0041] S103: based on the positions of facial feature points, obtaining a region of interest; calculating RGB signals in the region of interest, and extracting a non-contact PPG signal from the RGB signals; extracting a plurality of respiratory modality signals from the non-contact PPG signal; fusing the plurality of respiratory modality signals, filtering the fused signal, and obtaining a respiratory signal based on the non-contact PPG signal;

[0042] S104: Extracting supplementary information of the signals from the respiratory signal based on the chest movement and the respiratory signal based on the non-contact PPG signal respectively, and inputting the extracted supplementary information and the two signals into the trained neural network at the same time to obtain the respiratory rate.

[0043] Furthermore, the S101: obtaining a video of the target to be measured refers to obtaining a frontal video of the subject.

[0044] Further, the S101: performing feature point recognition on the video frame to identify the position of the nose feature point, the left shoulder feature point, the right shoulder feature point and the facial feature point includes:

[0045] Use the Mediapipe Pose open source tool to identify the locations of the nose feature points, the left shoulder feature points, and the right shoulder feature points;

[0046] Use the Mediapipe Face Mesh open source tool to identify the locations of facial landmarks.

[0047] Further, the S102: connecting the left shoulder feature point position and the right shoulder feature point position, calculating the distance from the nose feature point position to the connecting line; using the distance that changes over time as a thorax motion signal, filtering the thorax motion signal, and obtaining a breathing signal based on thorax motion, includes:

[0048] Assume that the coordinates of the nose feature point are (x0, y0), the coordinates of the left shoulder feature point are (x1, y1), and the coordinates of the right shoulder feature point are (x2, y2). The straight line connecting the left and right shoulders is Ax+By+C=0, where A, B, and C are:

[0049] A=y2-y1 (1)

[0050] B=x1-x2 (2)

[0051] C=x1*(y1-y2)+y1*(x2-x1) (3)

[0052] At this time, the distance d from the nose feature point to the straight line is:

[0053]

[0054] Calculate the d value of each frame image to obtain the motion signal of the thorax;

[0055] The motion signal is filtered by Butterworth bandpass using the frequency band of 0.15-0.5 Hz where the respiratory rate is located to obtain the respiratory signal based on the chest movement, which is recorded as S m .

[0056] It should be understood that the realization of chest cage motion signal extraction first requires the use of a visible light camera to shoot a video of the subject. In order to obtain the ups and downs of the chest of the subject, the video must at least contain the face and shoulders of the subject. The present invention uses feature points 0, 11, and 12 of the body feature points provided by the Mediapipe Pose open source tool to detect chest cage motion. Feature point 0 is selected as the nose position, feature point 11 is located at the left shoulder, and feature point 12 is located at the right shoulder. Figure 1 shown.

[0057] The sternum is connected to the clavicle, and the shoulders will move when the sternum moves up and down. In order to reduce the noise introduced by the left and right rotation of the head or body, the present invention selects the feature points of the left and right shoulders. When extracting the motion signal, it is first necessary to obtain the coordinate positions of the three feature points in a frame of the image, and then calculate the distance from the nose feature point to the straight line connecting the two shoulders.

[0058] Further, S103: based on the positions of facial feature points, a region of interest is obtained, including:

[0059] The facial area below the eyes and above the lips is selected as the facial region of interest 4.

[0060] Further, S103: calculating the RGB signal in the region of interest, including:

[0061] Calculate the mean pixel values ​​of the three channels of red (Red, R), green (Green, G), and blue (Blue, B) for all pixels in the region of interest and save them in their respective signal sequences;

[0062]

[0063]

[0064]

[0065] Where C is the set of pixels in the region of interest, r, g, b represent the red, green, and blue components of the pixels, respectively, i is the number of pixels in C, R, G, B are the mean signals of the red, green, and blue channels, respectively, and t is the time sampling point.

[0066] After all video frames are processed, RGB three-channel signals with the same time length and sampling rate as the video are obtained.

[0067] Further, S103: extracting a non-contact PPG signal from the RGB signal, including:

[0068]

[0069] Among the three channels of R, G, and B, the green channel has the best signal quality. This is because green light is more easily absorbed by red hemoglobin, and the reflected light contains more blood perfusion information. By calculating the ratio of the green channel to other channels, the blood perfusion change information can be extracted while suppressing the noise in the signal.

[0070] Furthermore, if Figure 3 As shown, S103: extracting a plurality of respiratory mode signals from the non-contact PPG signal; wherein the plurality of respiratory mode signals include: a respiratory induced intensity change signal, a respiratory induced amplitude change signal and a respiratory induced frequency change signal.

[0071] The respiratory induced intensity change signal is obtained by performing Butterworth bandpass filtering on the non-contact PPG signal with the frequency band of 0.15 to 0.5 Hz of the respiratory frequency as the passband.

[0072] It should be understood that due to the respiratory induced intensity change signal (denoted as S i ) is the change in blood perfusion baseline caused by the change in intrathoracic pressure, so it can be obtained by simply performing Butterworth bandpass filtering on the non-contact PPG signal with the respiratory frequency band of 0.15-0.5 Hz as the passband.

[0073] Among them, the breathing-induced amplitude change signal S a, firstly, the non-contact PPG signal is subjected to Butterworth bandpass filtering with the heart rate frequency band of 0.7-3 Hz as the passband to obtain the pulse wave signal; the pulse wave signal is processed by the incremental-merge segmentation (IMS) algorithm; after processing, the obtained signal only contains the peak and valley value points of the original pulse wave signal, and the amplitude of the pulse wave is obtained by subtracting the peak value from the previous valley value. The obtained value is plotted on the time scale, and the plotted signal is interpolated to make its sampling rate the same as the original signal to obtain S a .

[0074] Among them, the breathing induced frequency change signal S f , firstly, the non-contact PPG signal is subjected to Butterworth bandpass filtering with the heart rate frequency band of 0.7-3 Hz as the passband to obtain the pulse wave signal; the pulse wave signal is processed by the incremental-merge segmentation (IMS) algorithm; after processing, the obtained signal only contains the peak and valley value points of the original pulse wave signal, and the pulse wave period is obtained by subtracting the positions of the two peaks. The obtained value is plotted on the time scale, and the plotted signal is interpolated to make its sampling rate the same as the original signal to obtain S f .

[0075] Among them, the incremental merge segmentation (IMS) algorithm includes: connecting a line segment between two points separated by a fixed number of sampling points. If the slopes of adjacent line segments are the same positive or negative, the intersection of the two line segments is removed and merged into one line segment; if one slope is positive and the other is negative, the intersection is recorded as the segmentation point, and the sampling points whose slopes are not the same negative are deleted.

[0076] It should be understood that for the respiratory induced amplitude change signal (S a ) and respiratory induced frequency change signal (S f ), which corresponds to the change in the pulse wave waveform caused by breathing, so it is necessary to first perform Butterworth bandpass filtering on the non-contact PPG signal with the heart rate frequency band of 0.7-3Hz as the passband to obtain the pulse wave signal. a Corresponding to the change of cardiac output, that is, the change of pulse wave signal amplitude; S f Corresponding to the change of heart rate, that is, the change of the single pulse wave cycle time of the pulse wave signal. Extract S a and S f When the pulse wave signal is detected, the peak and valley points are detected to obtain the amplitude and frequency changes of the signal.

[0077] Since the pulse wave signal here may be disturbed by facial movement or changes in surrounding light, there is a certain amount of noise, which brings a lot of interference to the peak-valley value detection. The present invention first processes the pulse wave signal through the incremental-merge segmentation (IMS) algorithm, which connects a line segment between two points separated by a fixed number of sampling points. If the slopes of adjacent line segments have the same sign, the intersection of the two line segments is removed and merged into one line segment; if the slopes have different signs, the intersection is recorded as the segmentation point.

[0078] This algorithm can remove some noise whose time domain length is obviously shorter than the pulse wave period. After being processed by this algorithm, the obtained signal only contains the peak and valley value points of the original pulse wave signal. The peak value minus the previous valley value is the amplitude of the pulse wave, and the position of the two peaks is subtracted to get the pulse wave period. The obtained value is plotted on the time scale, and the plotted signal is interpolated to make its sampling rate the same as the original signal, that is, S a and S f .

[0079] Further, S103: fusing a plurality of respiratory modality signals, filtering the fused signal, and obtaining a respiratory signal based on the non-contact PPG signal, including:

[0080] The three respiratory modal signals are added and averaged, and then Butterworth bandpass filtered to obtain the respiratory signal S based on the non-contact PPG signal. p .

[0081] It should be understood that the respiratory signal is extracted based on the non-contact PPG signal. First, it is necessary to obtain a non-contact PPG signal. In order to obtain skin perfusion information, the camera needs to capture the facial area of ​​the person. The present invention uses the facial feature points provided in the Mediapipe Face Mesh open source tool, and selects the skin in the middle area of ​​the face below the eyes and above the lips as the region of interest. The feature points used for the region of interest division are the feature points [111,340,376,147,111] provided by Mediapipe Face Mesh. The feature points are connected in sequence to circle the region of interest. The region of interest is also identified for each frame of the video to achieve tracking of the skin in the same area.

[0082] Further, S104: extracting supplementary information of the signal from the respiratory signal based on the chest movement and the respiratory signal based on the non-contact PPG signal, respectively, wherein the supplementary information includes: signal-to-noise ratio, number of peaks and skewness.

[0083] The signal-to-noise ratio (SNR) is calculated by converting the signal into the frequency domain through Fourier transform, and calculating the ratio of the sum of the frequency domain components within the respiratory frequency band to the sum of the other frequency domain components.

[0084] The peak number is the number of signal peaks in the selected respiratory signal.

[0085] The calculation formula of skewness is as follows:

[0086]

[0087] Where N is the signal length, S is the signal, and σ is the signal standard deviation.

[0088] Furthermore, if Figure 4 As shown, the extracted supplementary information and the two signals are simultaneously input into the trained neural network to obtain the respiratory rate. The trained neural network includes:

[0089] The first branch, the second branch and the third branch are arranged in parallel;

[0090] The first branch includes: a first convolutional layer, a second convolutional layer, a first mean pooling layer and a first Flatten layer connected in sequence;

[0091] The second branch includes: a third convolutional layer, a fourth convolutional layer, a second mean pooling layer and a second Flatten layer connected in sequence;

[0092] The third branch includes: a first fully connected layer and a second fully connected layer connected in sequence;

[0093] The input end of the first convolutional layer is used to input a breathing signal based on chest movement;

[0094] The input end of the third convolutional layer is used to input a breathing signal based on a non-contact PPG signal;

[0095] An input end of the first fully connected layer is used to input a signal-to-noise ratio, a number of peaks, and a skewness;

[0096] The output end of the first Flatten layer, the output end of the second Flatten layer, and the output end of the second fully connected layer are all connected to the input end of the adder, the output end of the adder is connected to the input end of the third fully connected layer, the output end of the third fully connected layer is connected to the input end of the fourth fully connected layer, and the output end of the fourth fully connected layer outputs the predicted value of the breathing rate.

[0097] Furthermore, the extracted supplementary information and the two signals are simultaneously input into the trained neural network to obtain the respiratory rate. The training process of the trained neural network includes:

[0098] Constructing a training set, wherein the training set is two signals of known respiratory rate and supplementary information; the two signals are a respiratory signal based on chest movement and a respiratory signal based on a non-contact PPG signal;

[0099] The training set is input into the neural network to train the network. When the loss function value of the network no longer decreases, or the number of iterations reaches the set number, the training is stopped to obtain the trained neural network.

[0100] The weights in the network are obtained through pre-training, and the loss function is the L1 loss function, that is, the mean absolute error between the true value and the predicted value of the respiratory rate.

[0101] It should be understood that the respiratory frequency is extracted from the respiratory signal based on the movement of the thorax and the respiratory signal based on the non-contact PPG signal. Due to the different generation mechanisms of the two signals, the change in blood perfusion will be slightly delayed compared to the movement of the thorax, and there may be a certain phase difference between the two signals, so the two signals cannot be simply added and averaged when fused. However, the respiratory rate will not mutate in a short time, so the present invention intercepts two respiratory signals for a period of time, extracts features from the two signals respectively through a one-dimensional convolutional network, uses a mean pooling layer to reduce the number of parameters, and converts the extracted features into one dimension through a Flatten layer for data addition, and calculates the respiratory rate through a fully connected layer. Since the two signals may be disturbed by the movement of the subject and the ambient light, the present invention simultaneously extracts the signal-to-noise ratio, the number of peaks and the skewness of the two signals as supplementary information, and inputs them into the network to assist in the extraction of the respiratory rate.

[0102] The present invention can extract respiratory signals based on non-contact PPG signals. The facial skin contains a large number of capillaries, and the heart pumps blood rhythmically, causing the blood perfusion in the skin to change rhythmically. Hemoglobin in the blood is the main component that absorbs light, and changes in its content make the light reflected from the skin contain information about blood perfusion. Non-contact PPG technology uses a visible light camera as a detection device to detect changes in facial reflected light, thereby obtaining a non-contact PPG signal.

[0103] The breathing of the present invention has a modulating effect on blood perfusion (i.e., non-contact PPG signal), and the specific modulation mechanisms are divided into three types:

[0104] 1) Respiration-induced frequency changes: The autonomic response to breathing causes changes in heart rate, which are synchronized with the respiratory cycle. The heart rate increases during inspiration and decreases during expiration, resulting in respiratory-induced frequency changes in the pulse wave waveform.

[0105] 2) Respiration-induced intensity changes: Changes in intrathoracic pressure cause blood exchange between the pulmonary circulation and the systemic circulation. This leads to changes in the perfusion baseline, which is called respiratory-induced intensity changes.

[0106] 3) Respiratory-induced amplitude changes: Due to the reduced ventricular filling, the corresponding reduction in cardiac output leads to respiratory-induced amplitude changes, that is, changes in the strength of peripheral pulses.

[0107] The present invention first extracts non-contact PPG signals from skin pixels, and then uses filtering, incremental merging and segmentation algorithms, peak and valley detection algorithms, etc. to extract three respiratory modal signals from the non-contact PPG signals. After obtaining the modal signals, the three signals are fused to obtain a respiratory signal based on the non-contact PPG signal.

[0108] The present invention can realize the extraction of thorax motion signals. Respiratory movement is the process in which the contraction and relaxation of respiratory muscles cause the thorax to expand and contract rhythmically. In order to detect the expansion and contraction of the thorax, it is usually necessary to track a specific point on the thorax and detect the real-time movement of the point. In order to measure the movement of the point, it is necessary to select a point at other positions on the human body as a reference point, and use the distance change between the two points as the thorax motion signal. The reference point is usually selected from the face. However, using only two feature detection points is easily disturbed by movement, such as turning the head, body deflection, etc. The present invention identifies two feature points of the left and right shoulders, uses the feature point at the nose position as a reference point, calculates the distance from the reference point to the line connecting the left and right shoulders, and uses the distance change as the thorax motion signal to reduce the noise introduced by the left and right rotation of the head or body.

[0109] Embodiment 2

[0110] This embodiment provides a respiratory rate measurement system integrating chest movement and non-contact signals, including:

[0111] An acquisition module is configured to: acquire a video of a target to be measured; perform feature point recognition on the video frame to identify the location of a nose feature point, a left shoulder feature point, a right shoulder feature point, and a facial feature point;

[0112] The breathing signal extraction module based on thoracic movement is configured as follows: connecting the left shoulder feature point position and the right shoulder feature point position, calculating the distance from the nose feature point position to the connecting line; using the distance that changes over time as a thoracic movement signal, filtering the thoracic movement signal, and obtaining a breathing signal based on thoracic movement;

[0113] A respiratory signal extraction module based on a non-contact PPG signal is configured to: obtain a region of interest based on the position of facial feature points; calculate an RGB signal in the region of interest, and extract a non-contact PPG signal from the RGB signal; extract several respiratory modality signals from the non-contact PPG signal; fuse the several respiratory modality signals, filter the fused signal, and obtain a respiratory signal based on the non-contact PPG signal;

[0114] The respiratory rate prediction module is configured to extract supplementary information of the signal from the respiratory signal based on the chest movement and the respiratory signal based on the non-contact PPG signal, respectively, and input the extracted supplementary information and the two signals into the trained neural network at the same time to obtain the respiratory rate.

[0115] It should be noted that the acquisition module, the respiratory signal extraction module based on chest movement, the respiratory signal extraction module based on non-contact PPG signals, and the respiratory rate prediction module correspond to steps S101 to S104 in Embodiment 1, and the examples and application scenarios implemented by the modules are the same as those of the corresponding steps, but are not limited to the contents disclosed in Embodiment 1. It should be noted that the modules as part of the system can be executed in a computer system such as a set of computer executable instructions.

[0116] The description of each embodiment in the above embodiments has different emphases. For parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0117] The proposed system can be implemented in other ways. For example, the system embodiment described above is only illustrative, and the division of the modules is only a logical function division. In actual implementation, there may be other division methods, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0118] Embodiment 3

[0119] This embodiment also provides an electronic device, including: one or more processors, one or more memories, and one or more computer programs; wherein the processor is connected to the memory, and the one or more computer programs are stored in the memory. When the electronic device is running, the processor executes the one or more computer programs stored in the memory so that the electronic device executes the method described in the above embodiment one.

[0120] It should be understood that in this embodiment, the processor may be a central processing unit CPU, and the processor may also be other general-purpose processors, digital signal processors DSP, application-specific integrated circuits ASIC, off-the-shelf programmable gate arrays FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0121] The memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

[0122] In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in a processor or an instruction in the form of software.

[0123] The method in the first embodiment can be directly embodied as a hardware processor, or a combination of hardware and software modules in the processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware. To avoid repetition, it will not be described in detail here.

[0124] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with this embodiment can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0125] Embodiment 4

[0126] This embodiment further provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the method described in the first embodiment is completed.

[0127] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for measuring respiratory rate by integrating chest movement and non-contact signals, characterized in that: include: Acquire a video of the target to be measured; Perform feature point recognition on the video frame to identify the location of the nose feature point, the left shoulder feature point, the right shoulder feature point and the facial feature point; Connect the left shoulder feature point position and the right shoulder feature point position, and calculate the distance from the nose feature point position to the connecting line; use the distance that changes over time as a thorax motion signal, filter the thorax motion signal, and obtain a breathing signal based on thorax motion; Based on the positions of facial feature points, a region of interest is obtained; RGB signals in the region of interest are calculated, and non-contact PPG signals are extracted from the RGB signals; Extract several respiratory modality signals from the contactless PPG signal; Fusing several respiratory modality signals, filtering the fused signal, and obtaining a respiratory signal based on a non-contact PPG signal; Supplementary information of the signals is extracted from the respiratory signal based on the chest movement and the respiratory signal based on the non-contact PPG signal, respectively, and the extracted supplementary information and the two signals are simultaneously input into the trained neural network to obtain the respiratory rate.

2. The respiratory rate measurement method integrating chest movement and non-contact signal as claimed in claim 1, characterized in that: Connect the left shoulder feature point position and the right shoulder feature point position, and calculate the distance from the nose feature point position to the connecting line; The distance that changes with time is used as a chest movement signal, and the chest movement signal is filtered to obtain a breathing signal based on the chest movement, including: Assume that the coordinates of the nose feature point are (x0, y0), the coordinates of the left shoulder feature point are (x1, y1), and the coordinates of the right shoulder feature point are (x2, y2). The straight line connecting the left and right shoulders is Ax+By+C=0, where A, B, and C are: A=y2-y1 (1) B=x1-x2 (2) C=x1*(y1-y2)+y1*(x2-x1) (3) At this time, the distance d from the nose feature point to the straight line is: Calculate the d value of each frame image to obtain the motion signal of the thorax; The motion signal is filtered by Butterworth bandpass using the frequency band of 0.15-0.5 Hz where the respiratory rate is located to obtain the respiratory signal based on the chest movement, which is recorded as S m .

3. The respiratory rate measurement method integrating thoracic motion and non-contact signal as claimed in claim 1, characterized in that: Calculate the RGB signal in the region of interest, including: Calculate the average pixel values ​​of the three channels of red R, green G, and blue B for all pixels in the region of interest and save them in their respective signal sequences; Where C is the set of pixels in the region of interest, r, g, b represent the red, green, and blue components of the pixels, i is the number of pixels in C, R, G, B are the mean signals of the red, green, and blue channels, respectively, and t is the time sampling point; After all video frames are processed, RGB three-channel signals with the same time length and sampling rate as the video are obtained.

4. The respiratory rate measurement method integrating chest movement and non-contact signal as claimed in claim 1, characterized in that: Extracting non-contact PPG signals from RGB signals, including:

5. The respiratory rate measurement method integrating chest movement and non-contact signal as claimed in claim 1, characterized in that: Extracting a plurality of respiratory mode signals from the non-contact PPG signal; wherein the plurality of respiratory mode signals include: a respiratory induced intensity change signal, a respiratory induced amplitude change signal and a respiratory induced frequency change signal; The respiratory induced intensity change signal is obtained by performing Butterworth bandpass filtering on the non-contact PPG signal with the frequency band of 0.15 to 0.5 Hz of the respiratory frequency as the passband; Among them, the breathing-induced amplitude change signal S a , firstly, the non-contact PPG signal is subjected to Butterworth bandpass filtering with the heart rate frequency band of 0.7-3Hz as the passband to obtain the pulse wave signal; the pulse wave signal is processed by the incremental merging and segmentation algorithm; after the processing, the obtained signal only contains the peak and valley value points of the original pulse wave signal, and the amplitude of the pulse wave is obtained by subtracting the peak value from the previous valley value. The obtained value is plotted on the time scale, and the plotted signal is interpolated to make its sampling rate the same as the original signal, and S is obtained. a ; Among them, the breathing induced frequency change signal S f , firstly, the non-contact PPG signal is subjected to Butterworth bandpass filtering with the heart rate frequency band of 0.7-3 Hz as the passband to obtain the pulse wave signal; the pulse wave signal is processed by the incremental merging and segmentation algorithm; after the processing, the obtained signal only contains the peak and valley value points of the original pulse wave signal, and the positions of the two peaks are subtracted to obtain the pulse wave period, and the obtained value is plotted on the time scale, and the plotted signal is interpolated to make its sampling rate the same as the original signal, and S is obtained. f .

6. The respiratory rate measurement method integrating chest movement and non-contact signal as claimed in claim 1, characterized in that: Extracting complementary information of the signal from the respiratory signal based on chest movement and the respiratory signal based on the non-contact PPG signal, respectively, wherein the complementary information includes: signal-to-noise ratio, number of peaks and skewness; The signal-to-noise ratio is calculated by converting the signal into the frequency domain through Fourier transform, and calculating the ratio of the sum of the frequency domain components in the respiratory frequency band to the sum of the other frequency domain components; Among them, the peak number is the number of signal peaks in the selected respiratory signal; The calculation formula of skewness is as follows: Where N is the signal length, S is the signal, and σ is the signal standard deviation.

7. The respiratory rate measurement method integrating chest movement and non-contact signal as claimed in claim 1, characterized in that: The extracted supplementary information and the two signals are simultaneously input into the trained neural network to obtain the respiratory rate. The trained neural network includes: A first branch, a second branch and a third branch are arranged in parallel; the first branch comprises: a first convolutional layer, a second convolutional layer, a first mean pooling layer and a first Flatten layer connected in sequence; the second branch comprises: a third convolutional layer, a fourth convolutional layer, a second mean pooling layer and a second Flatten layer connected in sequence; the third branch comprises: a first fully connected layer and a second fully connected layer connected in sequence; The input end of the first convolutional layer is used to input a respiratory signal based on chest movement; the input end of the third convolutional layer is used to input a respiratory signal based on a non-contact PPG signal; the input end of the first fully connected layer is used to input a signal-to-noise ratio, a number of peaks, and a skewness; The output end of the first Flatten layer, the output end of the second Flatten layer, and the output end of the second fully connected layer are all connected to the input end of the adder, the output end of the adder is connected to the input end of the third fully connected layer, the output end of the third fully connected layer is connected to the input end of the fourth fully connected layer, and the output end of the fourth fully connected layer outputs the predicted value of the breathing rate.

8. A respiratory rate measurement system integrating thoracic motion and non-contact signals, characterized in that: include: An acquisition module is configured to: acquire a video of a target to be measured; Perform feature point recognition on the video frame to identify the location of the nose feature point, the left shoulder feature point, the right shoulder feature point and the facial feature point; The breathing signal extraction module based on thoracic movement is configured as follows: connecting the left shoulder feature point position and the right shoulder feature point position, calculating the distance from the nose feature point position to the connecting line; using the distance that changes over time as a thoracic movement signal, filtering the thoracic movement signal, and obtaining a breathing signal based on thoracic movement; A respiratory signal extraction module based on a non-contact PPG signal is configured to: obtain a region of interest based on the location of facial feature points; Calculate the RGB signal in the region of interest and extract the non-contact PPG signal from the RGB signal; Extracting a plurality of respiratory modal signals from the non-contact PPG signal; fusing the plurality of respiratory modal signals, filtering the fused signal, and obtaining a respiratory signal based on the non-contact PPG signal; The respiratory rate prediction module is configured to extract supplementary information of the signal from the respiratory signal based on the chest movement and the respiratory signal based on the non-contact PPG signal, respectively, and input the extracted supplementary information and the two signals into the trained neural network at the same time to obtain the respiratory rate.

9. An electronic device, comprising: a memory for non-transitory storage of computer readable instructions; as well as a processor for executing the computer readable instructions, When the computer-readable instructions are executed by the processor, the method described in any one of claims 1 to 7 is executed.

10. A storage medium, characterized in that: The computer-readable instructions are non-transitory stored, wherein when the non-transitory computer-readable instructions are executed by a computer, the instructions of the method according to any one of claims 1 to 7 are executed.

Images