A non-contact respiratory rate monitoring method, system and device

Abstract

The present invention provides a non-contact respiratory rate monitoring method, system, and device, relating to the fields of computer vision and image processing. The method comprises: collecting lossless video information and physiological signals of ICU patients; using a ROI marking module to mark the chest region in the lossless video information; using a ROI tracking module to track the chest region of the ICU patient; training a spatiotemporal feature learning module based on the physiological signals; using the trained spatiotemporal feature learning module to generate a respiratory curve based on the ROI region video image of the chest region; and outputting the ICU patient's respiratory rate based on the respiratory curve; the spatiotemporal feature learning module includes a respiratory curve extractor and a respiratory rate estimator; both the respiratory curve extractor and the respiratory rate estimator are deep neural network models. The present invention can predict the respiratory rate of ICU patients in real medical scenarios.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and image processing, and in particular to a non-contact respiratory rate monitoring method, system and device. Background Art

[0002] Whether in hospital clinical treatment or daily exercise, measuring and monitoring respiratory rate plays a vital role. As a key physiological parameter, respiratory rate can promptly reflect the condition of lung-related diseases. For example, a slowed respiratory rate may be caused by a decreased metabolic rate, shock, or significantly increased intracranial pressure. Possible causes of an accelerated respiratory rate include hyperthyroidism, pneumonia, pulmonary embolism, pleurisy, etc.

[0003] Broadly speaking, the theory behind measuring respiratory rate is based on the rise and fall of the chest area during inhalation and exhalation. One rise and fall of the chest is counted as one breath, and the number of breaths in one minute is counted as the respiratory rate. Doctors sometimes roughly infer a patient's respiratory rate based on the rise and fall of the chest, thereby roughly judging the patient's physical condition. More accurately, the basic principle of measuring respiratory rate using commonly used vital signs monitors in clinical practice is the thoracic impedance theory, which states that when two electrodes are attached to the subject's chest area, the contraction and relaxation of the abdominal and intercostal muscles causes changes in the body's electrical resistance, thereby enabling the measurement of respiratory rate.

[0004] However, contact-based respiratory rate detection methods can cause many inconveniences. In clinical monitoring, the respiratory rate measurement process is often very complicated and requires professionally trained medical staff to operate and handle. In addition, considering public health factors, the sensor probe needs to be strictly disinfected before each use. The contact of the sensor may also cause irritation to some patients with fragile skin or trauma, and increase the risk of cross-infection during infectious disease pandemics.

[0005] The contactless respiratory rate detection method based on computer vision can well solve the above-mentioned more difficult problems. This method is also based on the movement of the chest area to achieve respiratory rate measurement, but only requires a video of the specified area without having to touch the skin, so it has the advantages of long-term monitoring and user-friendly. Most of the classic visual measurement methods are based on analyzing the changes in pixel brightness in the region of interest (ROI) to obtain the respiratory curve. The current mainstream contactless method also uses deep neural networks to autonomously learn the features of the ROI on the training set, so that the predicted value of the respiratory rate on the test set can be obtained. However, the research subjects of most contactless respiratory rate estimation methods are healthy adults, and the experimental environment is a dedicated laboratory with bright light, which cannot be applied to real medical scenarios. The effective application range of this measurement method is very limited. Summary of the Invention

[0006] The purpose of the present invention is to provide a non-contact respiratory rate monitoring method, system and device to solve the problem that the current non-contact respiratory rate estimation method cannot be applied in real medical scenarios and has a limited scope of application.

[0007] To achieve the above object, the present invention provides the following solutions:

[0008] A non-contact respiratory rate monitoring method comprising:

[0009] Collect lossless video information and physiological signals of ICU patients; the physiological signals include electrocardiogram, respiratory rate signal, non-invasive blood pressure, blood oxygen concentration and pulse rate;

[0010] Marking the chest area in the lossless video information using a ROI marking module;

[0011] Use the ROI tracking module to track the chest area of ICU patients;

[0012] training a spatiotemporal feature learning module according to the physiological signal;

[0013] The trained spatiotemporal feature learning module is used to generate a respiratory curve based on the ROI area video image of the chest area, and the respiratory rate of the ICU patient is output based on the respiratory curve; the spatiotemporal feature learning module includes a respiratory curve extractor and a respiratory rate estimator; the respiratory curve extractor and the respiratory rate estimator are both deep neural network models; the respiratory curve extractor is used to generate a respiratory curve based on the ROI area video image of the chest area; the respiratory rate estimator is used to output the respiratory rate based on the respiratory curve.

[0014] Optionally, the collecting of lossless video information and physiological signals of ICU patients further includes:

[0015] Marking a chest region in the lossless video information and using the chest region as a ROI region;

[0016] Using the KLT optical flow algorithm to perform ROI tracking on the ROI area, adjusting the resolution of the ROI area to 128×128, and generating a video image of the ROI area of the chest area;

[0017] Read the true value of physiological signals from the text file;

[0018] The ROI area video image of the chest area and the true value of the physiological signal are saved in an h5 file to construct a training set and a test set.

[0019] Optionally, the adopting of a deep learning method to train and learn a deep neural network model based on the lossless video information and the physiological signal specifically includes:

[0020] The respiratory curve extractor is trained using the ROI region video image in the training set, and the first weight of the deep neural network model is determined according to the ROI region video image in the test set and the true value of the respiratory rate signal in the true value of the physiological signal; the respiratory curve extractor is an unsupervised deep learning model;

[0021] determining a breathing curve according to the first weight;

[0022] The respiratory rate estimator is trained using the simulated signal of the true value of the physiological information in the training set and the true value of the simulated signal, and a second weight of the deep neural network model is generated according to the respiratory curve; the respiratory rate estimator is a supervised deep learning model;

[0023] The respiratory rate test result is determined according to the second weight, and a deep neural network model is generated.

[0024] Optionally, the breathing curve extractor specifically includes: a 3D convolution layer, a spatiotemporal signal collection layer, and a spatiotemporal sampling layer arranged in sequence;

[0025] The video image of the ROI area passes through different 3D convolution layers in sequence, and then passes through an adaptive average pooling layer to be down-sampled in the spatial dimension. The signal in the spatiotemporal collection layer is sampled in the temporal and spatial dimensions by the spatiotemporal sampling layer to generate multiple respiratory signal segments; and the respiratory curve is generated based on the multiple respiratory signal segments.

[0026] Optionally, the loss function of the breathing curve extractor is Loss=L pos +L neg ;in, L pos is the positive loss term, N is the number of video image slices in the ROI area, and f i is the PSD of the i-th feature from the video image of the first ROI region, f j is the PSD of the jth feature from the video image of the first ROI area, f' i is the PSD of the i-th feature from the video image of the second ROI area, f' j is the PSD of the jth feature from the video image of the second ROI area; L neg Negative loss item.

[0027] Optionally, the respiratory rate estimator specifically includes: two one-dimensional convolutional layers, three LSTM layers and one dense layer;

[0028] Generate a simulation signal sequence using the breathing curve, and input the simulation signal sequence into two one-dimensional convolutional layers to generate a first feature sequence with 64 channels;

[0029] Input the first feature sequence into a three-layer LSTM layer to generate a second feature sequence with 32 channels;

[0030] The second feature sequence is input into the dense layer for linearization calculation to generate a feature sequence with a channel number of 1, and the respiratory frequency is output.

[0031] Optionally, the loss function of the respiratory rate estimator is Where n is the batch size during the training of the respiratory rate estimator, k is the simulated signal index, is the predicted value of the simulation signal, f sim (k) is the true label value of the simulated signal.

[0032] A non-contact respiratory rate monitoring system comprising:

[0033] Video information and physiological signal acquisition module, used to collect lossless video information and physiological signals of ICU patients; the physiological signals include electrocardiogram, respiratory rate signal, non-invasive blood pressure, blood oxygen concentration and pulse rate;

[0034] an ROI marking module, configured to mark the chest area in the lossless video information;

[0035] ROI tracking module, used to track the chest area of ICU patients;

[0036] a spatiotemporal feature learning module training module, configured to train the spatiotemporal feature learning module according to the physiological signal;

[0037] The trained spatiotemporal feature learning module is used to generate a respiratory curve based on the ROI area video image of the chest area, and output the respiratory rate of the ICU patient based on the respiratory curve; the spatiotemporal feature learning module includes a respiratory curve extractor and a respiratory rate estimator; the respiratory curve extractor and the respiratory rate estimator are both deep neural network models; the respiratory curve extractor is used to generate a respiratory curve based on the ROI area video image of the chest area; the respiratory rate estimator is used to output the respiratory rate based on the respiratory curve.

[0038] An electronic device includes a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to perform any of the above-mentioned non-contact respiratory rate monitoring methods.

[0039] A computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the non-contact respiratory rate monitoring method as described in any one of the above items is implemented.

[0040] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects: The present invention provides a non-contact respiratory rate monitoring method, system and equipment, which collects lossless video information and physiological signals of ICU patients in ICU wards in real medical scenarios, trains and learns to generate a deep neural network model, and thus predicts the respiratory rate of ICU patients in real medical scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0042] Figure 1 This is a flow chart of the non-contact respiratory rate monitoring method provided by the present invention;

[0043] Figure 2 A schematic diagram of the collection device provided by the present invention;

[0044] Figure 3 The data processing flow chart provided by the present invention;

[0045] Figure 4 This is a flow chart of the neural network algorithm provided by the present invention;

[0046] Figure 5 This is a structural diagram of the deep neural network algorithm provided by the present invention;

[0047] Figure 6 This is a structural diagram of the breathing curve extractor provided by the present invention;

[0048] Figure 7 This is a structural diagram of the respiratory rate estimator provided by the present invention;

[0049] Figure 8 This is the ICU ward data collection scene diagram provided by the present invention;

[0050] Figure 9 This is a schematic diagram of the ROI marking provided by the present invention;

[0051] Figure 10 This is an example diagram of the simulation signal data set provided by the present invention; Figure 10 (a) is an example of a breathing signal. Figure 10(b) is an example of a PPG signal. Figure 10 (c) is an example diagram of a sinusoidal signal;

[0052] Figure 11 This is a schematic diagram of the test results of the respiratory curve extractor provided by the present invention; Figure 11 (a) is the breathing curve of test sample 1, Figure 11 (b) is the PSD graph corresponding to test sample 1. Figure 11 (c) is the breathing curve of test sample 2; Figure 11 (d) is the PSD graph corresponding to test sample 1;

[0053] Figure 12 This is a dynamic prediction curve diagram of respiratory rate provided by the present invention. DETAILED DESCRIPTION

[0054] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0055] The purpose of the present invention is to provide a non-contact respiratory rate monitoring method, system and equipment that can predict the respiratory rate of ICU patients in real medical scenarios.

[0056] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0057] Example 1

[0058] like Figure 1 As shown, the present invention provides a non-contact respiratory rate monitoring method, comprising:

[0059] Step 101: Collect lossless video information and physiological signals of ICU patients; the physiological signals include electrocardiogram, respiratory rate signal, non-invasive blood pressure, blood oxygen concentration and pulse rate.

[0060] In practical applications, the acquisition device is mainly divided into three modules: Figure 2 This is a schematic diagram of the collection device provided by the present invention, as shown in FIG. Figure 2 shown.

[0061] The video information acquisition module is used to obtain lossless video footage of the ICU patient's chest area. Position the camera directly above the ICU bed and adjust the camera angle to fully capture the patient's chest area. Keep the camera stationary while capturing video information.

[0062] The physiological signal acquisition module is used to acquire physiological signals from ICU patients, including electrocardiogram (ECG), respiratory rate, non-invasive blood pressure, blood oxygen concentration, and pulse rate. This module uses a traditional contact-type medical physiological signal acquisition device to obtain real-time and accurate physiological pulse signals, providing reference data for testing the respiratory rate prediction model.

[0063] The control terminal is used to process and store the acquired video information and physiological signals. The control terminal uses a high-speed storage device to store the lossless video information collected by the video information acquisition module and stores the physiological pulse signals collected by the physiological signal acquisition module as discrete formatted data.

[0064] The camera used in the present invention is HIKVISION DS-U102D, the contact medical monitoring device used is BROADSIMS TeleTouch, and the computing and control device (control terminal) used is NVIDIA Jetson Orin.

[0065] For contact devices, the respiratory rate sampling frequency is 500Hz, and the measurement accuracy can reach ±2%. The physiological signal waveform and value are displayed in real time on the screen, and the collected physiological signal values are saved as a text file every 2 seconds.

[0066] In practical applications, step 101 also includes: marking the chest area in the lossless video information and using the chest area as the ROI area; using the KLT optical flow algorithm to perform ROI tracking on the ROI area, adjusting the resolution of the ROI area to 128×128, and generating a ROI area video image of the chest area; reading the true value of the physiological signal in the text file; saving the ROI area video image of the chest area and the true value of the physiological signal in an h5 file to construct a training set and a test set.

[0067] Step 102: Mark the chest area in the lossless video information using a ROI marking module.

[0068] Step 103: Track the chest area of the ICU patient using the ROI tracking module.

[0069] Step 104: training a spatiotemporal feature learning module according to the physiological signal.

[0070] Step 105: Generate a respiratory curve based on the ROI area video image of the chest region using the trained spatiotemporal feature learning module, and output the respiratory rate of the ICU patient based on the respiratory curve; the spatiotemporal feature learning module includes a respiratory curve extractor and a respiratory rate estimator; the respiratory curve extractor and the respiratory rate estimator are both deep neural network models; the respiratory curve extractor is used to generate a respiratory curve based on the ROI area video image of the chest region; the respiratory rate estimator is used to output the respiratory rate based on the respiratory curve.

[0071] In practical applications, an unsupervised learning method is used to train the respiratory curve extractor, and a transfer learning method is used to train the respiratory rate estimator.

[0072] In practical applications, the step 105 specifically includes: training the respiratory curve extractor with the ROI area video image in the training set, determining the first weight of the deep neural network model according to the ROI area video image in the test set and the respiratory rate signal true value in the physiological signal true value; the respiratory curve extractor is an unsupervised deep learning model; according to the first weight (i.e. Figure 4 The final model weight 1 in the training set is used to determine the respiratory curve; the simulation signal of the true value of the physiological information in the training set and the true value of the simulation signal are used to train the respiratory rate estimator, and the second weight of the deep neural network model is generated according to the respiratory curve (i.e., Figure 4 The final model weight in 2); the respiratory rate estimator is a supervised deep learning model; the respiratory rate test result is determined according to the second weight to generate a deep neural network model.

[0073] In practical applications, the breathing curve extractor specifically includes: a 3D convolution layer, a spatiotemporal signal collection layer, and a spatiotemporal sampling layer arranged in sequence; the video image of the ROI area passes through different 3D convolution layers in sequence, and then passes through an adaptive average pooling layer to be down-sampled in the spatial dimension, and then the signal in the spatiotemporal collection layer is sampled in the time and space dimensions respectively through the spatiotemporal sampling layer to generate multiple breathing signal segments; the breathing curve is generated based on the multiple breathing signal segments.

[0074] In practical applications, the loss function of the breathing curve extractor is Loss=L pos +L neg ;in, L pos is the positive loss term, N is the number of video image slices in the ROI area, and f i is the PSD of the i-th feature from the video image of the first ROI region, f j is the PSD of the jth feature from the video image of the first ROI area, f'i is the PSD of the i-th feature from the video image of the second ROI area, f' j is the PSD of the jth feature from the video image of the second ROI area; L neg Negative loss item.

[0075] In practical applications, the respiratory rate estimator specifically includes: two one-dimensional convolutional layers, three LSTM layers and one dense layer; the respiratory curve is used to generate a simulated signal sequence, and the simulated signal sequence is input into two one-dimensional convolutional layers to generate a first feature sequence with 64 channels; the first feature sequence is input into three LSTM layers to generate a second feature sequence with 32 channels; the second feature sequence is input into the dense layer for linearization calculation to generate a feature sequence with 1 channel, and the respiratory frequency is output.

[0076] In practical applications, the loss function of the respiratory rate estimator is Where n is the batch size during the training of the respiratory rate estimator, k is the simulated signal index, is the predicted value of the simulation signal, f sim (k) is the true label value of the simulated signal.

[0077] In practical applications, the present invention includes a model learning phase and a respiratory rate measurement phase, such as Figure 3-Figure 4 As shown in the figure, the model learning stage processes the collected ICU patient video information and the corresponding real-time physiological signals, and uses deep learning methods to train and learn a deep neural network model for use in the respiratory rate measurement stage.

[0078] (1) The model learning stage includes the ROI labeling module, the ROI tracking module and the feature learning module.

[0079] The ROI marking module is used to mark the chest area in video information. In the captured video, an ICU patient lies in a bed, covered with bedding. The chest area fluctuates with the patient's breathing, containing rich respiratory rate information. In a given frame, a rectangular region, the ROI, is manually defined to cover the most visible part of the chest as possible, facilitating feature extraction and learning by the feature learning module.

[0080] The ROI tracking module is used to track the chest area of ICU patients. Patients lying in bed and breathing naturally inevitably produce small movements. Therefore, the Kanade-Lucas-Thomas (KLT) optical flow tracking algorithm, leveraging the principles of optical flow estimation, tracks multiple feature points within the ROI rectangle. This provides the chest area location information for all frames after the ROI is marked, significantly reducing the time and effort required to manually label each video frame.

[0081] The feature learning module is used to obtain a specific deep learning model. This module is mainly divided into two parts: breathing curve extractor and breathing rate estimator, such as Figure 5 As shown in the figure, the deep learning model corresponding to the respiratory curve extractor is an unsupervised deep learning model, which consists of a multi-layer structure, mainly a 3D convolution layer, a spatiotemporal signal collection layer, and a spatiotemporal sampling layer. The deep learning model corresponding to the respiratory rate estimator is a supervised deep learning model, which consists of two 1D convolution layers, three long short-term memory (LSTM) layers, and a dense layer.

[0082] The first part is the breathing curve extractor, such as Figure 6 As shown in the figure. First, a pair of ROI frame images of size 1800*128*128*3 are input, each of which passes through different 3D convolution layers and then downsamples in the spatial dimension through an adaptive average pooling layer to obtain a spatiotemporal signal collection layer. The spatial dimension can be set through the pooling operation. Then, the signal in the spatiotemporal collection layer is sampled in the temporal and spatial dimensions through the spatiotemporal sampling layer to obtain multiple respiratory signal segments. The spatial resolution is selected to be 2×2 and the time length is selected to be 10s for spatiotemporal sampling.

[0083] By transforming from the time domain to the frequency domain, a power spectral density (PSD) plot is obtained. Within the same video sample, within a small time period and within a certain neighborhood, respiratory signals are similar; however, they differ between different video samples. Based on this principle, the values of similar items are amplified and those of different items are reduced to calculate the contrastive loss function, which is used to train unsupervised deep learning models.

[0084] The second part is the respiratory rate estimator, such as Figure 7As shown in the figure. To improve training efficiency, a morphological physiological signal simulation dataset generated by the neurokit2 toolkit was used as the training set for the respiratory rate estimator. A one-dimensional simulated signal sequence with a length of 500 frames was input into two one-dimensional convolutional layers to obtain a feature sequence with 64 channels. This was then passed through three LSTM layers to obtain a feature sequence with 32 channels. Finally, a dense layer was used for linearization calculation to convert the feature sequence into a value with a channel of 1, which was output as a respiratory rate value. The mean square error (MSE) was used as the loss function to calculate the degree of difference between the estimated physiological signal values and the actual physiological signal values. Backpropagation was performed to continuously optimize the model parameters.

[0085] (2) In the respiratory rate measurement stage, the trained deep learning model is used to predict the respiratory rate information of ICU patients by inputting video information of ICU patients.

[0086] The respiratory rate measurement stage has a similar structure to the feature learning module, consisting primarily of a respiratory curve extractor and a respiratory rate estimator. The respiratory curve extractor takes as input a video image of the ICU patient's chest region (ROI) and outputs the extracted respiratory curve. The respiratory rate estimator takes as input the respiratory curve obtained in the previous step and outputs the respiratory rate value. This stage differs from the feature learning module in that it only performs forward computations on the respiratory signal data stream, without backpropagation. First, features are extracted from the ROI video image, followed by spatiotemporal sampling to generate a respiratory curve sequence. This sequence is then fed into the respiratory rate estimator, which then outputs the respiratory rate value.

[0087] Training strategy:

[0088] The training process of a deep neural network for contactless respiratory rate measurement of ICU patients can be divided into two stages:

[0089] The first stage: training the breathing curve extractor, whose neural network structure is as follows Figure 6 As shown in the figure, the extractor is trained using an unsupervised learning method. The training input is the video signal of the chest area of ICU patients. The video signal is input into the neural network in pairs to calculate the contrast loss function. It is defined as Loss = L pos +L neg .

[0090] in,

[0091] The contrastive loss function consists of a positive term and a negative term. The positive term measures the similarity between video signals, while the negative term measures the difference between them. During training, the model parameters are continuously optimized to minimize the contrastive loss function.

[0092] The second stage: training the respiratory rate estimator, whose neural network structure is as follows Figure 7 As shown in Figure 2, the estimator is trained using supervised learning. The input of the training process is the morphological simulated physiological signal data generated by the neurokit2 toolkit, and the corresponding generated true label value is used as the reference data. The simulated physiological signal mainly consists of respiratory signal (RSP), photoplethysmography signal (PPG) and sinusoidal signal, as shown in Figure 2. Figure 10 As shown. The loss function used in the training process is MSE, which is defined as

[0093] Step 103: Predict the respiratory rate of the ICU patient based on the deep neural network model.

[0094] This invention is an experimental study conducted based on data collected in a clinical medical setting in the ICU ward of a hospital.

[0095] The present invention collected video data from a total of 405 patients, of which 37.53% were male patients and 62.47% were female patients. The average age of the patients was 67.82 years. For each patient, 1-4 videos were collected, with a total duration of approximately 10 minutes. The total duration of video information in the clinical data set was 67.5 hours. Traditional contact physiological signal monitoring equipment collects physiological parameters every 2 seconds, transmits them to the control terminal, and saves them as text files. For each patient, the number of text files of real-time physiological signals is approximately 300.

[0096] Example 2

[0097] In order to execute the method corresponding to the above embodiment 1 and achieve the corresponding functions and technical effects, a non-contact respiratory rate monitoring system is provided below, including:

[0098] The video information and physiological signal acquisition module is used to collect lossless video information and physiological signals of ICU patients; the physiological signals include electrocardiogram, respiratory rate signal, non-invasive blood pressure, blood oxygen concentration and pulse rate.

[0099] an ROI marking module, configured to mark the chest area in the lossless video information;

[0100] ROI tracking module, used to track the chest area of ICU patients;

[0101] a spatiotemporal feature learning module training module, configured to train the spatiotemporal feature learning module according to the physiological signal;

[0102] The trained spatiotemporal feature learning module is used to generate a respiratory curve based on the ROI area video image of the chest area, and output the respiratory rate of the ICU patient based on the respiratory curve; the spatiotemporal feature learning module includes a respiratory curve extractor and a respiratory rate estimator; the respiratory curve extractor and the respiratory rate estimator are both deep neural network models; the respiratory curve extractor is used to generate a respiratory curve based on the ROI area video image of the chest area; the respiratory rate estimator is used to output the respiratory rate based on the respiratory curve.

[0103] Data collection process: Data collection scenarios such as Figure 8 The embodiment of the present invention collects video information and corresponding real-time physiological signal data of patients in a clinical scenario of an ICU ward in a hospital.

[0104] The present invention uses a standard consumer-grade video capture device and a dedicated contact-type medical monitoring device as the physiological signal collector. The collected physiological signals include ECG, respiratory rate, non-invasive blood pressure, blood oxygen concentration, and pulse rate. A computer controller controls the terminal to store the collected video information and the corresponding physiological signal data.

[0105] Data processing flow: The resolution of the ICU patient video data collected by the present invention is 1440×1080 or 1280×720. In the collected data set, the video data with Richmond Agitation-Sedation Scale (RASS) not greater than 0 and obvious changes in the chest area are selected as the training set and test set of the deep neural network. The data processing flow is as follows Figure 3 The present invention uses 103 training data sets and 44 test data sets. The data input to the neural network is 1800 frames long and 128×128 in size. The detailed process of data processing is as follows.

[0106] 1) Chest area marking: Since the chest area fluctuates most significantly during breathing, it is easier to extract useful respiratory information, so the chest area is selected as the ROI area. However, ICU patients are covered with bedding and wear a variety of body monitoring equipment, and the chest area is heavily obstructed. Therefore, the present invention uses a manual marking method to mark the chest area in the first frame of each ICU patient video. When manually selecting the ROI rectangle, the area with obvious fluctuations is covered as completely as possible. The ROI marking process is as follows: Figure 9As shown, the blue rectangle is the ROI area.

[0107] 2) ROI tracking algorithm: After manually marking the ROI of the first frame, the KLT optical flow tracking algorithm is used to track and automatically mark the ROI of each subsequent frame. Figure 9 As shown in the figure, the dots within the box are feature points calculated by the KLT tracking algorithm. For each ICU patient, 1800 ROI frames were marked and resized to 128×128 resolution. The video image sequence of the ROI region was then saved to an h5 file. Simultaneously, the real-time physiological signal ground truth was read from a text file and saved together with the ROI region video image information in the same h5 file.

[0108] Deep neural network algorithm process: For details on the training process of the deep neural network algorithm provided by this invention, please refer to the training strategy section in the technical solution. The training process is as follows: Figure 4 shown.

[0109] Respiratory rate measurement effect: The ICU patient test set data is input into the trained neural network to obtain the predicted respiratory rate value, which is compared with the true respiratory value recorded by the contact monitoring device. The test process is divided into two stages: first, the respiratory curve extractor outputs the respiratory curve and the respiratory rate value calculated by PSD; then the respiratory curve sequence is input into the respiratory rate estimator, and the output is the predicted respiratory rate value. Both the respiratory curve extractor and the respiratory rate estimator use a sliding window to slice and traverse the input sequence. By changing the length and step size of the sliding window, different respiratory rate prediction results can be obtained. The measurement results are as follows: Figure 11 and Figure 12 shown.

[0110] Example 3

[0111] An embodiment of the present invention provides an electronic device including a memory and a processor. The memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to perform the non-contact respiratory rate monitoring method provided in embodiment 1.

[0112] In practical applications, the above-mentioned electronic device may be a server.

[0113] In practical applications, an electronic device includes at least one processor, a memory, a bus, and a communication interface.

[0114] Wherein: the processor, the communication interface, and the memory communicate with each other via a communication bus.

[0115] Communication interface, used to communicate with other devices.

[0116] The processor is used to execute the program, and specifically can execute the method described in the above embodiment.

[0117] Specifically, the program may include program codes including computer operation instructions.

[0118] The processor may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention. The one or more processors included in the electronic device may be processors of the same type, such as one or more CPUs, or processors of different types, such as one or more CPUs and one or more ASICs.

[0119] Memory is used to store programs. The memory may include high-speed RAM memory, and may also include non-volatile memory (non-volatile memory), such as at least one disk storage.

[0120] Based on the description of the above embodiments, the present invention provides a storage medium on which computer program instructions are stored. The computer program instructions can be executed by a processor to implement the method described in any embodiment.

[0121] The non-contact respiratory rate monitoring system provided in the embodiments of the present application exists in various forms, including but not limited to:

[0122] (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and are primarily designed to provide voice and data communications. These terminals include smartphones (e.g., iPhones), multimedia phones, feature phones, and low-end phones.

[0123] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, have computing and processing capabilities, and generally also have mobile Internet access capabilities. These terminals include PDAs, MIDs, and UMPC devices, such as the iPad.

[0124] (3) Portable entertainment devices: These devices can display and play multimedia content. These devices include audio and video players (such as iPods), handheld game consoles, e-books, smart toys, and portable car navigation devices.

[0125] (4) Other electronic devices with data interaction functions.

[0126] Thus far, specific embodiments of the present subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing may be advantageous.

[0127] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0128] For the convenience of description, the above devices are described in terms of their functions and are divided into various units and described separately. Of course, when implementing the present application, the functions of each unit can be implemented in the same one or more software and / or hardware. It should be understood by those skilled in the art that the embodiments of the present application can be provided as methods, systems or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0129] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0130] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0131] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0132] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0133] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0134] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM),

[0135] A digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, disk storage or other magnetic storage device, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0136] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0137] The present application may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0138] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0139] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only intended to help understand the method and core concept of the present invention. At the same time, those skilled in the art will find that the specific implementation methods and application scopes may vary based on the concept of the present invention. In summary, the contents of this specification should not be construed as limiting the present invention.

Claims

1. A non-contact respiratory rate monitoring method, characterized in that: include: Collect lossless video information and physiological signals of ICU patients; The physiological signals include electrocardiogram, respiratory rate signal, non-invasive blood pressure, blood oxygen concentration and pulse rate; Marking the chest area in the lossless video information using a ROI marking module; Use the ROI tracking module to track the chest area of ICU patients; training a spatiotemporal feature learning module according to the physiological signal; A trained spatiotemporal feature learning module is used to generate a respiratory curve based on the ROI region video image of the chest region, and the respiratory rate of the ICU patient is output based on the respiratory curve; the spatiotemporal feature learning module includes a respiratory curve extractor and a respiratory rate estimator; the respiratory curve extractor is an unsupervised deep learning model, including: a 3D convolution layer, a spatiotemporal signal collection layer, and a spatiotemporal sampling layer arranged in sequence; the ROI region video image is sequentially passed through different 3D convolution layers, and then down-sampled in the spatial dimension through an adaptive average pooling layer, and the signal in the spatiotemporal signal collection layer is sampled in the temporal and spatial dimensions by the spatiotemporal sampling layer to generate multiple respiratory signal segments; the respiratory curve is generated based on the multiple respiratory signal segments; The respiratory rate estimator is a supervised deep learning model, comprising: two one-dimensional convolutional layers, three LSTM layers, and one dense layer; the respiratory curve is used to generate a simulated signal sequence, and the simulated signal sequence is input into the two one-dimensional convolutional layers to generate a first feature sequence with 64 channels; the first feature sequence is input into the three LSTM layers to generate a second feature sequence with 32 channels; the second feature sequence is input into the dense layer for linearization calculation to generate a feature sequence with 1 channel, and the respiratory frequency is output.

2. The non-contact respiratory rate monitoring method according to claim 1, characterized in that: The acquisition of lossless video information and physiological signals of ICU patients further includes: Marking a chest region in the lossless video information and using the chest region as a ROI region; Using the KLT optical flow algorithm to perform ROI tracking on the ROI area, adjusting the resolution of the ROI area to 128×128, and generating a video image of the ROI area of the chest area; Read the true value of physiological signals from the text file; The ROI area video image of the chest area and the true value of the physiological signal are saved in an h5 file to construct a training set and a test set.

3. The non-contact respiratory rate monitoring method according to claim 1, characterized in that: The loss function of the breathing curve extractor ;in, , L pos is the positive loss term, N is the number of video image slices in the ROI area, f i is the PSD of the i-th feature from the video image of the first ROI region, f j is the PSD of the jth feature from the video image of the first ROI region, f’ i is the PSD of the i-th feature from the video image of the second ROI area, f’ j is the PSD of the jth feature from the video image of the second ROI area; , L neg Negative loss item.

4. The non-contact respiratory rate monitoring method according to claim 1, characterized in that: The loss function of the respiratory rate estimator ;in, n is the batch size during training of the respiratory rate estimator, k is the simulation signal index, is the predicted value of the simulated signal, is the true label value of the simulation signal.

5. A non-contact respiratory rate monitoring system, characterized in that: The non-contact respiratory rate monitoring system adopts the non-contact respiratory rate monitoring method according to any one of claims 1 to 4, and the non-contact respiratory rate monitoring system includes: Video information and physiological signal acquisition module, used to collect lossless video information and physiological signals of ICU patients; the physiological signals include electrocardiogram, respiratory rate signal, non-invasive blood pressure, blood oxygen concentration and pulse rate; an ROI marking module, configured to mark the chest area in the lossless video information; ROI tracking module, used to track the chest area of ICU patients; a spatiotemporal feature learning module training module, configured to train the spatiotemporal feature learning module according to the physiological signal; The trained spatiotemporal feature learning module is used to generate a respiratory curve based on the ROI area video image of the chest area, and output the respiratory rate of the ICU patient based on the respiratory curve; the spatiotemporal feature learning module includes a respiratory curve extractor and a respiratory rate estimator; the respiratory curve extractor and the respiratory rate estimator are both deep neural network models; the respiratory curve extractor is used to generate a respiratory curve based on the ROI area video image of the chest area; the respiratory rate estimator is used to output the respiratory rate based on the respiratory curve.

6. An electronic device, characterized in that: The electronic device comprises a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to perform the non-contact respiratory rate monitoring method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that It stores a computer program, which, when executed by a processor, implements the non-contact respiratory rate monitoring method according to any one of claims 1 to 4.

Images