Heart rate and respiration rate monitoring method and system based on remote photoplethysmography

Abstract

The invention provides a heart rate and respiration rate monitoring method and system based on remote photoplethysmography, and the method comprises the steps: obtaining current image data with a face image based on a model reasoning end; inputting the current image data into a heart rate and respiratory rate monitoring model, and performing monitoring reasoning by using the heart rate and respiratory rate monitoring model to obtain the identity of the person, the heart rate of the person and the respiratory rate of the person in the current image data; wherein the heart rate and respiration rate monitoring model is obtained by training a remote photoplethysmography model at a model training end by adopting historical image data, and then is imported into the model reasoning end; a face detection model and a face tracking model are integrated in the model reasoning end. A mode of combining a model training end and a model reasoning end is adopted, so that a heart rate and respiratory rate monitoring model, a face detection model and a face tracking model are effectively combined, and heart rate and respiratory rate physiological signs of identities of multiple persons can be obtained and monitored in real time.

Description

Technical Field

[0001] The present invention belongs to the technical field of non-contact physiological signal monitoring, and in particular relates to a heart rate and respiratory rate monitoring method and system based on remote photoplethysmography. Background Art

[0002] rPPG (Remote PhotoplethysmoGraphy) is a contactless physiological signal measurement technology based on optical imaging. It originated from traditional photoplethysmography (PPG). PPG uses the reflection or transmission characteristics of light to monitor the blood flow of the human body through contact sensors (such as finger clips or wrist bands), and then extracts physiological signals such as heart rate and respiratory rate. rPPG uses a camera to collect tiny color changes on the skin surface, replacing contact sensors, so that the detection of signals such as heart rate can be completed without contact. Since the rPPG technology prototype was first proposed, the technology has gradually attracted widespread attention, especially in the fields of remote health monitoring, contactless biometrics, and psychological state assessment. It has shown great application potential. With the development of artificial intelligence and image processing technology, the detection accuracy and anti-interference ability of rPPG have gradually improved. It can accurately measure physiological signals under daily lighting conditions and realize real-time analysis on devices such as smartphones and surveillance cameras.

[0003] Although rPPG technology represents a major innovation in the field of physiological signal monitoring, and provides a wide range of possibilities for non-invasive and convenient health monitoring in the future, rPPG currently mainly stays in algorithm innovation, focusing on solving various technical challenges faced in single-person scenarios, and lacks model application deployment for complete dynamic multi-person scenarios. Summary of the invention

[0004] In order to overcome the above-mentioned deficiencies of the prior art, in the first aspect, the present invention proposes a heart rate and respiratory rate monitoring method based on remote photoplethysmography, comprising: a heart rate and respiratory rate monitoring method based on remote photoplethysmography, comprising:

[0005] Based on the model inference end, current image data with a face image is obtained; the current image data is input into a heart rate and respiratory rate monitoring model, and the heart rate and respiratory rate of the person in the current image data are obtained by monitoring and reasoning using the heart rate and respiratory rate monitoring model;

[0006] Among them, the heart rate and respiratory rate monitoring model is obtained by training the remote photoplethysmography model using historical image data at the model training end, and then imported into the model inference end; the model inference end integrates a face detection model and a face tracking model.

[0007] Preferably, the step of inputting the current image data into a heart rate and respiratory rate monitoring model, performing monitoring reasoning using the heart rate and respiratory rate monitoring model, and obtaining the identity of the person in the current image data, the heart rate and respiratory rate of the person, comprises:

[0008] Inputting the current image data into a heart rate and respiratory rate monitoring model, performing monitoring reasoning using the heart rate and respiratory rate monitoring model, and acquiring a pulse wave signal of a person in the current image data;

[0009] The model inference end is used to perform frequency domain signal and time domain signal analysis on the pulse wave signal of the person to obtain an analysis result; the analysis result is subjected to fast Fourier transform and peak detection processing to obtain the identity of the person in the current image data, the heart rate and the respiratory rate of the person.

[0010] Preferably, the face detection model and the face tracking model are coupled to each other;

[0011] The model-based reasoning end obtains current image data with a face image, including:

[0012] Based on the model inference end, a current image with a face image is obtained in an online and / or offline manner;

[0013] Using the face detection model in the model inference end, extracting a face image of a person in the current image;

[0014] Based on the facial image, the facial tracking model in the model inference end is used to perform facial tracking to obtain a facial image queue of the person; and the facial image queue of the person is determined as current image data with a facial image.

[0015] Preferably, before inputting the current image data into a heart rate and respiratory rate monitoring model, and performing monitoring reasoning using the heart rate and respiratory rate monitoring model to obtain the identity of the person in the current image data, and the heart rate and respiratory rate of the person, the method further includes:

[0016] The model inference end is used to compress the model file of the heart rate and respiratory rate monitoring model imported by the model training end.

[0017] Preferably, the method of acquiring the current image with the face image in an online and / or offline manner includes:

[0018] Using a camera and a real-time topic subscription method based on the robot operating system to perform online detection and obtain a current image with a face image; and / or

[0019] A picture set and a video file with a face image are obtained offline, and the picture set and the video file are determined as current images with a face image.

[0020] Preferably, the heart rate and respiratory rate monitoring model is obtained by training a remote photoplethysmography model using historical image data at a model training end, including:

[0021] Acquire historical image data and measured historical pulse wave signals of persons in the historical image data;

[0022] Using the historical image data as input to a remote photoplethysmography model within a model training end;

[0023] Using the measured historical pulse wave signal of the person in the historical image data as the output of the remote photoplethysmography model;

[0024] Based on the input and output of the remote photoplethysmography model, the remote photoplethysmography model is trained to obtain a trained remote photoplethysmography model;

[0025] The trained remote photoplethysmography model is determined as a heart rate and respiratory rate monitoring model.

[0026] Preferably, the remote photoplethysmography model has built-in multiple deep learning models and unsupervised algorithms based on the PyTorch framework.

[0027] Preferably, the step of inputting the current image data into a heart rate and respiratory rate monitoring model, performing monitoring reasoning using the heart rate and respiratory rate monitoring model, and obtaining a pulse wave signal of a person in the current image data includes:

[0028] Processing the current image data using a pre-processing algorithm to obtain processed image data;

[0029] Inputting the processed image data into a heart rate and respiratory rate monitoring model for monitoring reasoning to obtain an initial monitoring signal of the person in the current image data;

[0030] The initial monitoring signal of the person in the current image data is processed using a post-processing algorithm to obtain the pulse wave signal of the person in the current image data.

[0031] Preferably, after the model-based reasoning end obtains the current image data with the face image, it also includes:

[0032] The current image data with the face image is standardized and normalized.

[0033] In a second aspect, the present invention application also proposes a heart rate and respiratory rate monitoring system based on remote photoplethysmography, including: a model inference end, a model training end;

[0034] The model training end is used to train the remote photoplethysmography model using historical image data to obtain a heart rate and respiratory rate monitoring model, and import the heart rate and respiratory rate monitoring model into the model inference end;

[0035] The model inference end is used to obtain current image data with a facial image; input the current image data into the heart rate and respiratory rate monitoring model, and use the heart rate and respiratory rate monitoring model to perform monitoring and inference to obtain the identity of the person in the current image data, the heart rate and respiratory rate of the person; the model inference end integrates a face detection model and a face tracking model.

[0036] Preferably, the model inference end includes: a PPG signal analysis submodule, which is used to input the current image data into a heart rate and respiratory rate monitoring model, and use the heart rate and respiratory rate monitoring model to perform monitoring inference to obtain the pulse wave signal of the person in the current image data; use the model inference end to perform frequency domain signal and time domain signal analysis on the pulse wave signal of the person to obtain an analysis result; perform fast Fourier transform and peak detection processing on the analysis result to obtain the identity of the person in the current image data, and the heart rate and respiratory rate of the person.

[0037] Preferably, the face detection model and the face tracking model are coupled to each other;

[0038] The model inference end includes: a data loading submodule, which is used to obtain a current image with a face image in an online and / or offline manner; use the face detection model in the model inference end to extract a face image of a person in the current image; based on the face image, use the face tracking model in the model inference end to track the face to obtain a face image queue of the person; and determine the face image queue of the person as the current image data with the face image.

[0039] Preferably, the model inference end further includes: a model compression submodule, which is used to compress the model file of the heart rate and respiratory rate monitoring model imported by the model training end using the model inference end.

[0040] Preferably, the data loading submodule is specifically used to: perform online detection using a camera and a real-time topic subscription method based on a robot operating system to obtain a current image with a face image; and / or obtain a picture set and a video file with a face image offline, and determine the picture set and the video file as the current image with a face image.

[0041] Preferably, the model training end is specifically used for:

[0042] Acquire historical image data and measured historical pulse wave signals of persons in the historical image data;

[0043] Using the historical image data as input to a remote photoplethysmography model within a model training end;

[0044] Using the measured historical pulse wave signal of the person in the historical image data as the output of the remote photoplethysmography model;

[0045] Based on the input and output of the remote photoplethysmography model, the remote photoplethysmography model is trained to obtain a trained remote photoplethysmography model;

[0046] The trained remote photoplethysmography model is determined as a heart rate and respiratory rate monitoring model.

[0047] Preferably, the remote photoplethysmography model has built-in multiple deep learning models and unsupervised algorithms based on the PyTorch framework.

[0048] Preferably, the PPG signal analysis submodule is specifically used for:

[0049] Processing the current image data using a pre-processing algorithm to obtain processed image data;

[0050] Inputting the processed image data into a heart rate and respiratory rate monitoring model for monitoring reasoning to obtain an initial monitoring signal of the person in the current image data;

[0051] The initial monitoring signal of the person in the current image data is processed using a post-processing algorithm to obtain the pulse wave signal of the person in the current image data.

[0052] Preferably, the data loading submodule is further used to: perform standardization and normalization processing on the current image data with the face image.

[0053] In a third aspect, the present invention application further proposes an electronic device, comprising: at least one processor and a memory; the memory and the processor are connected via a bus;

[0054] The memory is used to store one or more programs;

[0055] When the one or more programs are executed by the at least one processor, the heart rate and respiratory rate monitoring method based on remote photoplethysmography is implemented.

[0056] In a fourth aspect, the present application further proposes a readable storage medium having an execution program stored thereon, and when the execution program is executed, the heart rate and respiratory rate monitoring method based on remote photoplethysmography is implemented.

[0057] Compared with the closest prior art, the present invention has the following beneficial effects:

[0058] The heart rate and respiratory rate monitoring method and system based on remote photoplethysmography of the present invention include: based on the model inference end, obtaining the current image data with the face image; inputting the current image data into the heart rate and respiratory rate monitoring model, using the heart rate and respiratory rate monitoring model to perform monitoring inference, and obtaining the identity of the person in the current image data, the heart rate and respiratory rate of the person; wherein the heart rate and respiratory rate monitoring model is obtained by training the remote photoplethysmography model using historical image data at the model training end, and then imported into the model inference end; the model inference end integrates the face detection model and the face tracking model. By combining the model training end with the model inference end, the heart rate and respiratory rate monitoring model, the face detection model and the face tracking model are effectively combined, and the real-time monitoring of the heart rate and respiratory rate physiological signs of multiple identities can be obtained. The model inference end combines multiple technologies such as face detection, face tracking, signal extraction and analysis, realizes the combination with the model training end, and realizes the high-precision real-time monitoring of the physiological signs of the heart rate and respiratory rate of multiple people. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 A flowchart of a method for monitoring heart rate and respiratory rate based on remote photoplethysmography provided by the present invention;

[0060] Figure 2 An architecture diagram of a heart rate and respiratory rate monitoring system based on remote photoplethysmography provided by the present invention;

[0061] Figure 3 A specific architecture diagram of the heart rate and respiratory rate monitoring system based on remote photoplethysmography provided by the present invention;

[0062] Figure 4 A specific architecture diagram of the model inference end of the heart rate and respiratory rate monitoring system based on remote photoplethysmography provided by the present invention;

[0063] Figure 5 A specific architecture diagram of the model training end of the heart rate and respiratory rate monitoring system based on remote photoplethysmography provided by the present invention;

[0064] Figure 6 A schematic diagram of the operation of an electronic device provided in the present invention application. DETAILED DESCRIPTION

[0065] The specific implementation methods of the present invention are further described in detail below with reference to the accompanying drawings.

[0066] Embodiment 1:

[0067] like Figure 1 As shown, the present invention application proposes a heart rate and respiratory rate monitoring method based on remote photoplethysmography, which may include:

[0068] Step 1: Based on the model inference end, current image data with a face image is obtained; the model inference end integrates a face detection model and a face tracking model;

[0069] Step 2: Input the current image data into the heart rate and respiratory rate monitoring model, and use the heart rate and respiratory rate monitoring model to perform monitoring and reasoning to obtain the identity of the person in the current image data, the heart rate and respiratory rate of the person; the heart rate and respiratory rate monitoring model is obtained by training the remote photoplethysmography model using historical image data at the model training end, and then imported into the model reasoning end.

[0070] As mentioned above, both the model training end and the model inference end can rely on development environments such as Python, Numpy, OpenCV, PyTorch, and Matplotlib.

[0071] In step 1, the face detection model and the face tracking model are coupled to each other;

[0072] The acquiring of current image data with a face image based on the model inference end may include:

[0073] Step 1.1: Based on the model inference end, obtain the current image with the face image in an online and / or offline manner;

[0074] Step 1.2: using the face detection model in the model inference end to extract a face image of a person from the current image;

[0075] Step 1.3: Based on the facial image, the facial tracking model in the model inference end is used to perform facial tracking to obtain a facial image queue of the person; and the facial image queue of the person is determined as current image data with a facial image.

[0076] In the above steps 1.2-1.3, the face detection model is used to detect the face image of the current frame; the face tracking model is used to compare the face image of the current frame with the face image of the historical frame to complete the identity recognition; the face detection model is based on the Google face detection (BlazeFace) algorithm, and the face tracking model is based on the DeepSort (target tracking) algorithm, but it has been greatly modified and deeply coupled to support automatic tracking and recognition of multiple faces. In addition, the face tracking module has made a significant improvement to the DeepSort algorithm, so that it can maintain the historical image queue of each ID (Identity document) face during the tracking process.

[0077] In the above, after obtaining the current image data with the face image at the model-based reasoning end, the following may also be included:

[0078] The current image data with the face image is standardized and normalized.

[0079] As mentioned above, for the current image data with face images, in addition to the original input, three other common processing methods are supported, specifically including differential normalization, standardized input, and normalization. Similar to image sequence processing, the model inference end can also support two processing methods for the label data of the PPG signal output by the remote photoplethysmography model in addition to the original input: differential normalization and standardized input.

[0080] In the above step 1.1, the current image with the face image is obtained in an online and / or offline manner, including:

[0081] Step 1.1.1, using a camera and a real-time topic subscription method based on the robot operating system to perform online detection to obtain a current image with a face image; and / or

[0082] Step 1.1.2, offline acquiring a picture set and a video file with a face image, and determining the picture set and the video file as a current image with a face image.

[0083] In the above steps 1.1.1-1.1.2, the model inference end supports multiple monitoring methods, which can be divided into two input methods: online and offline. The online method can include real-time cameras, online video streams, ros (robot) topics, etc. The offline method can include local videos, picture collections, and rosbag files, etc.

[0084] In the above step 1, based on the model inference end, when obtaining the current image data with the face image, multiple data loading methods are supported. The loading method is specified by the relevant command line parameters, and the current image is obtained at every moment;

[0085] The above-mentioned face detection model is implemented based on the open source face detection algorithm and is used to detect all faces in the current image frame. The original BlazeFace implementation is based on the symbolic mathematical system (TensorFlow) framework for data flow programming, and is reproduced based on PyTorch (an open source deep learning framework for machine learning and deep learning). The face detection model mainly includes an image pre-processing step and a detection result post-processing step. The image pre-processing step is responsible for appropriately cropping and channel inverting the current image, and the detection result post-processing step is responsible for filtering the detection results, such as executing a non-maximum suppression algorithm to obtain a filtered ROI (region of interest) face frame, and each face avatar is captured from the current image based on the filtered face frame;

[0086] The above face tracking model is based on the multi-target tracking algorithm, but a lot of improvements have been made to its original implementation, so that it can complete face tracking and maintain the historical image queue of each ID's face;

[0087] In the above step 2, the remote photoplethysmography model has built-in multiple deep learning models and unsupervised algorithms based on the PyTorch framework.

[0088] As mentioned above, the remote photoplethysmography model can have 8 classic rPPG algorithms built in, all of which are implemented based on the PyTorch framework, including 4 unsupervised algorithms and 4 supervised algorithms. The unsupervised algorithm does not require training and mainly provides a benchmark for the rPPG algorithm. The supervised rPPG algorithm takes a data-processed facial image sequence as input and outputs a predicted PPG signal sequence. The rPPG unsupervised algorithm can also be built in. The unsupervised algorithm is mainly used to provide a training benchmark for the rPPG deep learning algorithm because it is easily affected by lighting conditions and environmental noise.

[0089] In the above step 2, the heart rate and respiratory rate monitoring model is obtained by training the remote photoplethysmography model using historical image data at the model training end, which may include:

[0090] Step a, obtaining historical image data and the measured historical pulse wave signal of the person in the historical image data;

[0091] Step b, using the historical image data as input to a remote photoplethysmography model within a model training end;

[0092] Step c, using the measured historical pulse wave signal of the person in the historical image data as the output of the remote photoplethysmography model;

[0093] Step d, training the remote photoplethysmography model based on the input and output of the remote photoplethysmography model to obtain a trained remote photoplethysmography model;

[0094] Step e: determining the trained remote photoplethysmography model as a heart rate and respiratory rate monitoring model.

[0095] In the above steps ae, the model training end can collect face image sequences, PPG signals, and heart rate and respiratory rate measurements. The heart rate and respiratory rate measurements are obtained from the measured historical pulse wave signals, and the face image sequences and PPG signals are used as training inputs; the heart rate and respiratory rate measurements are used to verify the training results.

[0096] As mentioned above, during the training process, the model training end collects training data and verification data of the rPPG algorithm in the remote photoplethysmography model; the training data includes historical image data of face images and PPG signals, and the verification data includes heart rate and respiratory rate values ​​collected synchronously with the PPG signals. The model training end can also collect image data and store it locally, and supports custom rPPG data collection. The collected verification data can also be used as input to output the training results as graphical analysis data, so as to realize the intuitive display of rPPG model training results.

[0097] As mentioned above, the model training end can preprocess the facial image sequence stored in .png format and the PPG signal stored in .txt format, and the preprocessed image data is locally saved as the .npy format image data generated in the Numpy tool. During training, the model corresponding to the algorithm can be loaded according to the specified computing resources, and the image data in .npy format after preprocessing can be dynamically loaded, and the input and label data saved can be used for model training. The model will dynamically generate intermediate logs (.txt) and the current optimal model (.pth or .pt) during the training process. During the training process, distributed training and incremental model training are supported, which greatly reduces the repeated consumption of computing resources. The training results can provide dual display of model PPG signal fitting and heart rate and respiratory rate prediction accuracy based on the Matplotlib visualization toolkit. The training log (.txt) and the collected real heart rate and respiratory rate measurements are analyzed to generate graphical analysis data. Graphical analysis data include: bar charts, pie charts, and linear charts to achieve intuitive display of training results.

[0098] In the above step 2, before inputting the current image data into the heart rate and respiratory rate monitoring model, using the heart rate and respiratory rate monitoring model to perform monitoring reasoning to obtain the identity of the person in the current image data, the heart rate and respiratory rate of the person, the following may also be included:

[0099] The model inference end is used to compress the model file of the heart rate and respiratory rate monitoring model imported by the model training end.

[0100] As mentioned above, the model training end can convert between PyTorch files and ONNX model file formats, support exporting deep learning rPPG models in ONNX format, and converting .pth or .pt model files under the PyTorch framework to .onnx model files.

[0101] The model file of the heart rate and respiratory rate monitoring model that can be exported by the model training end is a model file in the ONNX format. When exporting, it can support 4 supervised algorithms to export in a unified ONNX format. The model inference end compresses the model file, which can delete the redundancy in the rPPG model, reduce the demand for computing resources, and further reduce the storage and computing requirements for hardware to achieve the purpose of accelerating model inference or training; when the model inference end compresses the ONNX model file provided by the training end, it performs operations such as data quantization, model sparsification, knowledge distillation, lightweight network design, and tensor decomposition according to the selection strategy. Afterwards, the model inference end extracts physiological signals from the output of the compressed heart rate and respiratory rate monitoring model. The heart rate detection algorithm and the respiratory rate detection algorithm are executed for each PPG signal to predict the heart rate and respiratory rate values ​​at the current time and the current identity ID.

[0102] The above-mentioned inputting the current image data into a heart rate and respiratory rate monitoring model, using the heart rate and respiratory rate monitoring model to perform monitoring reasoning to obtain the identity of the person in the current image data, the heart rate and respiratory rate of the person, may include:

[0103] Step 2.1: inputting the current image data into a heart rate and respiratory rate monitoring model, performing monitoring reasoning using the heart rate and respiratory rate monitoring model, and obtaining a pulse wave signal of the person in the current image data;

[0104] Step 2.2: Use the model inference end to perform frequency domain signal and time domain signal analysis on the pulse wave signal of the person to obtain an analysis result; perform fast Fourier transform and peak detection processing on the analysis result to obtain the identity of the person in the current image data, the heart rate and respiratory rate of the person.

[0105] In the above steps 2.1-2.2, the rPPG model algorithm extracts and analyzes the PPG signal in the face image, and extracts the heart rate and respiratory rate prediction values ​​from the PPG signal. The model inference end extracts the PPG signal from the current image data based on the rPPG model algorithm, which is the pulse wave signal of the person, and then analyzes the PPG signal from two feature dimensions, frequency domain and time domain. Through technical means such as fast Fourier transform and peak detection, real-time heart rate and respiratory rate analysis and detection of long-distance and multi-target are achieved.

[0106] The above-mentioned model inference end supports multiple monitoring methods and combines multiple advanced technologies such as face detection, face tracking, model compression, PPG signal extraction and analysis to achieve high-precision real-time monitoring of multi-person heart rate and respiratory rate physiological signs on devices with limited computing power.

[0107] As mentioned above, after obtaining the identity, heart rate and respiration rate of the person in the current image data, the face (ROI) detection result box, identity information (ID) tracking mark and its corresponding physiological feature prediction output (PPG signal, heart rate value and respiration rate prediction value) of the current image frame can be displayed in real time based on the OpenCV system.

[0108] In the above step 2.2, inputting the current image data into a heart rate and respiratory rate monitoring model, performing monitoring reasoning using the heart rate and respiratory rate monitoring model, and obtaining a pulse wave signal of a person in the current image data may include:

[0109] Step 2.2.1: Processing the current image data using a pre-processing algorithm to obtain processed image data;

[0110] Step 2.2.2: inputting the processed image data into a heart rate and respiratory rate monitoring model for monitoring reasoning, and obtaining an initial monitoring signal of the person in the current image data;

[0111] Step 2.2.3: The initial monitoring signal of the person in the current image data is processed using a post-processing algorithm to obtain the pulse wave signal of the person in the current image data.

[0112] In the above step 2.2.3, after obtaining the pulse wave signal of the person in the current image data, due to the use of pre-processing algorithms and post-processing algorithms, the ONNX format export file covers pre-processing and post-processing, which simplifies the difficulty of deployment on the model inference end.

[0113] In addition, for the technology in this field, it is difficult to provide a unified training interface for the rPPG algorithm library due to the different forms of modules such as data collection, data processing, model construction, model training and post-processing of various rPPG algorithms. Therefore, how to eliminate the gap between rPPG model training and model deployment is a technical difficulty that needs to be solved urgently. The method of the present invention consists of two parts: a model training end and a model reasoning end. The rPPG model training and deployment processes are integrated to achieve seamless connection from model development to actual application deployment. A unified training interface can be provided for the rPPG algorithm, eliminating the gap between rPPG model training and model deployment.

[0114] In summary, the method of the present invention includes two modules: a model training end and a model reasoning end. The model training end focuses on the development of rPPG deep learning algorithms, supports a variety of rPPG algorithms, and covers a complete training process from data collection, processing, training, result verification, and model export, which can significantly improve the development and training efficiency of rPPG algorithm models. The model reasoning end is responsible for the integrated deployment of the rPPG algorithm model, supports a variety of data loading methods, and integrates model compression, face detection, face tracking, PPG signal extraction and analysis, and visualization display methods. It can achieve real-time and high-precision monitoring of the heart rate and respiratory rate of multiple people in a resource-constrained environment. This method integrates the training and deployment process of the rPPG model to achieve a smooth transition from model development to actual application deployment, reducing manual operations during model training, conversion, optimization, and deployment, thereby shortening the development cycle, reducing the consumption of manpower and hardware resources, and accelerating the iterative update of the model.

[0115] Embodiment 2:

[0116] like Figure 2 , 3 As shown, the present invention also provides a heart rate and respiratory rate monitoring system based on remote photoplethysmography, including: a heart rate and respiratory rate monitoring system based on remote photoplethysmography, including: a model inference end, a model training end;

[0117] The model training end is used to train the remote photoplethysmography model using historical image data to obtain a heart rate and respiratory rate monitoring model, and import the heart rate and respiratory rate monitoring model into the model inference end;

[0118] The model inference end is used to obtain current image data with a facial image; input the current image data into the heart rate and respiratory rate monitoring model, and use the heart rate and respiratory rate monitoring model to perform monitoring and inference to obtain the identity of the person in the current image data, the heart rate and respiratory rate of the person; the model inference end integrates a face detection model and a face tracking model.

[0119] like Figure 4 As shown, the above-mentioned model inference end may include a data loading submodule, a face detection model, a face tracking model, a model compression submodule, a PPG signal analysis submodule, and a visualization display submodule, etc. The submodules are tightly coupled and are responsible for the integrated deployment of the rPPG model;

[0120] Among them, the data loading submodule supports multiple data loading methods, which are specified by relevant command line parameters and output the current image at every moment;

[0121] Among them, the face detection model is implemented based on the face detection open source algorithm, which is used to detect all faces in the current image frame. The original face detection open source algorithm implements a symbolic mathematical system framework based on data flow programming, and is reproduced based on the PyTorch framework. The face detection model mainly includes image pre-processing steps and detection result post-processing steps. The pre-processing step is responsible for appropriate cropping and channel inversion of the current image, and the post-processing step is responsible for filtering the detection results, such as executing the non-maximum suppression algorithm to obtain the filtered ROI face frame, and each face portrait is captured from the current image based on the filtered ROI face frame;

[0122] The face tracking model is based on a multi-target tracking algorithm, but a lot of improvements have been made to its original implementation, so that it can complete face tracking and maintain the image queue of each ID's face.

[0123] Furthermore, the PPG signal analysis submodule of the model inference end is used to input the current image data into a heart rate and respiratory rate monitoring model, perform monitoring inference using the heart rate and respiratory rate monitoring model, and obtain the pulse wave signal of the person in the current image data; perform frequency domain signal and time domain signal analysis on the pulse wave signal of the person using the model inference end to obtain an analysis result; perform fast Fourier transform and peak detection processing on the analysis result to obtain the identity of the person in the current image data, and the heart rate and respiratory rate of the person.

[0124] As mentioned above, the PPG signal analysis submodule can be responsible for extracting physiological signals from the compressed rPPG model output, executing the heart rate detection algorithm and the respiratory rate detection algorithm on each predicted output PPG signal, and predicting the heart rate and respiratory rate values ​​of the current time and the current person.

[0125] As mentioned above, the PPG signal analysis submodule at the model inference end first extracts the PPG signal from the face image based on the rPPG model algorithm, and then analyzes the PPG signal from two feature dimensions, frequency domain and time domain. Through technical means such as fast Fourier transform and peak detection, it realizes long-distance, multi-target real-time heart rate and respiratory rate analysis and detection.

[0126] Furthermore, the face detection model and the face tracking model are coupled to each other;

[0127] The model inference end includes: a data loading submodule, which is used to obtain a current image with a face image in an online and / or offline manner; use the face detection model in the model inference end to extract a face image of a person in the current image; based on the face image, use the face tracking model in the model inference end to track the face to obtain a face image queue of the person; and determine the face image queue of the person as the current image data with the face image.

[0128] The above-mentioned face detection model can be based on the Google BlazeFace model; the face tracking model can be based on the DeepSort target tracking open source algorithm, but it has been significantly modified so that it can maintain the historical image queue of each ID face during the tracking process; and the deep coupling of the two is achieved to support automatic tracking and recognition of multiple faces.

[0129] Furthermore, the model inference end also includes: a model compression submodule, which is used to use the model inference end to compress the model file of the heart rate and respiratory rate monitoring model imported by the model training end.

[0130] As mentioned above, the model compression submodule can be responsible for compressing the ONNX model files provided by the model training end, and performing operations such as numerical quantization, model sparsification, knowledge distillation, lightweight network design, and tensor decomposition according to the selection strategy. The ONNX model files provided by the model training end can be compressed to reduce the demand for computing resources and accelerate the model inference speed.

[0131] Furthermore, the data loading submodule is specifically used to: perform online detection using a camera and a real-time topic subscription method based on the robot operating system to obtain a current image with a face image; and / or obtain a picture set and a video file with a face image offline, and determine the picture set and the video file as the current image with a face image.

[0132] The data loading submodule supports two modes: online and offline. The offline input mode includes picture sets and video files; the online mode includes raw image data collected by online cameras and ROS real-time topic image data.

[0133] The above-mentioned visualization display submodule can display the face detection results, identity information and corresponding physiological feature prediction output (PPG signal, heart rate value and respiratory rate prediction value) of the current image frame in real time based on OpenCV.

[0134] The above-mentioned model inference end combines a variety of advanced technologies to achieve real-time monitoring of multiple people's heart rate and respiratory rate physiological signs. The face detection model is used to detect the face image of the current frame; the face tracking model is used to compare the face image of the current frame with the face image of the historical frame to complete identity recognition; the model compression submodule is used to delete the redundancy in the rPPG model, thereby reducing the storage and computing requirements for hardware to achieve the purpose of accelerating model inference or training; the PPG signal analysis submodule is used to extract the PPG signal from the face image, and extract the heart rate and respiratory rate prediction values ​​from the PPG signal.

[0135] Furthermore, the model training end is specifically used for:

[0136] Acquire historical image data and measured historical pulse wave signals of persons in the historical image data;

[0137] Using the historical image data as input to a remote photoplethysmography model within a model training end;

[0138] Using the measured historical pulse wave signal of the person in the historical image data as the output of the remote photoplethysmography model;

[0139] Based on the input and output of the remote photoplethysmography model, the remote photoplethysmography model is trained to obtain a trained remote photoplethysmography model;

[0140] The trained remote photoplethysmography model is determined as a heart rate and respiratory rate monitoring model.

[0141] like Figure 5 As shown above, the model training end can include 6 sub-modules: a data acquisition sub-module, a data processing sub-module, an algorithm library sub-module, a distributed training sub-module, a training result verification sub-module and a model export sub-module; the 6 sub-modules cover the entire process of rPPG algorithm training.

[0142] Among them, the data acquisition submodule of the model training end is used for the collection of rPPG algorithm training data and verification data. The training data includes face image sequences and PPG signals, and the verification data includes heart rate and respiratory rate values ​​collected synchronously with the PPG signals. The data acquisition submodule provides an interface to complete the collection and local storage of the above data by specifying corresponding parameters. The data acquisition submodule supports customized rPPG data collection. The collected data is divided into two types: training data and verification data. The image sequence data and PPG signals are training data, and the historical real measurement values ​​of heart rate and respiratory rate are verification data.

[0143] Among them, the data processing submodule of the model training end supports 4 preprocessing methods for image sequence data (original input, differential normalization, standardized input and normalization), and supports 3 preprocessing methods for PPG signal label data (original input, differential normalization and standardized input). The data processing submodule preprocesses the face image sequence stored in .png format and the PPG signal stored in .txt format, and the preprocessed data is locally saved in the .npy format generated in the Numpy tool. For the face image sequence, similar to the image sequence processing, the PPG signal label data supports 2 processing methods in addition to the original input: differential normalization and standardized input.

[0144] Among them, the algorithm library submodule of the model training end implements 8 classic rPPG models, all of which are implemented based on the PyTorch framework; among them, there are 4 unsupervised models and 4 supervised models. The unsupervised model does not need to be trained and mainly provides a benchmark for the rPPG system. The supervised rPPG model takes the facial image sequence processed by the data processing submodule as input and outputs a predicted PPG signal sequence. The rPPG unsupervised algorithm is mainly used to provide a training benchmark for the rPPG deep learning model because it is easily affected by lighting conditions and environmental noise.

[0145] Among them, the distributed training submodule of the model training end is the core of the entire model training end, and is also a bridge for the data processing submodule, algorithm library submodule, training result verification submodule and model export submodule to connect with each other. The distributed training submodule loads the corresponding .py format model in the algorithm library submodule according to the specified computing resources, dynamically loads the pre-processed .npy format data in the data processing submodule, and saves the input and label data to perform remote photoplethysmography model training. The remote photoplethysmography model will dynamically generate intermediate logs (.txt) and the current optimal model (.pth or .pt) during the training process. The distributed training submodule supports gpu distributed training and model incremental training, which greatly reduces the repeated consumption of computing resources.

[0146] Among them, the training result verification submodule of the model training end takes the collected verification data as input, outputs the graphical analysis data of the training results, and intuitively displays the rPPG model training results. The training result verification submodule analyzes the training log (.txt) and the collected real heart rate and respiratory rate measurement values ​​to generate graphical analysis data; the graphical analysis data includes: bar charts, pie charts, and linear charts, and then intuitively displays the training results. The training result verification submodule can provide a dual display of model PPG signal fitting and heart rate and respiratory rate prediction accuracy based on the Matplotlib visualization toolkit.

[0147] Among them, the model export submodule of the model training end is responsible for the conversion between PyTorch model files and ONNX model file formats, supports the export of deep learning rPPG models in ONNX format, and converts .pth or .pt models under the PyTorch framework into .onnx files. The model export submodule supports the export of 4 supervised rPPG algorithms in a unified ONNX format; and at the same time, the model data pre-processing and post-processing operators are embedded in the model training end.

[0148] Furthermore, the remote photoplethysmography model has built-in multiple deep learning models and unsupervised algorithms based on the PyTorch framework.

[0149] Furthermore, the PPG signal analysis submodule is specifically used for:

[0150] Processing the current image data using a pre-processing algorithm to obtain processed image data;

[0151] Inputting the processed image data into a heart rate and respiratory rate monitoring model for monitoring reasoning to obtain an initial monitoring signal of the person in the current image data;

[0152] The initial monitoring signal of the person in the current image data is processed using a post-processing algorithm to obtain the pulse wave signal of the person in the current image data.

[0153] Furthermore, the data loading submodule is also used to: perform standardization and normalization processing on the current image data with the face image.

[0154] Embodiment 3:

[0155] like Figure 6 As shown, the present invention also provides an electronic device, which may be a computer device, a single-chip device, an intelligent mobile device, etc. The electronic device in this embodiment may include a processor, a memory, a transceiver component, etc. The memory, the processor, and the transceiver component are connected via a bus; the memory may be used to store an execution program, and an exemplary execution program may include instructions; the processor is used to execute the instructions stored in the memory. The memory may also be used to store data, which may be called and / or modified when the instructions are executed.

[0156] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions in a storage medium to implement the corresponding method flow or corresponding functions, so as to implement the steps of the heart rate and respiratory rate monitoring method based on remote photoplethysmography in the above-mentioned embodiment.

[0157] Embodiment 4:

[0158] Based on the same inventive concept, the present invention also provides a readable storage medium, specifically an electronic device readable storage medium (Memory), which is a memory device in an electronic device for storing programs and data. It is understandable that the storage medium here can include both a built-in storage medium in an electronic device and an extended storage medium supported by the electronic device. The storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by a processor are also stored in the storage space, and these instructions can be one or more execution programs (including program codes). It should be noted that the storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory. The processor loads and executes one or more instructions stored in the storage medium, which can implement the steps of the heart rate and respiratory rate monitoring method based on remote photoplethysmography in the above embodiment.

[0159] Those skilled in the art will appreciate that the embodiments of the present invention application can be provided as methods, systems, or computer program products. Therefore, the present invention application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present invention application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0160] The present invention application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present invention application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0161] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0162] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0163] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention application rather than to limit its protection scope. Although the present invention application is described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that after reading the present invention application, those skilled in the art can still make various changes, modifications or equivalent substitutions to the specific implementation methods of the application, but these changes, modifications or equivalent substitutions are all within the protection scope of the claims to be approved.

Claims

1. A heart rate and respiratory rate monitoring method based on remote photoplethysmography, characterized in that: include: Based on the model inference end, obtain the current image data with the face image; Inputting the current image data into a heart rate and respiratory rate monitoring model, and using the heart rate and respiratory rate monitoring model to perform monitoring reasoning to obtain the identity of the person in the current image data, the heart rate and respiratory rate of the person; Among them, the heart rate and respiratory rate monitoring model is obtained by training the remote photoplethysmography model using historical image data at the model training end, and then imported into the model inference end; the model inference end integrates a face detection model and a face tracking model.

2. The method according to claim 1, characterized in that The step of inputting the current image data into a heart rate and respiratory rate monitoring model, and using the heart rate and respiratory rate monitoring model to perform monitoring reasoning to obtain the identity of the person in the current image data, the heart rate and respiratory rate of the person, includes: Inputting the current image data into a heart rate and respiratory rate monitoring model, performing monitoring reasoning using the heart rate and respiratory rate monitoring model, and acquiring a pulse wave signal of a person in the current image data; The model inference end is used to perform frequency domain signal and time domain signal analysis on the pulse wave signal of the person to obtain an analysis result; the analysis result is subjected to fast Fourier transform and peak detection processing to obtain the identity of the person in the current image data, the heart rate and the respiratory rate of the person.

3. The method according to claim 1 or 2, characterized in that: The face detection model and the face tracking model are coupled to each other; The model-based reasoning end obtains current image data with a face image, including: Based on the model inference end, a current image with a face image is obtained in an online and / or offline manner; Using the face detection model in the model inference end, extracting a face image of a person in the current image; Based on the facial image, the facial tracking model in the model inference end is used to perform facial tracking to obtain a facial image queue of the person; and the facial image queue of the person is determined as current image data with a facial image.

4. The method according to claim 3, characterized in that Before inputting the current image data into the heart rate and respiratory rate monitoring model, and performing monitoring reasoning using the heart rate and respiratory rate monitoring model to obtain the identity of the person in the current image data, and the heart rate and respiratory rate of the person, the method further includes: The model inference end is used to compress the model file of the heart rate and respiratory rate monitoring model imported by the model training end.

5. The method according to claim 3, characterized in that: The method of obtaining the current image with the face image in an online and / or offline manner includes: Using a camera and a real-time topic subscription method based on the robot operating system to perform online detection and obtain a current image with a face image; and / or A picture set and a video file with a face image are obtained offline, and the picture set and the video file are determined as current images with a face image.

6. The method according to claim 1, characterized in that The heart rate and respiratory rate monitoring model is obtained by training the remote photoplethysmography model using historical image data at the model training end, including: Acquire historical image data and measured historical pulse wave signals of persons in the historical image data; Using the historical image data as input to a remote photoplethysmography model within a model training end; Using the measured historical pulse wave signal of the person in the historical image data as the output of the remote photoplethysmography model; Based on the input and output of the remote photoplethysmography model, the remote photoplethysmography model is trained to obtain a trained remote photoplethysmography model; The trained remote photoplethysmography model is determined as a heart rate and respiratory rate monitoring model.

7. The method according to claim 6, characterized in that The remote photoplethysmography model has built-in multiple deep learning models and unsupervised algorithms based on the PyTorch framework.

8. The method according to claim 4, characterized in that The step of inputting the current image data into a heart rate and respiratory rate monitoring model, performing monitoring reasoning using the heart rate and respiratory rate monitoring model, and obtaining a pulse wave signal of a person in the current image data includes: Processing the current image data using a pre-processing algorithm to obtain processed image data; Inputting the processed image data into a heart rate and respiratory rate monitoring model for monitoring reasoning to obtain an initial monitoring signal of the person in the current image data; The initial monitoring signal of the person in the current image data is processed using a post-processing algorithm to obtain the pulse wave signal of the person in the current image data.

9. The method according to claim 5, characterized in that After obtaining the current image data with the face image at the model-based reasoning end, the method further includes: The current image data with the face image is standardized and normalized.

10. A heart rate and respiratory rate monitoring system based on remote photoplethysmography, characterized in that: include: Model inference end and model training end; The model training end is used to train the remote photoplethysmography model using historical image data to obtain a heart rate and respiratory rate monitoring model, and import the heart rate and respiratory rate monitoring model into the model inference end; The model inference end is used to obtain current image data with a facial image; input the current image data into the heart rate and respiratory rate monitoring model, and use the heart rate and respiratory rate monitoring model to perform monitoring and inference to obtain the identity of the person in the current image data, the heart rate and respiratory rate of the person; the model inference end integrates a face detection model and a face tracking model.

Images