Non-contact respiration monitoring method, system and equipment and storage medium

Abstract

The embodiment of the invention provides a non-contact respiration monitoring method, system and device and a storage medium, and relates to the technical field of image recognition. The method comprises the steps that two-dimensional images of a breathing monitoring target object at multiple visual angles are correspondingly obtained through multiple cameras, two-dimensional skeleton key points of the target object at each visual angle are obtained according to the two-dimensional images, and three-dimensional skeleton key points of the target object are obtained through triangulation according to the two-dimensional skeleton key points; representative skeleton key points associated with respiration of the target object in the three-dimensional skeleton key points are extracted, and displacement wave crests and displacement wave troughs of the representative skeleton key points are obtained; and judging a breathing stage of the target object based on the displacement wave crest and the displacement wave trough so as to perform breathing monitoring on the target object. Compared with the mode of measuring the pixel or optical flow change on the image, the breathing monitoring based on the displacement of the key points of the human skeleton has the advantages that higher precision is realized, and more cooperation does not need to be provided for the target.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and in particular, to a non-contact respiratory monitoring method, system, device and storage medium. Background Art

[0002] Respiratory monitoring is an important part of monitoring patients' vital signs. In existing technologies, it is often implemented through contact sensors, wearable devices, or active signal transmission (such as radar, audio, WiFi, etc.).

[0003] In the existing technology, in the form of contact sensors or wearable devices, the monitored object needs to cooperate in behavior and wearing. For example, an instrument and a catheter are connected to the mouth and nose of a person to measure the airflow and then monitor the exhalation condition; or the monitored object is asked to wear a specific device with a sensor installed on it to calculate the exhalation condition by collecting relevant physiological indicators. In the form of actively sending signals, radar is used to send and receive signals, measure the displacement changes of a person during breathing, and combine data analysis to obtain the exhalation condition.

[0004] The above methods are not convenient enough or have certain limitations in application scenarios. They require active cooperation from users, cannot monitor the breathing conditions of multiple people at the same time, and have poor real-time performance. Summary of the invention

[0005] In order to solve one of the above-mentioned technical defects, a non-contact respiratory monitoring method, device and storage medium are provided in an embodiment of the present application.

[0006] According to a first aspect of an embodiment of the present application, a non-contact respiratory monitoring method is provided, the method comprising: Acquire two-dimensional images of a respiratory monitoring target object at multiple perspectives through multiple cameras, acquire two-dimensional skeleton key points of the target object at each perspective based on the two-dimensional images, and acquire three-dimensional skeleton key points of the target object through triangulation based on the two-dimensional skeleton key points; Extract representative bone key points associated with the breathing of the target object from the three-dimensional bone key points, and obtain displacement peaks and displacement troughs of the representative bone key points; The breathing phase of the target object is determined based on the displacement peak and the displacement trough to perform breathing monitoring on the target object.

[0007] In an optional embodiment of the present application, the steps of acquiring two-dimensional images of multiple perspectives of a respiratory monitoring target object through multiple cameras, acquiring two-dimensional skeleton key points of the target object at each perspective according to the two-dimensional images, and acquiring three-dimensional skeleton key points of the target object by triangulation according to the two-dimensional skeleton key points also include: The cameras are synchronous cameras, which are synchronized by pulse signals so that each frame of two-dimensional image is consistent in time.

[0008] In an optional embodiment of the present application, the step of extracting representative skeletal key points associated with the breathing of the target object from the three-dimensional skeletal key points and obtaining displacement peaks and displacement troughs of the representative skeletal key points further includes: The key points representing the skeleton are low-pass filtered and smoothed using a Butterworth filter to remove high-frequency noise.

[0009] In an optional embodiment of the present application, the step of extracting representative skeletal key points associated with the breathing of the target object from the three-dimensional skeletal key points and obtaining displacement peaks and displacement troughs of the representative skeletal key points further includes: The key points representing the skeleton are smoothed by a Kalman filter to filter out instantaneous fluctuations.

[0010] In an optional embodiment of the present application, the step of extracting representative skeletal key points associated with the breathing of the target object from the three-dimensional skeletal key points and obtaining displacement peaks and displacement troughs of the representative skeletal key points further includes: When obtaining the displacement peaks and displacement troughs, the preset minimum peak-to-trough spacing is met.

[0011] In an optional embodiment of the present application, the step of determining the breathing stage of the target object based on the displacement peak and the displacement trough to perform breathing monitoring on the target object further includes: The time interval between the displacement peak and the displacement trough is calculated to obtain the breathing frequency of the target subject.

[0012] In an optional embodiment of the present application, the step of determining the breathing stage of the target object based on the displacement peak and the displacement trough to perform breathing monitoring on the target object further includes: A sliding window is set for the displacement process representing the key points of the skeleton, the wave value type that appears most times in the sliding window is used as the result wave value type of the sliding window, and the breathing stage of the sliding window is determined according to the result wave value type.

[0013] According to a second aspect of an embodiment of the present application, a non-contact respiratory monitoring system is provided, the system comprising a three-dimensional reconstruction module, a signal acquisition module and a respiratory monitoring module; wherein: A three-dimensional reconstruction module is used to obtain two-dimensional images of multiple perspectives of a respiratory monitoring target object through multiple cameras, obtain two-dimensional skeleton key points of the target object at each perspective based on the two-dimensional images, and obtain three-dimensional skeleton key points of the target object through triangulation based on the two-dimensional skeleton key points; A signal acquisition module, used to extract representative bone key points associated with the breathing of the target object from the three-dimensional bone key points, and to obtain displacement peaks and displacement troughs of the representative bone key points; The breathing monitoring module is used to determine the breathing stage of the target object based on the displacement peak and displacement trough to perform breathing monitoring on the target object.

[0014] According to a third aspect of an embodiment of the present application, there is provided a computer device, including: a memory; A processor; and a computer program; wherein the computer program is stored in a memory and is configured to be executed by the processor to implement a non-contact respiratory monitoring method as described in any one of the first aspects of the embodiments of the present application.

[0015] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored; the computer program is executed by a processor to implement a non-contact respiratory monitoring method as described in any one of the first aspects of the embodiments of the present application.

[0016] The non-contact respiratory monitoring method provided in the embodiment of the present application has the following beneficial effects: This application uses multiple cameras to capture the 2D skeletal key points of the human body, and uses the multi-camera calibration information to calculate the 3D skeletal coordinates of the human body in space. Through real-time data processing, the current breathing stage of the person is calculated and output, thereby achieving real-time respiratory monitoring. Respiration monitoring based on the displacement of key skeletal points of the human body achieves higher accuracy than the method of measuring pixel or optical flow changes on the image. In addition, this application is based on image processing and does not require users to wear equipment, avoiding the various inconveniences of traditional methods, and does not rely on active signal transmission (such as radar, audio, WiFi), and is suitable for a wider range of application scenarios with higher accuracy monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 A flowchart of a non-contact respiratory monitoring method provided in an embodiment of the present application; Figure 2 A structural diagram of a non-contact respiratory monitoring system provided in an embodiment of the present application; Figure 3 A schematic diagram of the computer device structure provided for one embodiment of the present application. DETAILED DESCRIPTION

[0018] In order to make the technical solutions and advantages in the embodiments of the present application more clearly understood, the exemplary embodiments of the present application are further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than an exhaustive list of all the embodiments. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0019] This application proposes a non-contact respiratory monitoring method. Figure 1 For a flowchart of the non-contact respiratory monitoring method provided in the embodiment of the present application, please refer to Figure 1 : S1: Acquire two-dimensional images of multiple perspectives of a respiratory monitoring target object through multiple cameras, acquire two-dimensional skeleton key points of the target object at each perspective based on the two-dimensional images, and acquire three-dimensional skeleton key points of the target object through triangulation based on the two-dimensional skeleton key points.

[0020] In the specific implementation, multiple cameras are used to capture the images of the target object for respiratory monitoring, and the 2D skeleton key points of the human body are extracted from each frame of each camera; in this embodiment, by detecting the 2D skeleton key points of the human body from the video stream, basic data can be provided for subsequent 3D reconstruction and respiratory monitoring; specifically, the 3D skeleton coordinates of the human body in space are reconstructed by triangulation method using the calibration information and 2D key points of multiple cameras. Based on this, the foundation is laid for the subsequent monitoring of respiratory status by means of displacement of key point coordinates.

[0021] In some embodiments of the present application, the cameras are synchronous cameras, and the synchronous cameras are synchronized by pulse signals so that each frame of the two-dimensional image is consistent in time.

[0022] In specific implementations, cameras are synchronized through pulse signals to ensure the temporal consistency of each frame of the image, which can further improve the accuracy of respiratory monitoring.

[0023] In an embodiment of the present application, deep learning image processing technology is used to extract 2D skeleton key points of the human body from each frame image of each camera; specifically, key point detection is performed using a deep learning algorithm, which can efficiently and accurately detect each joint point of the human body and is suitable for a variety of postures and motion states.

[0024] In the specific implementation, data preprocessing is first performed to obtain video streams from multiple synchronized infrared cameras, each video stream representing a camera view; further, the video streams are decomposed into frames and each frame is processed.

[0025] In this embodiment, the deep learning model uses the OpenPose model, which is a widely used open source human posture estimation model that can detect 18 key points of the human body, including the head, shoulders, elbows, wrists, hips, knees, ankles, etc.; optionally, a pre-trained model is used, or training is performed based on a public data set to improve adaptability to different scenarios and postures.

[0026] For the OpenPose model of this embodiment, its network structure includes two parts: a feature extraction network and a key point detection network. The feature extraction network usually uses a convolutional neural network to extract image features, and the key point detection network gradually refines the prediction through convolution operations. Furthermore, OpenPose adopts a multi-stage prediction strategy, and each stage is refined according to the prediction results of the previous stage, gradually improving the accuracy of key point detection. In addition, OpenPose generates confidence maps and partial association fields for key points. The confidence map represents the probability that each pixel belongs to a certain key point, and the PAF (Part Affinity Fields) represents the association between key points.

[0027] Specifically, based on the confidence map and PAF, a greedy algorithm is used to match key points and build a complete skeleton. The detected key points are screened and points with low confidence are removed to improve the accuracy and stability of the overall detection.

[0028] In the specific implementation, after the two-dimensional skeleton key points are obtained, the 3D skeleton coordinates of the human body in space are reconstructed through triangulation method using the calibration information of multiple cameras and 2D key points.

[0029] Specifically, camera calibration is to obtain the intrinsic and extrinsic parameters of the camera to determine the geometric characteristics of the camera; based on multiple cameras, through the cross-3D reconstruction results of multiple perspectives, weighted averaging or other fusion algorithms are used to further optimize the accuracy of the three-dimensional points.

[0030] Please see further Figure 1 : S2: extract representative bone key points associated with the breathing of the target object from the three-dimensional bone key points, and obtain displacement peaks and displacement troughs of the representative bone key points.

[0031] In a specific implementation, representative skeleton key points related to breathing are extracted as the basis of the breathing signal. Optionally, it includes but is not limited to the 3D coordinates of the shoulder.

[0032] In some embodiments of the present application, the representative skeleton key points are low-pass filtered and smoothed by a Butterworth filter to filter out high-frequency noise, and the representative skeleton key points are smoothed by a Kalman filter to filter out instantaneous fluctuations.

[0033] In the specific implementation, the processed breathing signal is analyzed in real time to identify the peaks and troughs of breathing. Specifically, the peak is the local maximum point of the key point displacement, and the trough is the local minimum point. The signal is traversed, and the size of the current point and its left and right adjacent points are compared to determine the peaks and troughs.

[0034] In some embodiments of the present application, when obtaining the displacement peaks and displacement troughs, a preset minimum peak-to-trough spacing is met.

[0035] Specifically, the minimum distance between adjacent peaks and valleys is considered to avoid pseudo peaks and pseudo valleys.

[0036] S3: Determine the breathing phase of the target object based on the displacement peak and the displacement trough to perform breathing monitoring on the target object.

[0037] In a specific implementation, the inhalation and exhalation phases of breathing are determined by identifying the peaks and troughs of breathing.

[0038] In some embodiments of the present application, the time interval between the displacement peak and the displacement trough is calculated to obtain the breathing frequency of the target object.

[0039] In a specific implementation, the breathing frequency is output in real time by calculating the time interval between the peak and the trough. Preferably, a dynamic threshold and confidence calculation method are used to improve the accuracy of the breathing stage recognition.

[0040] In some embodiments of the present application, a sliding window is set for the displacement process representing the key points of the skeleton, the wave value type that appears most frequently in the sliding window is used as the result wave value type of the sliding window, and the breathing stage of the sliding window is determined based on the result wave value type.

[0041] In the specific implementation, a window voting mechanism is used to avoid misjudgment caused by abnormal instantaneous data. The number of occurrences of each breathing stage (inhalation or exhalation) in the window is counted, and the stage with the most occurrences is taken as the result.

[0042] In summary, the method provided in the embodiments of the present application can effectively overcome the shortcomings of the existing respiratory monitoring methods and provide an efficient, convenient and accurate respiratory monitoring method, specifically: 1. Real-time: Through efficient image processing and data analysis algorithms, it can calculate and output respiratory rate and respiratory stage in real time; it is suitable for scenarios that require rapid feedback, such as medical monitoring, sports health, etc.

[0043] 2. High precision: Utilizes the calibration information of multiple cameras and deep learning technology to accurately extract and reconstruct the 3D skeleton coordinates of the human body; uses multiple smoothing and confidence calculations to improve the reliability and accuracy of respiratory signals; compared with directly measuring the changes in pixels or optical flow on the image, it can provide more accurate monitoring during human movement and exercise.

[0044] 3. Non-contact, non-wearable, and non-cooperative: It is completely based on image processing and does not require users to wear equipment, avoiding the various inconveniences of traditional methods; it does not rely on active signal transmission (such as radar, audio, WiFi), and is suitable for a wider range of application scenarios; the monitoring process does not require users to be informed in advance or to make any behavioral and psychological cooperation, and can also be measured in displacement scenarios such as human walking.

[0045] 4. Multi-target monitoring: It supports simultaneous monitoring of multiple human bodies and can be widely used in public places, medical institutions and other scenarios. Through synchronous monitoring and independent analysis of multiple targets, real-time respiratory monitoring of multiple people can be achieved.

[0046] It should be understood that, although the various steps in the flow chart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the figure may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0047] See also Figure 2 An embodiment of the present application provides a non-contact respiratory monitoring system, which includes a three-dimensional reconstruction module 10, a signal acquisition module 20 and a respiratory monitoring module 30; wherein, A three-dimensional reconstruction module 10 is used to obtain two-dimensional images of multiple perspectives of a respiratory monitoring target object through multiple cameras, obtain two-dimensional skeleton key points of the target object at each perspective according to the two-dimensional images, and obtain three-dimensional skeleton key points of the target object through triangulation according to the two-dimensional skeleton key points; The signal acquisition module 20 is used to extract representative skeleton key points associated with the breathing of the target object from the three-dimensional skeleton key points, and to obtain displacement peaks and displacement troughs of the representative skeleton key points; The breathing monitoring module 30 is used to determine the breathing stage of the target object based on the displacement peak and the displacement trough, so as to perform breathing monitoring on the target object.

[0048] For the specific definition of the above-mentioned non-contact respiratory monitoring system, please refer to the definition of the non-contact respiratory monitoring method above, which will not be repeated here. Each module in the above-mentioned non-contact respiratory monitoring system can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0049] In one embodiment, a computer device is provided, the internal structure diagram of which can be as follows: Figure 3 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the above-mentioned non-contact respiratory monitoring method is implemented. It includes: a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, the above-mentioned non-contact respiratory monitoring method is implemented.

[0050] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the non-contact respiratory monitoring method as described above can be implemented.

[0051] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program codes. The scheme in the embodiments of the present application may be implemented in various computer languages, for example, C language, VHDL language, Verilog language, object-oriented programming language Java, and literal scripting language JavaScript, etc.

[0052] The present application is described with reference to the flowcharts and / or block diagrams of the systems and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0053] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0054] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0055] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0056] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A non-contact respiratory monitoring method, characterized in that: include: Acquire two-dimensional images of a respiratory monitoring target object at multiple viewing angles through multiple cameras, acquire two-dimensional skeleton key points of the target object at each viewing angle according to the two-dimensional images, and acquire three-dimensional skeleton key points of the target object through triangulation according to the two-dimensional skeleton key points; Extracting representative skeletal key points associated with the breathing of the target object from among the three-dimensional skeletal key points, and obtaining displacement peaks and displacement troughs of the representative skeletal key points; The breathing phase of the target object is determined based on the displacement peak and the displacement trough to perform breathing monitoring on the target object.

2. The non-contact respiratory monitoring method according to claim 1, characterized in that: The steps of acquiring two-dimensional images of multiple perspectives of a respiratory monitoring target object through multiple cameras, acquiring two-dimensional skeleton key points of the target object at each perspective according to the two-dimensional images, and acquiring three-dimensional skeleton key points of the target object through triangulation according to the two-dimensional skeleton key points also include: The cameras are synchronous cameras, which are synchronized by pulse signals so that each frame of the two-dimensional image is consistent in time.

3. The non-contact respiratory monitoring method according to claim 1, characterized in that: The step of extracting representative skeleton key points associated with the breathing of the target object from the three-dimensional skeleton key points and obtaining displacement peaks and displacement troughs of the representative skeleton key points also includes: The representative skeleton key points are subjected to low-pass filtering and smoothing processing through a Butterworth filter to filter out high-frequency noise.

4. The non-contact respiratory monitoring method according to claim 3, characterized in that: The step of extracting representative skeleton key points associated with the breathing of the target object from the three-dimensional skeleton key points and obtaining displacement peaks and displacement troughs of the representative skeleton key points also includes: The representative skeleton key points are smoothed by a Kalman filter to filter out instantaneous fluctuations.

5. The non-contact respiratory monitoring method according to claim 1 or 4, characterized in that: The step of extracting representative skeleton key points associated with the breathing of the target object from the three-dimensional skeleton key points and obtaining displacement peaks and displacement troughs of the representative skeleton key points also includes: When obtaining the displacement peak and the displacement trough, a preset minimum peak-to-trough spacing is satisfied.

6. The non-contact respiratory monitoring method according to claim 1, characterized in that: The step of determining the breathing stage of the target object based on the displacement peak and the displacement trough to perform breathing monitoring on the target object also includes: The time interval between the displacement peak and the displacement trough is calculated to obtain the breathing frequency of the target object.

7. The non-contact respiratory monitoring method according to claim 1 or 6, characterized in that: The step of determining the breathing stage of the target object based on the displacement peak and the displacement trough to perform breathing monitoring on the target object also includes: A sliding window is set for the displacement process representing the key points of the skeleton, the wave value type that appears most times in the sliding window is used as the result wave value type of the sliding window, and the breathing stage of the sliding window is determined according to the result wave value type.

8. A non-contact respiratory monitoring system, characterized in that: include: Three-dimensional reconstruction module, signal acquisition module and respiratory monitoring module; among them, A three-dimensional reconstruction module, used to obtain two-dimensional images of multiple perspectives of a respiratory monitoring target object through multiple cameras, obtain two-dimensional skeleton key points of the target object at each perspective according to the two-dimensional images, and obtain three-dimensional skeleton key points of the target object through triangulation according to the two-dimensional skeleton key points; A signal acquisition module, used to extract representative skeleton key points associated with the breathing of the target object from the three-dimensional skeleton key points, and to obtain displacement peaks and displacement troughs of the representative skeleton key points; The breathing monitoring module is used to determine the breathing stage of the target object based on the displacement peak and the displacement trough so as to perform breathing monitoring on the target object.

9. A computer device, characterized in that: include: Memory; processor; as well as Computer programs; The computer program is stored in the memory and is configured to be executed by the processor to implement the non-contact respiratory monitoring method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon; the computer program is executed by a processor to implement the non-contact respiratory monitoring method as described in any one of claims 1-7.

Images