A non-contact respiratory monitoring method, system, terminal device and medium

Abstract

The present invention discloses a non-contact respiratory monitoring method, system, terminal device and medium, the method comprising: obtaining a video of the chest and abdomen area of ​​a user, and segmenting each frame of the video to obtain a number of corresponding image blocks; calculating the velocity components of the image blocks to obtain velocity signals of respiratory motion in different directions; performing singular value decomposition and signal fusion on the velocity signal to obtain a respiratory signal; calculating the heartbeat interval of the respiratory signal to obtain the user's respiratory rate. The present invention introduces a singular value decomposition method in non-contact respiratory monitoring, decomposes and fuses the optical flow velocities in two directions in all image blocks of the video image, and finally generates a more stable and robust respiratory signal, which can maintain a high signal extraction accuracy even when there is an angle deviation between the camera and the object being measured.

Description

Technical Field

[0001] The present invention relates to the technical field of non-contact respiratory monitoring, and in particular to a non-contact respiratory monitoring method, system, terminal equipment and medium for multi-rotation angle images. Background Art

[0002] As an important physiological parameter for evaluating cardiopulmonary function, respiratory rate is of great significance for many clinical applications, especially in the fields of intensive care, surgical anesthesia, and neonatal care. Although traditional respiratory monitoring methods such as electrical impedance plethysmography, airflow sensors, and capnography can provide highly accurate respiratory data, they often require direct contact with patients or the use of invasive equipment, which may not only cause discomfort to patients but also interfere with their natural breathing patterns.

[0003] Camera-based respiratory monitoring is a completely non-invasive, non-intrusive, and non-contact technology that is gradually gaining attention in the medical field. This technology uses a high-resolution camera to capture the tiny movements of the patient's chest and abdomen, and uses image processing algorithms to extract respiratory signals. The advantage of this method is its non-contact nature, making it particularly suitable for patients with sensitive skin, infectious diseases, or those who need long-term monitoring, such as newborns, burn patients, and immunocompromised people. In addition, camera-based respiratory monitoring can achieve remote monitoring and continuous monitoring, which is of great significance for home care and telemedicine.

[0004] Camera-based respiratory monitoring technology has been developed to a certain extent, and the main implementation method is the PixFlow optical flow algorithm. By analyzing the pixel displacement between adjacent frames, the horizontal and vertical movements of the chest and abdomen are calculated respectively, and the direction with a higher signal-to-noise ratio is selected as the respiratory signal. In addition, the horizontal and vertical directions are fused as the angle between the orthogonal respiratory velocity vectors to obtain the angular velocity of the respiratory movement. However, the above algorithm is angle-dependent, and accurate respiratory signals can only be obtained when the patient is at a specific angle, usually facing the camera. If the angle between the camera and the patient is not ideal, the chest and abdominal movements in the image may not be accurately captured, causing the algorithm to output inaccurate or unstable respiratory signals.

[0005] Therefore, it is necessary to propose a technology that can achieve accurate respiratory signal monitoring at any angle. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a non-contact respiratory monitoring method, system, terminal device and medium in view of the above-mentioned defects of the prior art, aiming to solve the problem that the existing non-contact respiratory detection method requires the patient to be at a specific angle, usually facing the camera, to obtain an accurate respiratory signal. Specifically, in the prior art, if the angle between the camera and the patient is not ideal, the chest and abdominal movements in the image may not be accurately captured, resulting in the algorithm outputting inaccurate or unstable respiratory signals.

[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is as follows:

[0008] In a first aspect, the present invention provides a non-contact respiratory monitoring method, the method comprising:

[0009] Acquire a video of the chest and abdomen area of ​​the user, and segment each frame of the video to obtain a number of image blocks corresponding to each frame of the video;

[0010] Calculating the velocity component of each of the image blocks to obtain velocity signals of the respiratory motion of each of the image blocks in different directions;

[0011] For each image block, singular value decomposition is performed on the speed signals in different directions to obtain corresponding speed features, and based on the speed features, signal fusion is performed on the speed signals in different directions to obtain a fused breathing signal for each image block;

[0012] fusing the fused respiratory signals of all the image blocks to obtain an overall respiratory signal;

[0013] The overall breathing signal is accumulated and processed, and the heartbeat interval is calculated to obtain the user's breathing rate.

[0014] In one implementation, the step of acquiring a video of the chest and abdomen area of ​​the user and segmenting each frame of the video to obtain a plurality of image blocks corresponding to each frame of the video includes:

[0015] Shoot the chest and abdomen area of ​​the user at any shooting angle to obtain a video of the chest and abdomen area of ​​the user;

[0016] Based on the shooting angle, image segmentation parameters are obtained, and based on the image segmentation parameters, each frame of the image in the video is segmented to obtain a plurality of image blocks corresponding to each frame of the image.

[0017] In one implementation, calculating the velocity component of each image block to obtain the velocity signal of the respiratory motion of each image block in different directions includes:

[0018] For each of the image blocks, based on the difference of the image block in different frame images, calculating the velocity components of the image block in the horizontal direction and the vertical direction;

[0019] Based on the speed component of the image block in each frame of the image, speed signals of the image block in the horizontal direction and the vertical direction are obtained.

[0020] In one implementation, the calculating of the velocity component of each image block to obtain the velocity signal of the respiratory motion of each image block in different directions further includes:

[0021] Detrending the velocity signals in the horizontal and vertical directions is performed to eliminate low-frequency noise and baseline drift.

[0022] In one implementation, for each image block, performing singular value decomposition on the speed signals in different directions to obtain corresponding speed features includes:

[0023] According to the speed signal, obtaining speed matrices of the image block in the horizontal direction and the vertical direction;

[0024] Perform singular value decomposition on the velocity matrix to obtain the eigenvector and eigenvalue corresponding to the image block.

[0025] In one implementation, based on the speed feature, performing signal fusion on the speed signals in different directions to obtain a fused breathing signal for each image block includes:

[0026] The eigenvector with the largest eigenvalue is selected, and based on the selected eigenvector, signal fusion is performed on the speed signals in the horizontal direction and the vertical direction to obtain a fused respiratory signal of the image block.

[0027] In one implementation, fusing the fused respiratory signals of all the image blocks to obtain an overall respiratory signal includes:

[0028] Selecting a target local respiratory signal according to an average heartbeat interval feature of the fused respiratory signal of the image block;

[0029] The selected target local respiratory signals are averaged and fused to obtain a one-dimensional overall respiratory signal.

[0030] In a second aspect, an embodiment of the present invention further provides a non-contact respiratory monitoring system, the system comprising:

[0031] An image block acquisition module is used to acquire a video of the chest and abdomen area of ​​the user, and segment each frame of the video to obtain a number of image blocks corresponding to each frame of the video;

[0032] A speed signal acquisition module, used for calculating the speed component of each image block to obtain the speed signal of the respiratory movement of each image block in different directions;

[0033] The fused breathing signal acquisition module of the image block is used to perform singular value decomposition on the speed signals in different directions for each image block to obtain corresponding speed features, and perform signal fusion on the speed signals in different directions based on the speed features to obtain a fused breathing signal for each image block;

[0034] An overall respiratory signal acquisition module, used for fusing the fused respiratory signals of all the image blocks to obtain an overall respiratory signal;

[0035] The breathing rate acquisition module is used to accumulate and process the overall breathing signal and calculate the heartbeat interval to obtain the user's breathing rate.

[0036] In a third aspect, an embodiment of the present invention further provides a terminal device, comprising a memory, a processor, and a non-contact respiratory monitoring program stored in the memory and executable on the processor, wherein when the processor executes the non-contact respiratory monitoring program, the steps of the non-contact respiratory monitoring method of any one of the above-mentioned schemes are implemented.

[0037] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a non-contact respiratory monitoring program is stored. When the non-contact respiratory monitoring program is executed by a processor, the steps of the non-contact respiratory monitoring method described in any one of the above schemes are implemented.

[0038] Beneficial effects: The present invention discloses a non-contact respiratory monitoring method, system, terminal device and medium. The method first obtains a video of the chest and abdomen area of ​​the user, and segments each frame of the video to obtain a number of image blocks corresponding to each frame of the image. Then, the velocity components of the image blocks are calculated to obtain velocity signals of respiratory motion in different directions. Next, singular value decomposition and signal fusion are performed on the velocity signal to obtain a respiratory signal. Finally, the heartbeat interval of the respiratory signal is calculated to obtain the user's respiratory rate. The present invention introduces a singular value decomposition method in non-contact respiratory monitoring, decomposes and fuses the optical flow velocities in two directions in all image blocks of the video image, and finally generates a more stable and robust respiratory signal, which can maintain a high signal extraction accuracy even when there is an angle deviation between the camera and the object being measured. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 The present invention provides a flowchart of a specific implementation of the non-contact respiratory monitoring method.

[0040] Figure 2 It is a flow chart of non-contact respiratory monitoring signal extraction provided by an embodiment of the present invention.

[0041] Figure 3 It is a principle block diagram of a non-contact respiratory monitoring device provided by an embodiment of the present invention.

[0042] Figure 4 It is a block diagram of the internal structure principle of the terminal device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0044] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations or steps, nor must they be executed in the order described. For example, some operations or steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0045] It should be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0046] It should be understood that, in order to clearly describe the technical solutions of the embodiments of the present invention, in the embodiments of the present invention, words such as "first" and "second" are used to distinguish between identical or similar items with substantially identical functions and effects. For example, the first control information and the second control information are only used to distinguish different control information, and their order is not limited.

[0047] Those skilled in the art can understand that the words "first", "second", etc. do not limit the quantity and execution order, and the words "first", "second", etc. do not necessarily limit the differences.

[0048] It should be further understood that the term “and / or” used in the present specification and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0049] Respiratory rate is a key physiological parameter for assessing cardiopulmonary function and is of great significance in clinical fields such as intensive care, surgical anesthesia, and neonatal care. Traditional respiratory monitoring methods, such as electrical impedance plethysmography, airflow sensors, and carbon dioxide waveforms, can provide high-precision respiratory data, but they require direct contact with patients or the use of invasive equipment, which can easily cause discomfort to patients and interfere with natural breathing patterns. Camera-based respiratory monitoring, as a non-invasive, non-intrusive, non-contact technology, is gaining attention in the medical field. It uses a high-resolution camera to capture tiny movements of the patient's chest and abdomen, and uses image processing algorithms to extract respiratory signals. Due to its non-contact characteristics, it is particularly suitable for patients with sensitive skin, infectious diseases, or those who need long-term monitoring. It can also achieve remote and continuous monitoring, which is crucial for home care and telemedicine. At present, this technology is mainly implemented through the PixFlow optical flow algorithm, which analyzes the displacement of pixels in adjacent frames to calculate the amount of motion, selects the direction with a high signal-to-noise ratio as the respiratory signal, and fuses the direction angle to obtain the angular velocity of the respiratory movement. However, this algorithm is angle-dependent and can only accurately obtain respiratory signals when the patient is facing the camera. If the angle between the camera and the patient is not good, chest and abdominal movements are difficult to capture accurately, making the respiratory signal output by the algorithm inaccurate or unstable.

[0050] To solve the above problems, the applicant introduced the singular value decomposition (SVD) method. Specifically, the method first fuses the optical flow velocities in the horizontal and vertical directions. During the fusion process, the optical flow velocity matrix is ​​decomposed into multiple independent components. This decomposition method can effectively reduce the impact of angle changes on the optical flow velocity. Because the changes in optical flow velocity at different angles are complex, these changes can be analyzed and processed more accurately after being decomposed into independent components. By fusing these independent components, a more stable and robust respiratory signal can be generated. Even in the case of an angular deviation between the camera and the object being measured, the method can still maintain a high signal extraction accuracy. This improvement improves the reliability and practicality of non-contact respiratory monitoring technology, especially for scenarios that require long-term, multi-angle monitoring, such as home care, telemedicine, and some special medical environments, such as neonatal care, infection patient monitoring, etc.

[0051] This embodiment provides a non-contact respiratory monitoring method, such as Figure 1 As shown, the specific steps include:

[0052] Step S100: obtaining a video of the chest and abdomen area of ​​the user, and segmenting each frame of the video to obtain a plurality of image blocks corresponding to each frame of the video.

[0053] In this embodiment, as a non-contact breathing detection method, it is necessary to analyze the image and video data of the user, and the first step is to collect the video image data of the user. Specifically, when the user performs breathing movements, the lungs retract and expand and the volume of the chest cavity changes under the contraction and extension of the diaphragm and external intercostal muscles. In other words, when the user performs breathing movements, it is ultimately manifested as the rise and fall of the chest and abdomen. Therefore, collecting the video image data of the user, specifically collecting video images of the chest and abdomen area of ​​the user, obtains the rise and fall state of the chest and abdomen of the user. Preferably, the chest and abdomen area of ​​the user is continuously captured at a fixed angle and distance. Furthermore, after obtaining a complete video image of the chest and abdomen of the user, under the premise that the camera angle is fixed, such as Figure 2 As shown, the image screen of each frame in the video is segmented to obtain a number of image blocks corresponding to the image screen of each frame. Generally, each frame of the image is divided into M×N image blocks, and each image block is used as an independent signal detection area for subsequent extraction of respiratory signals. In this way, a frame sequence of each of the M×N image blocks in the video is obtained, that is, an M×N image block video frame sequence. In each image block video frame sequence, each frame is an image block. By combining the image blocks in the same frame of all image block video frame sequences, a complete image screen of the original video image in the frame can be obtained.

[0054] In one implementation, a video of the chest and abdomen area of ​​a user is obtained, and each frame of the video is segmented to obtain a plurality of image blocks corresponding to each frame of the video, specifically including the following steps:

[0055] Step S110, photographing the chest and abdomen area of ​​the user at any shooting angle to obtain a video of the chest and abdomen area of ​​the user;

[0056] Step S120: obtaining image segmentation parameters based on the shooting angle, and segmenting each frame of the video based on the image segmentation parameters to obtain a plurality of image blocks corresponding to each frame of the video.

[0057] In this embodiment, when the user's chest and abdomen area is photographed to obtain a video image, a fixed shooting angle and distance are required so that the spatial posture of the shooting device relative to the user's chest and abdomen is fixed during the shooting process, the image size remains consistent, and the size and spatial orientation of the segmented image blocks in each video frame are ensured to be consistent, so as to be used to calculate the changes of subsequent image blocks in consecutive frames. Although the shooting angle needs to be fixed, the selection of the shooting angle is not restricted. It is only necessary to ensure that the fixed shooting angle can capture the ups and downs of the user's chest and abdomen. While fixing the shooting angle, the relative distance between the shooting device and the user also needs to be fixed. The distance is generally selected between 50cm and 100cm. Preferably, in this embodiment, the relative distance between the shooting device and the user is set at 50cm. Further, after the shooting angle and distance are fixed, the relevant parameters of image segmentation can be obtained through the shooting angle and distance. The relevant parameters include at least the image size and ratio and the position and angle of the user's chest and abdomen in the image. Specifically, first, through the shooting distance, combined with the focal length or other camera parameters, the proportional relationship between the actual size of the object in the image and the imaged size can be determined. When the shooting distance is fixed, different focal lengths will make the size of the chest and abdomen in the image different. The image size parameters specify the pixel values ​​of the width and height of the image, which are used to determine the size and number of the image blocks after segmentation. In other words, when segmenting each frame of the user's chest and abdomen into M×N image blocks, it is necessary to calculate the pixel range of each image block based on the above size information to obtain the image block number parameters M and N.

[0058] Step S200: Calculate the velocity component of each image block to obtain velocity signals of the respiratory motion of each image block in different directions.

[0059] In this embodiment, after each frame of the video image is divided into M×N image blocks, an M×N image block video frame sequence is obtained. For each image block video frame sequence in the M×N image block video frame sequence, the speed component of each image block can be calculated based on the difference between an image block in an image block video frame sequence and the image blocks before and after the image block in the image block video frame sequence. Combining the speed components of each image block in an image block video frame sequence, a speed signal of the image block video frame sequence can be obtained. Furthermore, by summarizing the speed signals of all image block video frame sequences, a speed signal of respiratory movement can be obtained.

[0060] In one implementation, the calculating of the velocity component of each image block to obtain the velocity signal of the respiratory motion of each image block in different directions specifically includes the following steps:

[0061] Step S210: for each of the image blocks, based on the difference of the image block in different frame images, calculate the velocity components of the image block in the horizontal direction and the vertical direction;

[0062] Step S220: Obtain speed signals of the image block in the horizontal direction and the vertical direction based on the speed component of the image block in each frame of the image.

[0063] In this embodiment, after each frame of the video image is divided into M×N image blocks, an M×N image block video frame sequence is obtained. For each image block video frame sequence in the M×N image block video frame sequence, as shown in FIG. Figure 2 As shown, using the PixFlow algorithm, the speed component of each frame of the image block can be calculated based on the contrast difference between a frame of the image block in an image block video frame sequence and the previous and next frame of the image block in the image block video frame sequence. Specifically, under the premise of a fixed shooting angle and distance, and when the ambient lighting of the shooting does not change, when the pixels in the image block move between different frames, the brightness of the moving pixels will not change. Because the fluctuation of the chest and abdomen during breathing movement is small, by calculating the brightness change of the pixels in each image block, the optical flow constraint equation is used to solve the average horizontal and vertical movement speed of the pixels in the image block, and the speed component of the image block in the horizontal and vertical directions is obtained. Combined with the speed components in the horizontal and vertical directions of each frame of the image block in an image block video frame sequence, the speed signal in the horizontal and vertical directions of the image block video frame sequence can be obtained.

[0064] In one implementation, the calculating of the velocity component of each image block to obtain the velocity signal of the respiratory motion of each image block in different directions specifically includes the following steps:

[0065] Step S230: Detrending the speed signals in the horizontal and vertical directions to eliminate low-frequency noise and baseline drift.

[0066] In this embodiment, in the process of acquiring the horizontal and vertical velocity signals of respiratory movement, noise will be mixed into the velocity signal due to the influence of environmental factors such as low-frequency vibration near the monitoring equipment and low-frequency interference of the power system. Among them, low-frequency noise will cause unnecessary fluctuations in the velocity signal and mask the true characteristics of the respiratory signal. In addition, in respiratory monitoring, due to slight movements of the human body, posture adjustments or instability of the sensor, the velocity signal will produce baseline drift. The baseline drift refers to the DC component of the signal that changes slowly over time, which will change the overall trend of the velocity signal. Therefore, if Figure 2As shown, the speed signals in the horizontal and vertical directions are preprocessed for trend removal to eliminate low-frequency noise and baseline drift to ensure the accuracy of the signal. Specifically, the trend removal method includes a high-pass filtering method, a polynomial fitting removal method, and an empirical mode decomposition method. Preferably, in this embodiment, a high-pass filtering method is used to preprocess the speed signal. First, the coefficient of the filter is determined according to the set filter type and cut-off frequency. Then, the speed signals in the horizontal and vertical directions are respectively input into the set high-pass filters for filtering. In this embodiment, a digital filter design tool is used to implement the design of the high-pass filter and signal filtering.

[0067] Step S300: for each image block, singular value decomposition is performed on the speed signals in different directions to obtain corresponding speed features, and based on the speed features, signal fusion is performed on the speed signals in different directions to obtain a fused breathing signal for each image block.

[0068] In this embodiment, after obtaining the speed signals of respiratory movement in different directions, it is necessary to further perform singular value decomposition on the speed signals to obtain corresponding speed features, and based on the speed features, the speed signals of different directions of the image block are fused to obtain the fused respiratory signal of the image block.

[0069] In one implementation, for each image block, singular value decomposition is performed on the speed signals in different directions to obtain corresponding speed features, which specifically includes the following steps:

[0070] Step S310, obtaining a velocity matrix of the image block in the horizontal direction and the vertical direction according to the velocity signal;

[0071] Step S320, performing singular value decomposition on the velocity matrix to obtain eigenvectors and eigenvalues ​​corresponding to the image block;

[0072] In this embodiment, after completing the calculation of the velocity component, we obtain the velocity signal of each image block that changes with time in the horizontal direction and the vertical direction. The velocity signal is presented in the form of a time series, and each time point corresponds to a velocity value. In order to perform singular value decomposition later, the above-mentioned one-dimensional velocity signal is converted into a matrix form. Specifically, the velocity signal of each image block includes T time points, and the velocity matrix in the horizontal direction and the velocity matrix in the vertical direction are matrices with T rows, including T velocity values ​​corresponding to the velocity signal in the horizontal direction, and T velocity values ​​corresponding to the velocity signal in the vertical direction.

[0073] After obtaining the velocity matrices of the image block in the horizontal and vertical directions, as Figure 2As shown, the above velocity matrix is ​​subjected to singular value decomposition to obtain a left singular matrix, a diagonal matrix containing singular values, and a right singular matrix. Accordingly, through the three matrices obtained by the above decomposition, the eigenvalues ​​and eigenvectors corresponding to the velocity matrix in the horizontal and vertical directions can be obtained.

[0074] In one implementation, based on the speed feature, signal fusion is performed on the speed signals in different directions to obtain a fused breathing signal for each image block, which specifically includes the following steps:

[0075] Step S330: Select the eigenvector with the largest eigenvalue, and based on the selected eigenvector, perform signal fusion on the speed signals in the horizontal direction and the vertical direction to obtain a fused breathing signal of the image block.

[0076] In this embodiment, after performing singular value decomposition on the velocity matrix in the horizontal direction and the vertical direction, a series of eigenvectors and corresponding eigenvalues ​​are obtained, and the eigenvalue is the square of the singular value. The size of the eigenvalue reflects the importance of the pattern represented by the eigenvector in the original velocity signal. The larger the eigenvalue, the higher the signal energy carried by the eigenvector, and the more it can represent the main characteristics of the respiratory movement. Further, the eigenvalues ​​corresponding to all eigenvectors are traversed, and the eigenvector corresponding to the largest eigenvalue is determined by comparison. After the eigenvector is selected, as Figure 2 As shown, the speed signals in the horizontal direction and the vertical direction are fused. Preferably, in the present embodiment, the speed signals in the horizontal direction and the vertical direction of the image block are fused by weighted summation. According to the selected eigenvector, the speed signals in the horizontal direction and the vertical direction are respectively given weights corresponding to the eigenvector elements, and then weighted summation is performed. In other words, the speed signals in the horizontal direction and the vertical direction are fused according to the importance embodied by the eigenvector, and a fused respiratory signal of the image block that can comprehensively reflect the respiratory motion is obtained.

[0077] Step S400: Fusing the fused respiratory signals of all the image blocks to obtain an overall respiratory signal.

[0078] In this embodiment, after the original video image is segmented, a plurality of image block video frame sequences are obtained. That is, the fused breathing signal of the corresponding image block can be calculated for each image block. Although the fused breathing signal of each image block contains information about the respiratory movement, each signal may have certain fluctuations and noises due to the influence of factors such as the position and size of the image block. Figure 2As shown, by averaging and fusing the fused respiratory signals of all image blocks, the signal is smoothed and the influence of noise is reduced, and the respiratory information of each image block is integrated to obtain a signal representing the overall respiratory movement. Specifically, the fused respiratory signals corresponding to the M×N image blocks are averaged and fused to obtain the overall respiratory signal. More specifically, for a video image cut into a video frame sequence of M×N image blocks and containing T time points, for any time point t among the T time points, the respiratory signal values ​​of the corresponding M×N image blocks at time point t are averaged and summed to obtain the overall respiratory signal value at time point t. The overall respiratory signal values ​​of the T time points are combined to obtain the overall respiratory signal.

[0079] In one implementation, fusing the fused respiratory signals of all the image blocks to obtain an overall respiratory signal specifically includes the following steps:

[0080] Step S410, selecting a target local respiratory signal according to the average heartbeat interval feature of the fused respiratory signal of the image block;

[0081] Step S420: average and fuse the selected target local respiratory signals to obtain a one-dimensional overall respiratory signal.

[0082] In this embodiment, in any frame of video image, the fused respiratory signal value of the image block may still contain noise. Therefore, before averaging and fusing the fused respiratory signal corresponding to the image block, the image block needs to be screened. Specifically, since the respiratory signal will show periodic changes, similar to the ups and downs of a heartbeat, an amplitude threshold is preset. When the signal value exceeds the threshold, it is considered that a feature point similar to a heartbeat is detected. Then, the time interval between adjacent feature points is calculated, and the average value of the above time interval is calculated to obtain the average inter-beat interval (mIBI). Based on the age of the user, the range of the average inter-beat interval is determined. For example, for adults, the normal breathing rate is about 12-20 times per minute, which is converted into an average inter-beat interval of about 3-5 seconds, while the average inter-beat interval of infants and young children needs to be specifically determined based on their age. Based on the above-determined average inter-beat interval, further, such as Figure 2 As shown in the figure, the average heartbeat interval of the fused respiratory signals of all image blocks is calculated, and the fused respiratory signals of the image blocks whose average heartbeat interval is within the normal range are selected as high-quality target local respiratory signals. In this way, the influence of noise interference and other non-respiratory movements of the human body can be reduced, and the signal anomalies caused by unreasonable image block segmentation can be reduced. Finally, the selected target local respiratory signals are averaged and fused to obtain a one-dimensional overall respiratory signal.

[0083] Step S500: Accumulate and process the overall breathing signal and calculate the heartbeat interval to obtain the user's breathing rate.

[0084] In this embodiment, after acquiring the overall respiratory signal, the overall respiratory signal is accumulated and processed to smooth the signal and reduce the influence of noise, thereby obtaining a final overall respiratory movement signal. After obtaining the above-mentioned overall respiratory signal, the characteristic points of each respiratory cycle in the respiratory signal are determined, and the peaks or troughs of the respiratory signal are usually selected as characteristic points. Preferably, the characteristic points are detected using a threshold method or a derivative method. After determining the characteristic points of each respiratory cycle in the respiratory signal, the time intervals between adjacent characteristic points are calculated to obtain the respiratory interval. By using the average respiratory interval method or the segment-by-segment calculation method, such as Figure 2 As shown, the user's breathing rate is calculated.

[0085] In summary, under the technical solution of the above embodiment, the problem of respiratory signal extraction accuracy caused by the angle dependence of the traditional optical flow algorithm is solved by introducing the singular value decomposition method in non-contact respiratory monitoring. Specifically, the optical flow velocities in two directions in all image blocks of the video image are decomposed and fused, and finally a more stable and robust respiratory signal is generated. Even if there is an angle deviation between the camera and the object being measured, this technical solution can maintain a high signal extraction accuracy, thereby improving the reliability and practicality of non-contact respiratory monitoring technology.

[0086] like Figure 3 As shown in the figure, an embodiment of the present invention provides a non-contact respiratory monitoring system, which includes: an image block acquisition module 10, a speed signal acquisition module 20, an image block fusion respiratory signal acquisition module 30, an overall respiratory signal acquisition module 40, and a respiratory rate acquisition module 50.

[0087] Specifically, the image block acquisition module 10 is used to acquire a video of the chest and abdomen area of ​​the user, and to segment each frame of the video to obtain a number of image blocks corresponding to each frame of the image; the speed signal acquisition module 20 is used to calculate the speed component of each image block to obtain the speed signal of the respiratory movement of each image block in different directions; the image block fused respiratory signal acquisition module 30 is used to perform singular value decomposition on the speed signals in different directions for each image block to obtain the corresponding speed features, and based on the speed features, perform signal fusion on the speed signals in different directions to obtain a fused respiratory signal for each image block; the overall respiratory signal acquisition module 40 is used to fuse the fused respiratory signals of all the image blocks to obtain an overall respiratory signal; the respiratory rate acquisition module 50 is used to accumulate and process the overall respiratory signal and calculate the heartbeat interval to obtain the user's respiratory rate.

[0088] In one implementation, the image block acquisition module includes:

[0089] A chest and abdomen video shooting unit, used to shoot the chest and abdomen area of ​​the user at any shooting angle to obtain a video of the chest and abdomen area of ​​the user;

[0090] The image segmentation unit is used to obtain image segmentation parameters based on the shooting angle, and to segment each frame of the video based on the image segmentation parameters to obtain a plurality of image blocks corresponding to each frame of the video.

[0091] In one implementation, the speed signal acquisition module includes:

[0092] A velocity component acquisition unit, configured to calculate, for each of the image blocks, velocity components of the image block in the horizontal direction and the vertical direction based on differences of the image block in different frame images;

[0093] The speed signal acquisition unit is used to obtain the speed signal of the image block in the horizontal direction and the vertical direction based on the speed component of the image block in each frame of the image.

[0094] In one implementation, the speed signal acquisition module further includes:

[0095] The speed signal preprocessing unit is used to remove the trend of the speed signal in the horizontal direction and the vertical direction, and eliminate low-frequency noise and baseline drift.

[0096] In one implementation, the image block fusion respiratory signal acquisition module includes:

[0097] A velocity matrix acquisition unit, used to obtain velocity matrices of the image block in the horizontal direction and the vertical direction according to the velocity signal;

[0098] The singular value decomposition unit is used to perform singular value decomposition on the velocity matrix to obtain the eigenvector and eigenvalue corresponding to the image block.

[0099] The fused breathing signal acquisition unit of the image block is used to select the eigenvector with the largest eigenvalue, and based on the selected eigenvector, perform signal fusion on the speed signals in the horizontal direction and the vertical direction to obtain the fused breathing signal of the image block.

[0100] In one implementation, the overall breathing signal acquisition module includes:

[0101] A target local respiratory signal selection unit, configured to select a target local respiratory signal according to an average heartbeat interval feature of the fused respiratory signal of the image block;

[0102] The overall respiratory signal acquisition unit is used to average and fuse the selected target local respiratory signals to obtain a one-dimensional overall respiratory signal.

[0103] Based on the above embodiments, the present invention further provides a terminal device, whose principle block diagram can be shown as follows: Figure 4 As shown. The terminal device includes a processor, a memory, a network interface, a display screen, and a temperature sensor connected through a system bus. Among them, the processor of the terminal device is used to provide computing and control capabilities. The memory of the terminal device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the terminal device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a non-contact respiratory monitoring method is implemented. The display screen of the terminal device can be a liquid crystal display screen or an electronic ink display screen, and the temperature sensor of the terminal device is pre-set inside the terminal device to detect the operating temperature of the internal device.

[0104] Those skilled in the art will understand that Figure 4 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the scheme of the present invention, and does not constitute a limitation on the terminal device to which the scheme of the present invention is applied. The specific terminal device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0105] In one embodiment, a terminal device is provided, comprising a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors, wherein the one or more programs include instructions for performing the following operations:

[0106] Acquire a video of the chest and abdomen area of ​​the user, and segment each frame of the video to obtain a number of image blocks corresponding to each frame of the video;

[0107] Calculating the velocity component of the image block to obtain velocity signals of respiratory motion in different directions;

[0108] Performing singular value decomposition and signal fusion on the speed signal to obtain a breathing signal;

[0109] The heartbeat interval of the respiratory signal is calculated to obtain the user's respiratory rate.

[0110] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0111] In summary, the present invention provides a non-contact respiratory monitoring method, system, terminal device and medium. Compared with the prior art, the method first obtains a video of the chest and abdomen area of ​​the user, and segments each frame of the video to obtain a number of image blocks corresponding to each frame of the image. Then, the velocity components of the image blocks are calculated to obtain velocity signals of respiratory motion in different directions. Next, singular value decomposition and signal fusion are performed on the velocity signal to obtain a respiratory signal. Finally, the heartbeat interval of the respiratory signal is calculated to obtain the user's respiratory rate. The present invention introduces a singular value decomposition method in non-contact respiratory monitoring, decomposes and fuses the optical flow velocities in two directions in all image blocks of the video image, and finally generates a more stable and robust respiratory signal, which can still maintain a high signal extraction accuracy even when there is an angle deviation between the camera and the object being measured.

[0112] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0113] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the present application. It should be noted that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A non-contact respiratory monitoring method, characterized in that: The method comprises: Acquire a video of the chest and abdomen area of ​​the user, and segment each frame of the video to obtain a number of image blocks corresponding to each frame of the video; Calculating the velocity component of each of the image blocks to obtain velocity signals of the respiratory motion of each of the image blocks in different directions; For each image block, singular value decomposition is performed on the speed signals in different directions to obtain corresponding speed features, and based on the speed features, signal fusion is performed on the speed signals in different directions to obtain a fused breathing signal for each image block; fusing the fused respiratory signals of all the image blocks to obtain an overall respiratory signal; Accumulating and processing the overall breathing signal and calculating the heartbeat interval to obtain the user's breathing rate; For each image block, performing singular value decomposition on the speed signals in different directions to obtain corresponding speed features includes: According to the speed signal, a speed matrix of the image block in the horizontal direction and the vertical direction is obtained, wherein the speed signal is a time series, each time point corresponds to a speed value, the speed signal of the image block includes T time points, the speed matrix in the horizontal direction and the speed matrix in the vertical direction are matrices with T rows, including T speed values ​​corresponding to the speed signal in the horizontal direction, and T speed values ​​corresponding to the speed signal in the vertical direction; Performing singular value decomposition on the velocity matrix to obtain eigenvectors and eigenvalues ​​corresponding to the image block; The method of fusing the speed signals in different directions based on the speed feature to obtain a fused breathing signal of each image block includes: The eigenvector with the largest eigenvalue is selected, and based on the selected eigenvector, signal fusion is performed on the speed signals in the horizontal direction and the vertical direction to obtain a fused respiratory signal of the image block.

2. The non-contact respiratory monitoring method according to claim 1, characterized in that: The step of acquiring a video of the chest and abdomen area of ​​the user and segmenting each frame of the video to obtain a plurality of image blocks corresponding to each frame of the video includes: Shoot the chest and abdomen area of ​​the user at any shooting angle to obtain a video of the chest and abdomen area of ​​the user; Based on the shooting angle, image segmentation parameters are obtained, and based on the image segmentation parameters, each frame of the image in the video is segmented to obtain a plurality of image blocks corresponding to each frame of the image.

3. The non-contact respiratory monitoring method according to claim 1, characterized in that: The calculating of the velocity component of each image block to obtain the velocity signal of the respiratory motion of each image block in different directions includes: For each of the image blocks, based on the difference of the image block in different frame images, calculating the velocity components of the image block in the horizontal direction and the vertical direction; Based on the speed component of the image block in each frame of the image, speed signals of the image block in the horizontal direction and the vertical direction are obtained.

4. The non-contact respiratory monitoring method according to claim 3, characterized in that: The calculating of the velocity component of each image block to obtain the velocity signal of the respiratory motion of each image block in different directions also includes: Detrending the velocity signals in the horizontal and vertical directions is performed to eliminate low-frequency noise and baseline drift.

5. The non-contact respiratory monitoring method according to claim 1, characterized in that: The step of fusing the fused respiratory signals of all the image blocks to obtain an overall respiratory signal includes: Selecting a target local respiratory signal according to an average heartbeat interval feature of the fused respiratory signal of the image block; The selected target local respiratory signals are averaged and fused to obtain a one-dimensional overall respiratory signal.

6. A non-contact respiratory monitoring system, characterized in that: The system comprises: An image block acquisition module is used to acquire a video of the chest and abdomen area of ​​the user, and segment each frame of the video to obtain a number of image blocks corresponding to each frame of the video; A speed signal acquisition module, used for calculating the speed component of each image block to obtain the speed signal of the respiratory movement of each image block in different directions; The fused breathing signal acquisition module of the image block is used to perform singular value decomposition on the speed signals in different directions for each image block to obtain corresponding speed features, and perform signal fusion on the speed signals in different directions based on the speed features to obtain a fused breathing signal for each image block; An overall respiratory signal acquisition module, used for fusing the fused respiratory signals of all the image blocks to obtain an overall respiratory signal; A breathing rate acquisition module, used to accumulate and process the overall breathing signal and calculate the heartbeat interval to obtain the user's breathing rate; The fusion respiratory signal acquisition module of the image block includes: a speed matrix acquisition unit, configured to obtain speed matrices of the image block in the horizontal direction and in the vertical direction according to the speed signal, wherein the speed signal is a time series, each time point corresponds to a speed value, the speed signal of the image block includes T time points, the speed matrix in the horizontal direction and the speed matrix in the vertical direction are matrices with T rows, and include T speed values ​​corresponding to the speed signal in the horizontal direction and T speed values ​​corresponding to the speed signal in the vertical direction; A singular value decomposition unit, used to perform singular value decomposition on the velocity matrix to obtain an eigenvector and an eigenvalue corresponding to the image block; The fused breathing signal acquisition unit of the image block is used to select the eigenvector with the largest eigenvalue, and based on the selected eigenvector, perform signal fusion on the speed signals in the horizontal direction and the vertical direction to obtain the fused breathing signal of the image block.

7. A terminal device, characterized in that: The terminal device includes a memory, a processor, and a non-contact respiratory monitoring program stored in the memory and executable on the processor. When the processor executes the non-contact respiratory monitoring program, the steps of the non-contact respiratory monitoring method as described in any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a non-contact respiratory monitoring program, and when the non-contact respiratory monitoring program is executed by the processor, the steps of the non-contact respiratory monitoring method according to any one of claims 1 to 5 are implemented.

Images