Breathing state detection method and device, terminal and storage medium

Abstract

The invention provides a breathing state detection method and device, a terminal and a storage medium, and belongs to the technical field of intelligent detection.The method comprises the steps that a sleep monitoring video stream of a user is obtained, and the sleep monitoring video stream is divided into a plurality of sub-video streams according to a preset time interval; all frames in each sub-video stream are processed, a thoracico-abdominal breathing area and a breathing interference area of the user are determined, and the motion intensity of the thoracico-abdominal breathing area and the motion intensity of the breathing interference area are extracted; marking the effectiveness of the sub-video streams based on the motion intensity of the thoracico-abdominal breathing area and the motion intensity of the breathing interference area in each sub-video stream; and detecting the breathing state of the user based on the sub-video streams marked as effective. According to the method, the sub-video streams are extracted and screened based on the motion intensities of different areas, so that the interference of non-respiratory motion can be effectively removed, and the accuracy and reliability of user sleep respiratory event detection are further improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent detection, in particular to a respiratory state detection method and device, a terminal and a storage medium. BACKGROUND

[0002] With the development of science and technology, various sleep respiratory event detection technologies appear, which can be widely applied to intensive care units, home bedrooms and other scenes, and provide respiratory state detection services for the elderly, infants, critically ill patients, obese people and people with respiratory diseases.

[0003] At present, sleep respiratory event detection technologies can be divided into two categories: contact type and non-contact type. The contact type scheme (such as using a breathing belt, an oximeter or other devices for detection) can provide high-precision data, but it needs to directly contact the user's body, which may cause skin allergy, local compression or signal interruption due to user activity, affecting the stability of detection. The non-contact detection technology has become a research and application hotspot due to its non-invasive characteristics. Among them, the detection schemes based on radar or infrared cameras are widely used. The basic principle is to detect the micro displacement of the chest and abdomen caused by breathing. They all face the core challenge of motion artifacts. Any spontaneous, non-breathing body movement (such as head rotation, limb waving, body twisting, etc.) of the user during sleep will produce interference signals several orders of magnitude stronger than the true breathing signal, causing signal distortion, which easily leads to false respiratory rate calculation and false respiratory pause alarm, resulting in low accuracy and reliability of user sleep respiratory event detection.

[0004] Therefore, the prior art has defects and needs to be improved and developed. SUMMARY

[0005] The technical problem to be solved by the present application is to provide a respiratory state detection method, device, terminal and storage medium to solve the problem of low accuracy and reliability of user sleep respiratory event detection in the prior art.

[0006] The technical solution adopted by the present application to solve the technical problem is as follows: In a first aspect, the present application provides a respiratory state detection method, which comprises: obtaining a sleep monitoring video stream of a user, and dividing the sleep monitoring video stream into a plurality of sub-video streams according to a preset time interval; processing all frames in each sub-video stream to determine a chest and abdominal breathing area and a breathing disturbance area of the user, and extracting the motion intensity of the chest and abdominal breathing area and the motion intensity of the breathing disturbance area; labeling effectiveness of the sub-video stream based on the motion intensity of the chest and abdomen breathing region and the motion intensity of the breathing interference region in each sub-video stream; detecting a breathing state of the user based on the sub-video stream labeled as effective.

[0007] In an implementation, the chest and abdomen breathing region and the breathing interference region of the user are determined by processing all frames in each sub-video stream, including: inputting all frames in each sub-video stream into a pre-trained human parsing depth model to output semantic segmentation results of each frame; determining the chest and abdomen breathing region and the breathing interference region of the user based on the semantic segmentation results of all frames; The human parsing depth model is an M2FP model based on a Transformer architecture, the breathing interference region is a region containing limbs and a head, and the chest and abdomen breathing region is a region containing a chest and an abdomen.

[0008] In an implementation, the chest and abdomen breathing region and the breathing interference region of the user are determined by processing all frames in each sub-video stream, further including: inputting all frames in each sub-video stream into a pre-trained human pose estimation model; performing pose estimation on each frame by the human pose estimation model to output first key points and second key points corresponding to each frame; determining the chest and abdomen breathing region based on the first key points corresponding to all frames and determining the breathing interference region based on the second key points corresponding to all frames; The first key points include left shoulders, right shoulders, left hips and right hips, and the second key points include left elbows, right elbows, left knees, right knees and a head top.

[0009] In an implementation, the motion intensity of the chest and abdomen breathing region and the motion intensity of the breathing interference region are extracted, including: segmenting the chest and abdomen breathing region and the breathing interference region according to a preset size to obtain a plurality of pixel blocks; calculating displacement data of each pixel block between adjacent two frames by using an optical flow algorithm, the displacement data including horizontal direction displacement and vertical direction displacement; calculating and taking a standard deviation of all displacement data of the chest and abdomen breathing region as a corresponding motion intensity, and calculating and taking a standard deviation of all displacement data of the breathing interference region as a corresponding motion intensity.

[0010] In an implementation, the effectiveness of the sub-video stream is labeled based on the motion intensity of the chest and abdomen breathing region and the motion intensity of the breathing interference region in each sub-video stream, including: comparing the motion intensity of the breathing interference region and the motion intensity of the chest and abdominal breathing region in each sub-video stream by a preset multiple; if the motion intensity of the breathing interference region is less than or equal to the preset multiple of the motion intensity of the chest and abdominal breathing region, the sub-video stream is marked as valid; if the motion intensity of the breathing interference region is greater than the preset multiple of the motion intensity of the chest and abdominal breathing region, the sub-video stream is marked as invalid.

[0011] In an embodiment, the validity of the sub-video stream is marked based on the motion intensity of the chest and abdominal breathing region and the motion intensity of the breathing interference region in each sub-video stream, and further comprising: setting a sliding window and sliding the sliding window along the time axis by a preset step; for all sub-video streams covered by the current sliding window, respectively calculating a first baseline of the breathing interference region and a second baseline of the chest and abdominal breathing region in the sliding window, dividing the first baseline by the second baseline and taking the integer part to obtain an adaptive multiple; for each sub-video stream in the current sliding window, multiplying the motion intensity of the chest and abdominal breathing region by the adaptive multiple to obtain a dynamic threshold, comparing the motion intensity of the breathing interference region of the sub-video stream with the dynamic threshold, and marking the validity of the sub-video stream according to the comparison result; wherein the first baseline is the average of the motion intensity of the breathing interference region of all sub-video streams in the sliding window, and the second baseline is the average of the motion intensity of the chest and abdominal breathing region of all sub-video streams in the sliding window.

[0012] In an embodiment, the breathing state of the user is detected based on the sub-video stream marked as valid, comprising: inputting the sub-video stream marked as valid into a pre-trained three-dimensional convolutional neural network; after processing by the three-dimensional convolutional neural network, outputting the breathing state of the user.

[0013] In a second aspect, the embodiments of the present application also provide a breathing state detection device, the device comprising: a video stream division module for obtaining a sleep monitoring video stream of a user and dividing the sleep monitoring video stream into a plurality of sub-video streams according to a preset time interval; a motion intensity extraction module for processing all frames in each sub-video stream, determining the chest and abdominal breathing region and the breathing interference region of the user, and extracting the motion intensity of the chest and abdominal breathing region and the motion intensity of the breathing interference region; a marking module, configured to mark validity of the sub-video stream based on the motion intensity of the chest and abdomen breathing region and the motion intensity of the breathing interference region in each sub-video stream; a detection module, configured to detect the breathing state of the user based on the sub-video stream marked as valid.

[0014] In a third aspect, an embodiment of the present application further provides a terminal, which comprises a memory, a processor, and a breathing state detection program stored in the memory and capable of running on the processor, and the breathing state detection program, when executed by the processor, implements the steps of the breathing state detection method.

[0015] In a fourth aspect, an embodiment of the present application further provides a computer readable storage medium, which stores a breathing state detection program, and the breathing state detection program can be executed to implement the steps of the breathing state detection method.

[0016] The present application has the following beneficial effects: the present application divides a sleep monitoring video stream of a user into a plurality of sub-video streams according to a preset time interval, processes all frames in each sub-video stream, determines a chest and abdomen breathing region and a breathing interference region of the user, extracts motion intensity of the chest and abdomen breathing region and motion intensity of the breathing interference region, marks validity of the sub-video stream based on the motion intensity of the chest and abdomen breathing region and the motion intensity of the breathing interference region in each sub-video stream, and detects the breathing state of the user based on the sub-video stream marked as valid. The present application extracts and screens the sub-video stream based on the motion intensity of different regions, can effectively remove interference of non-breathing motion, and thus improves detection accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 is a flow chart of a preferable embodiment of the breathing state detection method in the present application.

[0018] Figure 2 is a schematic diagram of a breathing state detection processing flow in the present application.

[0019] Figure 3 is a structural schematic diagram of a preferable embodiment of a breathing state detection device in the present application.

[0020] Figure 4 is a principle block diagram of a terminal of the present application. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical scheme and advantages of the present application clearer and more explicit, the present application will be further described in detail below with reference to the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and not to limit the present application.

[0022] With the development of technology, various sleep respiratory event detection technologies appear, which can be widely applied to intensive care units, home bedrooms and other scenes, and provide respiratory state detection services for the elderly, infants, critically ill patients, obese people and people with respiratory diseases.

[0023] Currently, sleep respiratory event detection technologies can be divided into two categories: contact type and non-contact type. Although the contact type scheme (such as a breathing belt, an oximeter, etc.) can provide high-precision data, it needs to directly contact the user's body, which may cause skin allergy, local compression or signal interruption due to user activity, affecting the detection stability. The non-contact detection technology has become a research and application hotspot due to its non-invasive characteristics. Among them, the detection schemes based on radar or infrared cameras are widely used. The basic principle is to detect the micro displacement of the chest and abdomen caused by breathing. They all face the core challenge of motion artifacts. Any spontaneous, non-breathing body movement (such as head rotation, limb waving, body twisting, etc.) of the user during sleep will produce interference signals several orders of magnitude stronger than the true breathing signal, causing signal distortion, which easily leads to false respiratory rate calculation and false respiratory pause alarm, resulting in low accuracy and reliability of the user's sleep respiratory event detection.

[0024] In view of the above defects of the prior art, the present application provides a respiratory state detection method, device, terminal and storage medium. The method comprises: acquiring a sleep monitoring video stream of a user, and dividing the sleep monitoring video stream into a plurality of sub-video streams according to a preset time interval; processing all frames in each sub-video stream to determine a chest and abdominal breathing area and a breathing interference area of the user, and extracting the motion intensity of the chest and abdominal breathing area and the motion intensity of the breathing interference area; based on the motion intensity of the chest and abdominal breathing area and the motion intensity of the breathing interference area in each sub-video stream, marking the effectiveness of the sub-video stream; based on the sub-video stream marked as effective, detecting the respiratory state of the user. The present application can effectively remove the interference of non-breathing motion by extracting and based on the motion intensity of different areas to screen the sub-video stream, thereby improving the accuracy of detection.

[0025] It should be noted that the collection and processing of the sleep monitoring video stream involved in the present application have been authorized by the user or his legal guardian, and strictly comply with relevant laws, regulations and standards.

[0026] Please refer to Figure 1 The respiratory state detection method described in the embodiments of the present application comprises the following steps: Step S100, acquiring a sleep monitoring video stream of a user, and dividing the sleep monitoring video stream into a plurality of sub-video streams according to a preset time interval.

[0027] Specifically, a sleep monitoring video stream of a user is collected by using a video collection device, and the collected sleep monitoring video stream is divided into sub-video streams, so as to subsequently judge whether each sub-video stream is a valid video stream in units of sub-video streams, and only the valid video streams are processed to judge the breathing state of the user. In this way, the video stream contaminated by motion artifacts can be removed, and it is ensured that the processed is a reliable breathing motion signal, which can significantly improve the accuracy and reliability of the user's sleep respiratory event detection. The video collection device can be a camera or an RGB camera. The user can include the elderly, infants, critically ill patients, obese people and people with respiratory diseases.

[0028] Please refer to Figure 1 The breathing state detection method described in the embodiments of the present application further includes the following steps: Step S200, processing all frames in each sub-video stream to determine the chest and abdominal breathing region and the breathing interference region of the user, and extracting the motion intensity of the chest and abdominal breathing region and the motion intensity of the breathing interference region.

[0029] Specifically, the chest and abdominal breathing region is a region reflecting breathing motion, including the chest and abdomen, and the breathing interference region is a limb region generating motion interference artifacts, including the limbs and head. The present application distinguishes the region generating motion interference artifacts, and determines the validity of the sub-video stream based on the motion intensity of the chest and abdominal breathing region and the motion intensity of the breathing interference region. Subsequently, only the valid sub-video stream is processed to detect the breathing state of the user, and the influence of the artifacts is removed, which can effectively improve the accuracy and reliability of the user's sleep respiratory event detection.

[0030] In one implementation, processing all frames in each sub-video stream to determine the chest and abdominal breathing region and the breathing interference region of the user, including: inputting all frames in each sub-video stream into a pre-trained human body analysis depth model to output semantic segmentation results of each frame; determining the chest and abdominal breathing region and the breathing interference region of the user based on the semantic segmentation results of all frames; The human body analysis depth model is a M2FP model based on a Transformer architecture.

[0031] Specifically, for each pixel coordinate, the semantic label distribution of the pixel coordinate in all frames of the sub-video stream is counted, and the label with the highest frequency of occurrence is selected as the final classification label of the pixel coordinate; the pixels with the final classification label of the chest and the abdomen constitute a chest and abdominal breathing area, and the pixels with the final classification label of the head, the left arm, the right arm, the left leg or the right leg constitute a breathing interference area. The M2FP (Multi-scale Multi-hierarchical Feature Pyramid) model can perform multi-scale feature extraction and fusion on the input image, can effectively capture global and local details, and then obtain an accurate semantic segmentation result.

[0032] In an implementation manner, the chest and abdominal breathing area and the breathing interference area of the user are determined by processing all frames in each sub-video stream, and the method further includes: inputting all frames in each sub-video stream into a pre-trained human pose estimation model, performing pose estimation on each frame by the human pose estimation model, and outputting first key points and second key points corresponding to each frame; determining the chest and abdominal breathing area based on the first key points corresponding to all frames, and determining the breathing interference area based on the second key points corresponding to all frames; wherein the first key points include a left shoulder, a right shoulder, a left hip and a right hip, and the second key points include a left elbow, a right elbow, a left knee, a right knee and a head top.

[0033] Specifically, the present application also supports distinguishing different areas by using human pose estimation technology. The human pose estimation model can be an OpenPose (Real-time Multi-Person 2D Pose Estimation using PartAffinity Fields) model or an HRNet (High-Resolution Network) model.

[0034] In an implementation manner, the motion intensity of the chest and abdominal breathing area and the motion intensity of the breathing interference area are extracted, including: segmenting the chest and abdominal breathing area and the breathing interference area according to a preset size to obtain a plurality of pixel blocks; calculating displacement data of each pixel block between adjacent two frames by using an optical flow algorithm, the displacement data including horizontal displacement and vertical displacement; calculating and taking the standard deviation of all displacement data of the chest and abdominal breathing area as the corresponding motion intensity, and calculating and taking the standard deviation of all displacement data of the breathing interference area as the corresponding motion intensity.

[0035] Specifically, the preset size can be 20x20 pixels or 15x15 pixels. The optical flow algorithm can adopt a PixFlow algorithm. The displacement data of each pixel block between two adjacent frames can be regarded as a motion signal, and the standard deviation of the motion signal can be calculated to obtain the corresponding motion intensity.

[0036] Please refer to Figure 1 The breathing state detection method described in the embodiments of the present application further includes the following steps: In step S300, the effectiveness of each sub-video stream is marked based on the motion intensity of the chest and abdominal breathing region and the motion intensity of the breathing interference region in the sub-video stream.

[0037] Specifically, the motion intensity of the breathing interference region and the motion intensity of the chest and abdominal breathing region in each sub-video stream are compared by a preset multiple. If the motion intensity of the breathing interference region is less than or equal to the preset multiple of the motion intensity of the chest and abdominal breathing region, the sub-video stream is marked as valid. If the motion intensity of the breathing interference region is greater than the preset multiple of the motion intensity of the chest and abdominal breathing region, the sub-video stream is marked as invalid. When the motion intensity of the breathing interference region is less than or equal to the preset multiple of the chest and abdominal breathing region, it indicates that the breathing signal is not covered by the interference, and the sub-video stream at this time can be marked as valid. When the motion intensity of the breathing interference region is higher than the preset multiple of the chest and abdominal breathing region, it indicates that there is a severe body motion artifact, and the sub-video stream needs to be marked as invalid. The preset multiple can be 5. In this way, the input data quality of the respiratory time analysis can be effectively improved, and misjudgment caused by artifacts can be avoided.

[0038] In addition to the above manner, based on the motion intensity of the chest and abdominal breathing region and the motion intensity of the breathing interference region in each sub-video stream, marking the effectiveness of the sub-video stream further includes: setting a sliding window, and sliding the sliding window along the time axis at a preset step length, the size of the sliding window being equal to the preset step length, so that the windows continuously cover all sub-video streams without overlapping between adjacent windows when the window slides along the time axis at the step length. For all sub-video streams covered by the current sliding window, the first baseline of the breathing interference region and the second baseline of the chest and abdominal breathing region in the sliding window are calculated respectively, the first baseline is divided by the second baseline and rounded to obtain an adaptive multiple; for each sub-video stream in the current sliding window, the motion intensity of the chest and abdominal breathing region is multiplied by the adaptive multiple to obtain a dynamic threshold, the motion intensity of the breathing interference region of the sub-video stream is compared with the dynamic threshold, and the effectiveness of the sub-video stream is marked according to the comparison result. If the motion intensity of the breathing interference region is less than or equal to the dynamic threshold, the sub-video stream is marked as valid; if the motion intensity of the breathing interference region is greater than the dynamic threshold, the sub-video stream is marked as invalid. The first baseline is the average of the motion intensity of the breathing interference region of all sub-video streams in the sliding window, and the second baseline is the average of the motion intensity of the chest and abdominal breathing region of all sub-video streams in the sliding window. This marking method using the adaptive multiple significantly reduces the false judgment caused by accidental large body movements, and effectively improves the accuracy of the marking.

[0039] Referring to Figure 1 The breathing state detection method according to the embodiment of the present application further includes the following steps: Step S400, based on the sub-video stream marked as valid, detecting the breathing state of the user.

[0040] Specifically, the sub-video stream marked as valid can be input into a pre-trained three-dimensional convolutional neural network; after being processed by the three-dimensional convolutional neural network (3DCNN), the breathing state of the user is output. The three-dimensional convolutional neural network includes three convolutional layers. The breathing state includes normal breathing, low ventilation and apnea.

[0041] In addition, the motion signals of the chest and abdominal breathing regions and the breathing interference region can be filtered (such as band-pass filtering), and time domain features, frequency domain features and time-frequency domain features in the motion signals can be extracted. The motion signals are generated based on the horizontal and vertical displacement of the pixel blocks in each region between adjacent frames calculated by the PixFlow algorithm. The time domain features can be directly extracted from the motion signals, including the mean value (reflecting the average level of motion), variance (reflecting the fluctuation degree of motion), root mean square (reflecting the motion energy), etc.; the frequency domain features are extracted after the motion signals are converted from the time domain to the frequency domain by Fourier transform, including the frequency spectrum (reflecting the energy distribution of different frequencies), the main frequency (the frequency with the highest energy, corresponding to the main frequency of breathing or body movement), etc.; the time-frequency domain features combine the information of two dimensions of time and frequency, which can be extracted by wavelet transform, including the time-frequency energy distribution (reflecting the frequency energy change at different time points), etc. The above time domain features, frequency domain features and time-frequency domain features are input into a pre-trained support vector machine (SVM) classifier, and the breathing state corresponding to the current sub-video stream is output after processing. The support vector machine classifier uses the same features of the historical breathing state samples as the training data.

[0042] In the prior art, the contact scheme also includes polysomnography (PSG), which is the gold standard for clinical diagnosis. It records electroencephalogram, electrocardiogram, electromyogram, respiratory airflow, chest and abdominal movement, blood oxygen saturation and other physiological signals for comprehensive diagnosis, and has powerful functions. However, polysomnography needs to be performed in a professional sleep laboratory, and the equipment is numerous and the cost is high, which will greatly bind and discomfort the user, seriously interfere with their normal sleep, and is not suitable for regular and long-term home monitoring. In addition, there is also a contact scheme using a mattress, which also has the problem that the signal is easily distorted by the user's turning over and limb movement, resulting in a high false positive rate. In the scheme of non-contact detection equipment, there is also a detection technology using a microphone, which detects by capturing the user's breathing sound, but the signal-to-noise ratio is very low, and it is easily disturbed by environmental noise (such as air conditioner, noise outside the window), and it is difficult to detect silent apnea events.

[0043] The breathing state detection processing flow of the present application is as follows Figure 2As shown. When the sleep monitoring video stream is acquired, the video frames are processed in real time to determine the chest and abdominal breathing area and the limb area of the user, and the motion intensity between the two is compared to detect whether the user has a large amplitude motion, thereby screening out effective sub-video streams. Subsequently, the chest and abdominal breathing motion law under different breathing events is combined with a three-dimensional convolutional neural network to realize high-precision detection of normal breathing, apnea, low ventilation and other events. Through this "segmentation, then discrimination, and then analysis" cascade strategy, motion artifacts contaminated segments can be actively identified and removed, ensuring that the detection and analysis are based on reliable breathing motion signals, significantly improving the accuracy and reliability of user sleep breathing event detection.

[0044] In an embodiment, as shown in Figure 3 Based on the above-mentioned respiratory state detection method, the present application also correspondingly provides a respiratory state detection device, which comprises: A video stream division module 100 is configured to acquire a sleep monitoring video stream of a user, and divide the sleep monitoring video stream into a plurality of sub-video streams according to a preset time interval; A motion intensity extraction module 200 is configured to process all frames in each sub-video stream to determine a chest and abdominal breathing area and a breathing disturbance area of the user, and extract the motion intensity of the chest and abdominal breathing area and the motion intensity of the breathing disturbance area; A marking module 300 is configured to mark the effectiveness of the sub-video stream based on the motion intensity of the chest and abdominal breathing area and the motion intensity of the breathing disturbance area in each sub-video stream; A detection module 400 is configured to detect the respiratory state of the user based on the sub-video stream marked as effective.

[0045] In an embodiment, the device further comprises: A semantic segmentation unit is configured to input all frames in each sub-video stream into a pre-trained human parsing depth model, and output the semantic segmentation result of each frame, wherein the human parsing depth model is an M2FP model based on a Transformer architecture; A first area division unit is configured to determine the chest and abdominal breathing area and the breathing disturbance area of the user based on the semantic segmentation result of all frames, wherein the breathing disturbance area is an area containing limbs and head, and the chest and abdominal breathing area is an area containing chest and abdomen.

[0046] In an embodiment, the device further comprises: a posture estimation unit, configured to input all frames in each sub-video stream into a pre-trained human posture estimation model, perform posture estimation on each frame by the human posture estimation model, and output first key points and second key points corresponding to each frame, wherein the first key points include left shoulders, right shoulders, left hips and right hips, and the second key points include left elbows, right elbows, left knees, right knees and head tops; a second region division unit, configured to determine a chest and abdomen breathing region based on the first key points corresponding to all frames, and determine a breathing interference region based on the second key points corresponding to all frames.

[0047] In an embodiment, the device further comprises: a region segmentation unit, configured to segment the chest and abdomen breathing region and the breathing interference region according to a preset size to obtain a plurality of pixel blocks; a displacement calculation unit, configured to calculate displacement data of each pixel block between adjacent two frames by using an optical flow algorithm, wherein the displacement data includes horizontal direction displacement and vertical direction displacement; a parameter determination unit, configured to calculate a standard deviation of all displacement data of the chest and abdomen breathing region as a corresponding motion intensity, and calculate a standard deviation of all displacement data of the breathing interference region as a corresponding motion intensity.

[0048] In an embodiment, the marking module comprises: a comparison unit, configured to compare a motion intensity of the breathing interference region and a preset multiple of a motion intensity of the chest and abdomen breathing region in each sub-video stream; a first marking unit, configured to mark the sub-video stream as valid if the motion intensity of the breathing interference region is less than or equal to the preset multiple of the motion intensity of the chest and abdomen breathing region; a second marking unit, configured to mark the sub-video stream as invalid if the motion intensity of the breathing interference region is greater than the preset multiple of the motion intensity of the chest and abdomen breathing region.

[0049] In an embodiment, the marking module further comprises: a window setting unit, configured to set a sliding window and slide the sliding window along a time axis by a preset step size; an adaptive multiple calculation unit, configured to calculate a first baseline of the breathing interference region and a second baseline of the chest and abdomen breathing region in a sliding window for all sub-video streams covered by the sliding window, divide the first baseline by the second baseline and take an integer to obtain an adaptive multiple, wherein the first baseline is an average value of motion intensities of the breathing interference regions of all sub-video streams in the sliding window, and the second baseline is an average value of motion intensities of the chest and abdomen breathing regions of all sub-video streams in the sliding window; The third marking unit is configured to, for each sub-video stream in the current sliding window, multiply the motion intensity of the chest and abdomen breathing region and the adaptive multiple to obtain a dynamic threshold, compare the motion intensity of the breathing interference region of the sub-video stream with the dynamic threshold, and mark the validity of the sub-video stream according to the comparison result.

[0050] In an embodiment, the detection module comprises: The video input unit is configured to input the sub-video stream marked as valid into a pre-trained three-dimensional convolutional neural network. The state determination unit is configured to output the breathing state of the user through the three-dimensional convolutional neural network.

[0051] Based on the above-mentioned embodiments, the present application further provides a terminal, a structure diagram of which can be shown in Figure 4 The terminal comprises a processor, a memory, a network interface and a display screen connected through a device bus. The processor of the terminal is configured to provide computing and control capabilities. The memory of the terminal comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating device and a breathing state detection program. The internal memory provides an environment for the operation of the operating device and the breathing state detection program in the non-volatile storage medium. The network interface of the terminal is configured to communicate with external terminals through network connection. The breathing state detection program, when executed by the processor, implements the steps of any one of the above-mentioned breathing state detection methods. The display screen of the terminal can be a liquid crystal display screen or an electronic ink display screen.

[0052] Those skilled in the art can understand that Figure 4 The structure diagram shown in the above-mentioned embodiments is only a schematic diagram of part of the structure related to the present application scheme, and does not constitute a limitation on the terminal to which the present application scheme is applied. Specifically, the terminal can comprise more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0053] In an embodiment, a terminal is provided, which comprises a memory, a processor and a breathing state detection program stored in the memory and executable on the processor. When the breathing state detection program is executed by the processor, the steps of any one of the breathing state detection methods provided by the present application embodiment are implemented.

[0054] The present application embodiment further provides a computer readable storage medium, which stores a breathing state detection program. When the breathing state detection program is executed by a processor, the steps of any one of the breathing state detection methods provided by the present application embodiment are implemented.

[0055] It should be understood that the sequence of the steps in the above-described embodiments does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0056] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the above-mentioned device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be realized in the form of hardware or software function unit. In addition, the specific name of each functional unit and module is only for easy distinction, and does not limit the protection scope of the present application. The specific working process of the unit and module in the above-mentioned device can be referred to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0057] In the above-described embodiments, the description of each embodiment has its own emphasis, and the parts not described or recorded in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0058] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0059] In the embodiments provided by the present application, it should be understood that the disclosed device / terminal equipment and method can be implemented by other ways. For example, the above-described device / terminal equipment embodiments are only schematic, for example, the division of the above-mentioned modules or units is only a logical function division, and actual implementation can be in another division manner, for example, a plurality of units or components can be combined or integrated into another device, or some features can be ignored or not executed.

[0060] The above-described embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit the present application; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand; it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements are not the essence of the corresponding technical solutions, which deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A method for detecting respiratory status, characterized in that, The method includes: Acquire the user's sleep monitoring video stream and divide the sleep monitoring video stream into several sub-video streams according to a preset time interval; All frames within each sub-video stream are processed to determine the user's chest and abdominal breathing region and breathing interference region, and the motion intensity of the chest and abdominal breathing region and the motion intensity of the breathing interference region are extracted. The validity of each sub-video stream is marked based on the motion intensity of the chest and abdominal breathing region and the motion intensity of the breathing interference region within each sub-video stream; The user's breathing status is detected based on the sub-video streams that are marked as valid.

2. The respiratory state detection method according to claim 1, characterized in that, Process all frames within each sub-video stream to determine the user's chest and abdominal breathing region and breathing interference region, including: Input all frames in each sub-video stream into a pre-trained human parsing depth model and output the semantic segmentation result for each frame; Based on the semantic segmentation results of all frames, the user's chest and abdominal breathing region and breathing interference region are determined. The human body resolution depth model is an M2FP model based on the Transformer architecture. The chest and abdomen breathing region is the region that reflects respiratory movements, including the chest and abdomen. The breathing interference region is the limb region that generates motion interference artifacts, including the limbs and head.

3. The respiratory state detection method according to claim 1, characterized in that, Processing all frames within each sub-video stream to determine the user's chest and abdominal breathing region and breathing interference region also includes: All frames in each sub-video stream are input into a pre-trained human pose estimation model. The human pose estimation model performs pose estimation on each frame and outputs the first and second key points corresponding to each frame. The chest and abdominal breathing region is determined based on the first key point corresponding to all frames, and the breathing interference region is determined based on the second key point corresponding to all frames. The first key point includes the left shoulder, right shoulder, left hip, and right hip, while the second key point includes the left elbow, right elbow, left knee, right knee, and top of the head.

4. The respiratory state detection method according to claim 1, characterized in that, Extracting the motion intensity of the chest and abdominal respiratory region and the motion intensity of the respiratory disturbance region includes: The chest and abdominal breathing region and the breathing interference region are divided into several pixel blocks according to a preset size; The displacement data of each pixel block between two adjacent frames is calculated using an optical flow algorithm. The displacement data includes horizontal displacement and vertical displacement. The standard deviation of all displacement data in the chest and abdominal breathing region is calculated and used as the corresponding exercise intensity, and the standard deviation of all displacement data in the breathing interference region is calculated and used as the corresponding exercise intensity.

5. The respiratory state detection method according to claim 1, characterized in that, The validity of each sub-video stream is marked based on the motion intensity of the chest and abdominal breathing region and the motion intensity of the breathing interference region within each sub-video stream, including: The motion intensity of the breathing interference region and the motion intensity of the chest and abdominal breathing region within each sub-video stream are compared by a preset multiple. If the motion intensity of the breathing interference area is less than or equal to a preset multiple of the motion intensity of the chest and abdominal breathing area, then the sub-video stream is marked as valid. If the motion intensity of the breathing interference area is greater than a preset multiple of the motion intensity of the chest and abdominal breathing area, then the sub-video stream is marked as invalid.

6. The respiratory state detection method according to claim 1, characterized in that, The validity of each sub-video stream is marked based on the motion intensity of the chest and abdominal breathing region and the motion intensity of the breathing interference region within each sub-video stream, and also includes: Set a sliding window and slide the sliding window along the time axis at a preset step size; For all sub-video streams covered by the current sliding window, calculate the first baseline of the breathing interference region and the second baseline of the chest and abdominal breathing region within the sliding window respectively. Divide the first baseline by the second baseline and round down to obtain the adaptive multiplier. For each sub-video stream within the current sliding window, the motion intensity of the chest and abdominal breathing region is multiplied by the adaptive factor to obtain a dynamic threshold. The motion intensity of the breathing interference region of the sub-video stream is compared with the dynamic threshold, and the validity of the sub-video stream is marked based on the comparison result. Wherein, the first baseline is the average motion intensity of the breathing interference region of all sub-video streams within the sliding window, and the second baseline is the average motion intensity of the chest and abdominal breathing region of all sub-video streams within the sliding window.

7. The respiratory state detection method according to claim 1, characterized in that, Based on the sub-video streams marked as valid, the user's breathing state is detected, including: Input the labeled valid sub-video streams into a pre-trained 3D convolutional neural network; The user's breathing status is output after processing by the three-dimensional convolutional neural network.

8. A respiratory status detection device, characterized in that, include: The video stream segmentation module is used to acquire the user's sleep monitoring video stream and divide the sleep monitoring video stream into several sub-video streams according to a preset time interval. The motion intensity extraction module is used to process all frames in each sub-video stream, determine the user's chest and abdominal breathing region and breathing interference region, and extract the motion intensity of the chest and abdominal breathing region and the motion intensity of the breathing interference region. A marking module is used to mark the validity of a sub-video stream based on the motion intensity of the chest and abdominal breathing region and the motion intensity of the breathing interference region within each sub-video stream; The detection module is used to detect the user's breathing status based on sub-video streams that are marked as valid.

9. A terminal, characterized in that, The terminal includes: a memory, a processor, and a respiratory state detection program stored in the memory and executable on the processor. When the respiratory state detection program is executed by the processor, it implements the steps of the respiratory state detection method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a respiratory state detection program, which, when executed by a processor, implements the steps of the respiratory state detection method as described in any one of claims 1-7.

Images