A respiratory state detection method, device, terminal and storage medium

Abstract

This invention provides a method, device, terminal, and storage medium for detecting respiratory states, belonging to the field of intelligent detection technology. The method includes: acquiring a user's sleep monitoring video stream and dividing the sleep monitoring video stream into several sub-video streams according to a preset time interval; processing all frames in each sub-video stream to determine the user's chest and abdominal breathing region and breathing interference region, and extracting the motion intensity of the chest and abdominal breathing region and the motion intensity of the breathing interference region; marking the validity of the sub-video stream based on the motion intensity of the chest and abdominal breathing region and the motion intensity of the breathing interference region in each sub-video stream; and detecting the user's respiratory state based on the marked valid sub-video streams. This invention effectively removes interference from non-breathing movements by extracting and filtering sub-video streams based on the motion intensity of different regions, thereby improving the accuracy and reliability of user sleep breathing event detection.

Description

Technical Field

[0001] This invention relates to the field of intelligent detection technology, and in particular to a method, device, terminal, and storage medium for detecting respiratory status. Background Technology

[0002] With the development of technology, various sleep apnea event detection technologies have emerged, which can be widely used in multiple scenarios such as intensive care units and home bedrooms to provide respiratory status monitoring services for the elderly, infants, critically ill patients, obese people, and people with respiratory diseases.

[0003] Currently, sleep apnea event detection technologies can be broadly categorized into contact and non-contact methods. Contact-based methods (such as those using breathing belts or pulse oximeters) provide high-precision data, but require direct contact with the user's body, potentially causing skin allergies, localized pressure, or signal interruption due to user movement, affecting detection stability. Non-contact detection technologies, due to their non-invasive nature, have become a hot topic in research and application. Among these, radar or infrared camera-based detection schemes are widely used. Their basic principle is to detect minute displacements in the chest and abdomen caused by breathing. These methods share the core challenge of motion artifacts. Any spontaneous, non-respiratory body movement during sleep (such as head turning, limb waving, or body twisting) generates interference signals several orders of magnitude stronger than the actual breathing signal, causing signal distortion and easily leading to incorrect respiratory rate calculations and false apnea alarms, resulting in low accuracy and reliability of sleep apnea event detection.

[0004] Therefore, existing technologies have shortcomings and need to be improved and developed. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a breathing state detection method, device, terminal and storage medium to address the above-mentioned deficiencies of the prior art, and to solve the problem of low accuracy and reliability of user sleep breathing event detection in the prior art.

[0006] The technical solution adopted by this invention to solve the technical problem is as follows:

[0007] In a first aspect, embodiments of the present invention provide a method for detecting respiratory status, the method comprising:

[0008] Acquire the user's sleep monitoring video stream and divide the sleep monitoring video stream into several sub-video streams according to a preset time interval;

[0009] All frames within each sub-video stream are processed to determine the user's chest and abdominal breathing region and breathing interference region, and the motion intensity of the chest and abdominal breathing region and the motion intensity of the breathing interference region are extracted.

[0010] The validity of each sub-video stream is marked based on the motion intensity of the chest and abdominal breathing region and the motion intensity of the breathing interference region within each sub-video stream;

[0011] The user's breathing status is detected based on the sub-video streams that are marked as valid.

[0012] In one implementation, all frames within each sub-video stream are processed to determine the user's chest and abdominal breathing region and breathing interference region, including:

[0013] Input all frames in each sub-video stream into a pre-trained human parsing depth model and output the semantic segmentation result for each frame;

[0014] Based on the semantic segmentation results of all frames, the user's chest and abdominal breathing region and breathing interference region are determined.

[0015] The human body resolution depth model is an M2FP model based on the Transformer architecture, the breathing interference region is the region including the limbs and head, and the chest and abdomen breathing region is the region including the chest and abdomen.

[0016] In one implementation, processing all frames within each sub-video stream to determine the user's chest and abdominal breathing region and breathing interference region further includes:

[0017] Input all frames within each sub-video stream into the pre-trained human pose estimation model;

[0018] The human pose estimation model estimates the pose of each frame and outputs the first and second key points for each frame.

[0019] The chest and abdominal breathing region is determined based on the first key point corresponding to all frames, and the breathing interference region is determined based on the second key point corresponding to all frames.

[0020] The first key point includes the left shoulder, right shoulder, left hip, and right hip, while the second key point includes the left elbow, right elbow, left knee, right knee, and top of the head.

[0021] In one embodiment, extracting the motion intensity of the chest and abdominal respiratory region and the motion intensity of the respiratory disturbance region includes:

[0022] The chest and abdominal breathing region and the breathing interference region are divided into several pixel blocks according to a preset size;

[0023] The displacement data of each pixel block between two adjacent frames is calculated using an optical flow algorithm. The displacement data includes horizontal displacement and vertical displacement.

[0024] The standard deviation of all displacement data in the chest and abdominal breathing region is calculated and used as the corresponding exercise intensity, and the standard deviation of all displacement data in the breathing interference region is calculated and used as the corresponding exercise intensity.

[0025] In one implementation, the validity of a sub-video stream is marked based on the motion intensity of the chest and abdominal breathing region and the motion intensity of the breathing interference region within each sub-video stream, including:

[0026] The motion intensity of the breathing interference region and the motion intensity of the chest and abdominal breathing region within each sub-video stream are compared by a preset multiple.

[0027] If the motion intensity of the breathing interference area is less than or equal to a preset multiple of the motion intensity of the chest and abdominal breathing area, then the sub-video stream is marked as valid.

[0028] If the motion intensity of the breathing interference area is greater than a preset multiple of the motion intensity of the chest and abdominal breathing area, then the sub-video stream is marked as invalid.

[0029] In one implementation, the validity of a sub-video stream is marked based on the motion intensity of the chest and abdominal breathing region and the motion intensity of the breathing interference region within each sub-video stream, further including:

[0030] Set a sliding window and slide the sliding window along the time axis at a preset step size;

[0031] For all sub-video streams covered by the current sliding window, calculate the first baseline of the breathing interference region and the second baseline of the chest and abdominal breathing region within the sliding window respectively. Divide the first baseline by the second baseline and round down to obtain the adaptive multiplier.

[0032] For each sub-video stream within the current sliding window, the motion intensity of the chest and abdominal breathing region is multiplied by the adaptive factor to obtain a dynamic threshold. The motion intensity of the breathing interference region of the sub-video stream is compared with the dynamic threshold, and the validity of the sub-video stream is marked based on the comparison result.

[0033] Wherein, the first baseline is the average motion intensity of the breathing interference region of all sub-video streams within the sliding window, and the second baseline is the average motion intensity of the chest and abdominal breathing region of all sub-video streams within the sliding window.

[0034] In one implementation, detecting the user's breathing state based on a sub-video stream marked as valid includes:

[0035] Input the labeled valid sub-video streams into a pre-trained 3D convolutional neural network;

[0036] The user's breathing status is output after processing by the three-dimensional convolutional neural network.

[0037] Secondly, embodiments of the present invention also provide a respiratory state detection device, the device comprising:

[0038] The video stream segmentation module is used to acquire the user's sleep monitoring video stream and divide the sleep monitoring video stream into several sub-video streams according to a preset time interval.

[0039] The motion intensity extraction module is used to process all frames in each sub-video stream, determine the user's chest and abdominal breathing region and breathing interference region, and extract the motion intensity of the chest and abdominal breathing region and the motion intensity of the breathing interference region.

[0040] A marking module is used to mark the validity of a sub-video stream based on the motion intensity of the chest and abdominal breathing region and the motion intensity of the breathing interference region within each sub-video stream;

[0041] The detection module is used to detect the user's breathing status based on sub-video streams that are marked as valid.

[0042] Thirdly, embodiments of the present invention also provide a terminal, the terminal comprising: a memory, a processor, and a respiratory state detection program stored in the memory and executable on the processor, wherein the respiratory state detection program, when executed by the processor, implements the steps of the respiratory state detection method as described above.

[0043] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a respiratory state detection program, the respiratory state detection program being executable to implement the steps of the respiratory state detection method as described above.

[0044] The beneficial effects of this invention are as follows: This invention acquires a user's sleep monitoring video stream and divides the video stream into several sub-video streams according to a preset time interval; it processes all frames within each sub-video stream to determine the user's chest and abdominal breathing region and breathing interference region, and extracts the motion intensity of the chest and abdominal breathing region and the motion intensity of the breathing interference region; based on the motion intensity of the chest and abdominal breathing region and the motion intensity of the breathing interference region within each sub-video stream, the validity of the sub-video stream is marked; based on the marked valid sub-video streams, the user's breathing state is detected. This invention, by extracting and filtering sub-video streams based on the motion intensity of different regions, can effectively remove interference from non-breathing motions, thereby improving the accuracy of detection. Attached Figure Description

[0045] Figure 1This is a flowchart of a preferred embodiment of the respiratory state detection method of the present invention.

[0046] Figure 2 This is a schematic diagram of the respiratory state detection and processing flow in this invention.

[0047] Figure 3 This is a schematic diagram of a preferred embodiment of the respiratory state detection device of the present invention.

[0048] Figure 4 This is a block diagram of the terminal principle of the present invention. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0050] With the development of technology, various sleep apnea event detection technologies have emerged, which can be widely used in multiple scenarios such as intensive care units and home bedrooms to provide respiratory status monitoring services for the elderly, infants, critically ill patients, obese people, and people with respiratory diseases.

[0051] Currently, sleep apnea event detection technologies can be broadly categorized into contact and non-contact methods. While contact-based solutions (such as breathing belts and pulse oximeters) provide high-precision data, they require direct contact with the user's body, potentially causing skin allergies, localized pressure, or signal interruption due to user movement, affecting detection stability. Non-contact detection technologies, due to their non-invasive nature, have become a hot topic in research and application. Among these, radar or infrared camera-based detection solutions are widely used. Their basic principle is to detect minute displacements in the chest and abdomen caused by breathing. These methods share the core challenge of motion artifacts. Any spontaneous, non-respiratory body movement during sleep (such as head turning, limb waving, or body twisting) generates interference signals several orders of magnitude stronger than the actual breathing signal, causing signal distortion and easily leading to incorrect respiratory rate calculations and false apnea alarms, resulting in low accuracy and reliability of sleep apnea event detection.

[0052] To address the aforementioned deficiencies in existing technologies, this invention provides a method, apparatus, terminal, and storage medium for detecting respiratory status. The method includes: acquiring a user's sleep monitoring video stream and dividing the sleep monitoring video stream into several sub-video streams according to a preset time interval; processing all frames within each sub-video stream to determine the user's chest and abdominal breathing region and breathing interference region, and extracting the motion intensity of the chest and abdominal breathing region and the motion intensity of the breathing interference region; marking the validity of each sub-video stream based on the motion intensity of the chest and abdominal breathing region and the motion intensity of the breathing interference region; and detecting the user's respiratory status based on the marked valid sub-video streams. This invention, by extracting and filtering sub-video streams based on the motion intensity of different regions, can effectively remove interference from non-breathing movements, thereby improving the accuracy of detection.

[0053] It should be noted that the collection and processing of the sleep monitoring video streams involved in this application have been authorized by the user or their legal guardian and have strictly complied with relevant laws, regulations and standards.

[0054] Please see Figure 1 The respiratory state detection method described in this embodiment of the invention includes the following steps:

[0055] Step S100: Obtain the user's sleep monitoring video stream and divide the sleep monitoring video stream into several sub-video streams according to a preset time interval.

[0056] Specifically, video capture devices are used to acquire sleep monitoring video streams from users. These streams are then divided into sub-streams for subsequent evaluation to determine their validity. Only valid sub-streams are processed to assess the user's breathing status. This method eliminates video streams contaminated by motion artifacts, ensuring the processing of reliable respiratory motion signals and significantly improving the accuracy and reliability of sleep apnea event detection. The video capture device can be a webcam or an RGB camera. Users may include the elderly, infants, seriously ill patients, obese individuals, and those with respiratory diseases.

[0057] Please see Figure 1 The respiratory state detection method described in this embodiment of the invention further includes the following steps:

[0058] Step S200: Process all frames in each sub-video stream to determine the user's chest and abdominal breathing region and breathing interference region, and extract the motion intensity of the chest and abdominal breathing region and the motion intensity of the breathing interference region.

[0059] Specifically, the chest and abdomen breathing region reflects respiratory movements, including the chest and abdomen, while the breathing interference region refers to the limb areas that generate motion interference artifacts, including the limbs and head. This invention distinguishes the regions that generate motion interference artifacts and determines the validity of sub-video streams based on the motion intensity of the chest and abdomen breathing region and the motion intensity of the breathing interference region. Subsequent processing is performed only on the valid sub-video streams to detect the user's breathing state, removing the influence of artifacts and effectively improving the accuracy and reliability of sleep apnea event detection.

[0060] In one implementation, all frames within each sub-video stream are processed to determine the user's chest and abdominal breathing region and breathing interference region, including:

[0061] Input all frames in each sub-video stream into a pre-trained human parsing depth model and output the semantic segmentation result for each frame;

[0062] Based on the semantic segmentation results of all frames, the user's chest and abdominal breathing region and breathing interference region are determined.

[0063] The human body analysis depth model is an M2FP model based on the Transformer architecture.

[0064] Specifically, for each pixel coordinate, its semantic label distribution across all frames of the sub-video stream is statistically analyzed, and the label with the highest frequency is selected as the final classification label for that pixel coordinate. Pixels with the final classification labels of chest and abdomen form the chest and abdomen breathing region, while pixels with the final classification labels of head, left arm, right arm, left leg, or right leg form the breathing interference region. The M2FP (Multi-scale Multi-hierarchical Feature Pyramid) model can perform multi-scale feature extraction and fusion on the input image, effectively capturing global and local details, thereby obtaining accurate semantic segmentation results.

[0065] In one implementation, processing all frames within each sub-video stream to determine the user's chest and abdominal breathing region and breathing interference region also includes:

[0066] All frames in each sub-video stream are input into a pre-trained human pose estimation model. The human pose estimation model performs pose estimation on each frame and outputs the first and second key points corresponding to each frame.

[0067] The chest and abdominal breathing region is determined based on the first key point corresponding to all frames, and the breathing interference region is determined based on the second key point corresponding to all frames.

[0068] The first key point includes the left shoulder, right shoulder, left hip, and right hip, while the second key point includes the left elbow, right elbow, left knee, right knee, and top of the head.

[0069] Specifically, this invention also supports the use of human pose estimation techniques to distinguish different regions. The human pose estimation model can be an OpenPose (Real-time Multi-Person 2D Pose Estimation using PartAffinity Fields) model or an HRNet (High-Resolution Network) model.

[0070] In one implementation, extracting the motion intensity of the chest and abdominal respiratory region and the motion intensity of the respiratory interference region includes:

[0071] The chest and abdominal breathing region and the breathing interference region are divided into several pixel blocks according to a preset size;

[0072] The displacement data of each pixel block between two adjacent frames is calculated using an optical flow algorithm. The displacement data includes horizontal displacement and vertical displacement.

[0073] The standard deviation of all displacement data in the chest and abdominal breathing region is calculated and used as the corresponding exercise intensity, and the standard deviation of all displacement data in the breathing interference region is calculated and used as the corresponding exercise intensity.

[0074] Specifically, the preset size can be 20×20 pixels or 15×15 pixels. The optical flow algorithm can be the PixFlow algorithm. The displacement data of each pixel block between two adjacent frames can be regarded as a motion signal. The standard deviation of the motion signal can be calculated to obtain the corresponding motion intensity.

[0075] Please see Figure 1 The respiratory state detection method described in this embodiment of the invention further includes the following steps:

[0076] Step S300: Mark the validity of the sub-video stream based on the motion intensity of the chest and abdominal breathing region and the motion intensity of the breathing interference region within each sub-video stream.

[0077] Specifically, the motion intensity of the breathing interference region within each sub-video stream is compared to a preset multiple of the motion intensity of the chest and abdominal breathing region. If the motion intensity of the breathing interference region is less than or equal to the preset multiple of the motion intensity of the chest and abdominal breathing region, the sub-video stream is marked as valid; if the motion intensity of the breathing interference region is greater than the preset multiple of the motion intensity of the chest and abdominal breathing region, the sub-video stream is marked as invalid. When the motion intensity of the breathing interference region is less than or equal to the preset multiple of the chest and abdominal breathing region, it indicates that the breathing signal is not masked by interference, and the sub-video stream can be marked as valid. However, when the motion intensity of the breathing interference region is greater than the preset multiple of the chest and abdominal breathing region, it indicates the presence of severe body motion artifacts, and the sub-video stream needs to be marked as invalid. The preset multiple can be 5. This method can effectively improve the input data quality of respiratory time analysis and avoid misjudgments caused by artifacts.

[0078] In addition to the methods described above, marking the validity of a sub-video stream based on the motion intensity of the chest and abdominal breathing region and the motion intensity of the breathing interference region within each sub-video stream further includes: setting a sliding window and sliding the sliding window along the time axis with a preset step size. The size of the sliding window is equal to the preset step size, ensuring that when the window slides along the time axis with this step size, adjacent windows continuously cover all sub-video streams without overlap. For all sub-video streams covered by the current sliding window, a first baseline of the breathing interference region and a second baseline of the chest and abdominal breathing region within the sliding window are calculated respectively. The first baseline is divided by the second baseline and rounded to obtain an adaptive multiple. For each sub-video stream within the current sliding window, the motion intensity of the chest and abdominal breathing region is multiplied by the adaptive multiple to obtain a dynamic threshold. The motion intensity of the breathing interference region of the sub-video stream is compared with the dynamic threshold, and the validity of the sub-video stream is marked according to the comparison result. If the motion intensity of the breathing interference region is less than or equal to the dynamic threshold, the sub-video stream is marked as valid; if the motion intensity of the breathing interference region is greater than the dynamic threshold, the sub-video stream is marked as invalid. The first baseline is the average motion intensity of the breathing interference region in all sub-video streams within the sliding window, and the second baseline is the average motion intensity of the chest and abdominal breathing region in all sub-video streams within the sliding window. This labeling method, which combines adaptive scaling, significantly reduces misjudgments caused by accidental large body movements and effectively improves labeling accuracy.

[0079] Please see Figure 1 The respiratory state detection method described in this embodiment of the invention further includes the following steps:

[0080] Step S400: Detect the user's breathing state based on the sub-video streams marked as valid.

[0081] Specifically, the labeled valid sub-video streams can be input into a pre-trained three-dimensional convolutional neural network (3DCNN); after processing by the 3DCNN, the user's breathing state is output. The 3DCNN consists of three convolutional layers. The breathing states include normal breathing, hypoventilation, and apnea.

[0082] Furthermore, motion signals from the chest and abdominal breathing regions and breathing interference regions can be filtered (e.g., bandpass filtering) to extract time-domain, frequency-domain, and time-frequency-domain features. The motion signal is generated by calculating the horizontal and vertical displacements of pixel blocks within each region between adjacent frames using the PixFlow algorithm. Time-domain features can be directly extracted from the motion signal, including the mean (reflecting the average level of motion), variance (reflecting the degree of motion fluctuation), and root mean square (reflecting motion energy). Frequency-domain features are extracted after converting the motion signal from the time domain to the frequency domain using Fourier transform, including the spectrum (reflecting the energy distribution at different frequencies) and the dominant frequency (the frequency with the highest energy, corresponding to the main frequency of breathing or body movement). Time-frequency-domain features combine information from both time and frequency dimensions and can be extracted using methods such as wavelet transform, including time-frequency energy distribution (reflecting frequency energy changes at different time points). These time-domain, frequency-domain, and time-frequency-domain features are then input into a pre-trained Support Vector Machine (SVM) classifier, which processes and outputs the breathing state corresponding to the current sub-video stream. The support vector machine classifier uses similar features from historical respiratory state samples as training data.

[0083] In existing technologies, contact-based solutions include polysomnography (PSG), which is the gold standard for clinical diagnosis. It provides a comprehensive diagnosis by recording multiple physiological signals such as electroencephalography (EEG), electrocardiography (ECG), electromyography (EMG), respiratory airflow, chest and abdominal movements, and blood oxygen saturation, making it a powerful tool. However, PSG requires a specialized sleep laboratory, involves numerous and expensive equipment, and can cause significant restriction and discomfort for users, severely disrupting their normal sleep and making it unsuitable for routine, long-term home monitoring. Furthermore, contact-based solutions using mattresses are also susceptible to signal distortion due to user movement and limb movements, resulting in a very high false alarm rate. Non-contact detection devices also utilize microphone technology, which detects breathing sounds. However, its signal-to-noise ratio is extremely low, making it highly susceptible to interference from environmental noise (such as air conditioning or outside noise), and it struggles to detect silent sleep apnea events.

[0084] The respiratory state detection and processing flow of the present invention is as follows: Figure 2As shown, after acquiring the sleep monitoring video stream, the video frames are processed in real time to determine the user's chest and abdominal breathing areas and limb areas. The motion intensity between these areas is compared to detect whether the user is experiencing significant movement, thereby filtering out valid sub-video streams. Subsequently, the chest and abdominal breathing motion patterns under different breathing events are combined with a three-dimensional convolutional neural network to achieve high-precision detection of events such as normal breathing, sleep apnea, and hypoventilation. Through this cascaded strategy of "segmentation first, then discrimination, and then analysis," segments contaminated by motion artifacts can be actively identified and eliminated, ensuring that the detection and analysis are based on reliable respiratory motion signals, significantly improving the accuracy and reliability of user sleep breathing event detection.

[0085] In one embodiment, such as Figure 3 As shown, based on the above-described respiratory state detection method, the present invention also provides a respiratory state detection device, the device comprising:

[0086] The video stream segmentation module 100 is used to acquire the user's sleep monitoring video stream and divide the sleep monitoring video stream into several sub-video streams according to a preset time interval.

[0087] The motion intensity extraction module 200 is used to process all frames in each sub-video stream, determine the user's chest and abdominal breathing region and breathing interference region, and extract the motion intensity of the chest and abdominal breathing region and the motion intensity of the breathing interference region.

[0088] The marking module 300 is used to mark the validity of the sub-video stream based on the motion intensity of the chest and abdominal breathing region and the motion intensity of the breathing interference region within each sub-video stream;

[0089] The detection module 400 is used to detect the user's breathing state based on the sub-video streams that are marked as valid.

[0090] In one embodiment, the device further includes:

[0091] The semantic segmentation unit is used to input all frames in each sub-video stream into a pre-trained human parsing depth model and output the semantic segmentation result of each frame. The human parsing depth model is an M2FP model based on the Transformer architecture.

[0092] The first region segmentation unit is used to determine the user's chest and abdomen breathing region and breathing interference region based on the semantic segmentation results of all frames. The breathing interference region is a region that includes the limbs and head, and the chest and abdomen breathing region is a region that includes the chest and abdomen.

[0093] In one embodiment, the device further includes:

[0094] The pose estimation unit is used to input all frames in each sub-video stream into the pre-trained human pose estimation model, and the human pose estimation model performs pose estimation on each frame and outputs the first key point and the second key point corresponding to each frame. The first key point includes the left shoulder, right shoulder, left hip, and right hip, and the second key point includes the left elbow, right elbow, left knee, right knee, and top of the head.

[0095] The second region segmentation unit is used to determine the chest and abdominal breathing region based on the first key points corresponding to all frames, and to determine the breathing interference region based on the second key points corresponding to all frames.

[0096] In one embodiment, the device further includes:

[0097] A region segmentation unit is used to segment the chest and abdominal breathing region and the breathing interference region according to a preset size to obtain several pixel blocks;

[0098] The displacement calculation unit is used to calculate the displacement data of each pixel block between two adjacent frames using an optical flow algorithm. The displacement data includes horizontal displacement and vertical displacement.

[0099] The parameter determination unit is used to calculate and use the standard deviation of all displacement data in the chest and abdominal breathing region as the corresponding motion intensity, and to calculate and use the standard deviation of all displacement data in the breathing interference region as the corresponding motion intensity.

[0100] In one embodiment, the marking module includes:

[0101] The comparison unit is used to compare the motion intensity of the breathing interference region and the motion intensity of the chest and abdominal breathing region within each sub-video stream by a preset multiple.

[0102] The first marking unit is used to mark the sub-video stream as valid if the motion intensity of the breathing interference area is less than or equal to a preset multiple of the motion intensity of the chest and abdominal breathing area.

[0103] The second marking unit is used to mark the sub-video stream as invalid if the motion intensity of the breathing interference area is greater than a preset multiple of the motion intensity of the chest and abdominal breathing area.

[0104] In one embodiment, the marking module further includes:

[0105] A window setting unit is used to set a sliding window and slide the sliding window along the time axis at a preset step size;

[0106] An adaptive multiplier calculation unit is used to calculate, for all sub-video streams covered by the current sliding window, the first baseline of the breathing interference region and the second baseline of the chest and abdomen breathing region within the sliding window, respectively. The first baseline is divided by the second baseline and rounded to obtain the adaptive multiplier. The first baseline is the average motion intensity of the breathing interference region of all sub-video streams within the sliding window, and the second baseline is the average motion intensity of the chest and abdomen breathing region of all sub-video streams within the sliding window.

[0107] The third marking unit is used to multiply the motion intensity of the chest and abdominal breathing region by the adaptive factor for each sub-video stream within the current sliding window to obtain a dynamic threshold, compare the motion intensity of the breathing interference region of the sub-video stream with the dynamic threshold, and mark the validity of the sub-video stream based on the comparison result.

[0108] In one embodiment, the detection module includes:

[0109] A video input unit for inputting labeled valid sub-video streams into a pre-trained 3D convolutional neural network;

[0110] The state determination unit is used to output the user's breathing state after processing by the three-dimensional convolutional neural network.

[0111] Based on the above embodiments, the present invention also provides a terminal, the structural schematic diagram of which is as follows: Figure 4 As shown. The terminal includes a processor, memory, network interface, and display screen connected via a device bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores operating devices and a breathing state detection program. The internal memory provides an environment for the operation of the operating devices and the breathing state detection program stored in the non-volatile storage medium. The network interface is used for communication with external terminals via a network connection. When the breathing state detection program is executed by the processor, it implements the steps of any of the above-described breathing state detection methods. The display screen can be a liquid crystal display (LCD) or an e-ink display.

[0112] Those skilled in the art will understand that Figure 4 The structural schematic diagram shown is only a schematic diagram of a part of the structure related to the present invention solution, and does not constitute a limitation on the terminal on which the present invention solution is applied. The specific terminal may include more or fewer components than shown in the figure, or combine some components, or have different component arrangements.

[0113] In one embodiment, a terminal is provided, the terminal including a memory, a processor, and a respiratory state detection program stored in the memory and executable on the processor. When the respiratory state detection program is executed by the processor, it implements the steps of any respiratory state detection method provided in the embodiments of the present invention.

[0114] This invention also provides a computer-readable storage medium storing a respiratory state detection program, which, when executed by a processor, implements the steps of any respiratory state detection method provided in this invention.

[0115] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0116] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the above device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this invention. The specific working process of the units and modules in the above device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0117] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0118] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0119] In the embodiments provided by this invention, it should be understood that the disclosed apparatus / terminal devices and methods can be implemented in other ways. For example, the apparatus / terminal device embodiments described above are merely illustrative. For instance, the division of modules or units described above is only a logical functional division, and in actual implementation, it can be divided in other ways. For example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed.

[0120] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not mean that the essence of the corresponding technical solutions deviates from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A respiratory state detection method, characterized by, The method comprises: obtaining a sleep monitoring video stream of a user, and dividing the sleep monitoring video stream into a plurality of sub-video streams according to a preset time interval; processing all frames in each sub-video stream to determine a chest and abdomen breathing region and a breathing interference region of the user, and extracting motion intensity of the chest and abdomen breathing region and motion intensity of the breathing interference region; based on the motion intensity of the chest and abdomen breathing region and the motion intensity of the breathing interference region in each sub-video stream, marking the effectiveness of the sub-video stream; based on the sub-video stream marked as effective, detecting the breathing state of the user; extracting the motion intensity of the chest and abdomen breathing region and the motion intensity of the breathing interference region comprises: segmenting the chest and abdomen breathing region and the breathing interference region according to a preset size to obtain a plurality of pixel blocks; calculating displacement data of each pixel block between adjacent two frames by using an optical flow algorithm, the displacement data comprising horizontal displacement and vertical displacement; calculating and taking the standard deviation of all displacement data of the chest and abdomen breathing region as the corresponding motion intensity, and calculating and taking the standard deviation of all displacement data of the breathing interference region as the corresponding motion intensity; based on the motion intensity of the chest and abdomen breathing region and the motion intensity of the breathing interference region in each sub-video stream, marking the effectiveness of the sub-video stream comprises: setting a sliding window, and sliding the sliding window along a time axis by a preset step; for all sub-video streams covered by the current sliding window, calculating a first baseline of the breathing interference region and a second baseline of the chest and abdomen breathing region in the sliding window respectively, dividing the first baseline by the second baseline and taking the integer part to obtain an adaptive multiplier; for each sub-video stream in the current sliding window, multiplying the motion intensity of the chest and abdomen breathing region by the adaptive multiplier to obtain a dynamic threshold, comparing the motion intensity of the breathing interference region of the sub-video stream with the dynamic threshold, and marking the effectiveness of the sub-video stream according to the comparison result; wherein the first baseline is the average value of the motion intensity of the breathing interference region of all sub-video streams in the sliding window, and the second baseline is the average value of the motion intensity of the chest and abdomen breathing region of all sub-video streams in the sliding window.

2. The respiratory state detection method according to claim 1, characterized in that, processing all frames in each sub-video stream to determine the chest and abdomen breathing region and the breathing interference region of the user comprises: inputting all frames in each sub-video stream into a pre-trained human body analysis depth model to output semantic segmentation results of each frame; based on the semantic segmentation results of all frames, determining the chest and abdomen breathing region and the breathing interference region of the user; wherein the human body analysis depth model is an M2FP model based on a Transformer architecture, the chest and abdomen breathing region is a region reflecting breathing motion, including the chest and abdomen, and the breathing interference region is a limb region generating motion interference artifacts, including the limbs and the head.

3. The respiratory state detection method according to claim 1, characterized in that, processing all frames in each sub-video stream to determine the chest and abdomen breathing region and the breathing interference region of the user comprises: input all frames in each sub-video stream into a pre-trained human pose estimation model, perform pose estimation on each frame by the human pose estimation model, and output first key points and second key points corresponding to each frame; determine a chest and abdominal breathing region based on the first key points corresponding to all frames, and determine a breathing interference region based on the second key points corresponding to all frames; wherein the first key points include left shoulder, right shoulder, left hip and right hip, and the second key points include left elbow, right elbow, left knee, right knee and head top.

4. The respiratory state detection method according to claim 1, characterized by, based on the motion intensity of the chest and abdominal breathing region and the motion intensity of the breathing interference region in each sub-video stream, mark the effectiveness of the sub-video stream, including: compare the motion intensity of the breathing interference region and the motion intensity of the chest and abdominal breathing region in each sub-video stream by a preset multiple; if the motion intensity of the breathing interference region is less than or equal to the preset multiple of the motion intensity of the chest and abdominal breathing region, mark the sub-video stream as effective; if the motion intensity of the breathing interference region is greater than the preset multiple of the motion intensity of the chest and abdominal breathing region, mark the sub-video stream as invalid.

5. The respiratory state detection method according to claim 1, characterized in that, based on the sub-video stream marked as effective, detect the breathing state of the user, including: input the sub-video stream marked as effective into a pre-trained three-dimensional convolutional neural network; after processing by the three-dimensional convolutional neural network, output the breathing state of the user.

6. A respiratory state detection apparatus, characterized by, including: a video stream division module for obtaining a sleep monitoring video stream of a user, and dividing the sleep monitoring video stream into a plurality of sub-video streams according to a preset time interval; a motion intensity extraction module for processing all frames in each sub-video stream, determining a chest and abdominal breathing region and a breathing interference region of the user, and extracting the motion intensity of the chest and abdominal breathing region and the motion intensity of the breathing interference region; extracting the motion intensity of the chest and abdominal breathing region and the motion intensity of the breathing interference region includes: segmenting the chest and abdominal breathing region and the breathing interference region according to a preset size to obtain a plurality of pixel blocks; calculating displacement data of each pixel block between adjacent two frames by using an optical flow algorithm, the displacement data including horizontal displacement and vertical displacement; calculating and taking the standard deviation of all displacement data of the chest and abdominal breathing region as the corresponding motion intensity, and calculating and taking the standard deviation of all displacement data of the breathing interference region as the corresponding motion intensity; a marking module for marking the effectiveness of the sub-video stream based on the motion intensity of the chest and abdominal breathing region and the motion intensity of the breathing interference region in each sub-video stream; based on the motion intensity of the chest and abdominal breathing region and the motion intensity of the breathing interference region in each sub-video stream, mark the effectiveness of the sub-video stream, including: set a sliding window, and slide the sliding window along the time axis by a preset step size; for all sub-video streams covered by the current sliding window, calculate a first baseline of the breathing interference region and a second baseline of the chest and abdominal breathing region in the sliding window, divide the first baseline by the second baseline and take the integer part to obtain an adaptive multiple; For each sub-video stream in the current sliding window, the motion intensity of the chest and abdominal breathing region is multiplied by the adaptive multiple to obtain a dynamic threshold, the motion intensity of the breathing interference region of the sub-video stream is compared with the dynamic threshold, and the validity of the sub-video stream is marked according to the comparison result; Wherein, the first baseline is the average value of the motion intensity of the breathing interference region of all sub-video streams in the sliding window, and the second baseline is the average value of the motion intensity of the chest and abdominal breathing region of all sub-video streams in the sliding window; The detection module is configured to detect the breathing state of the user based on the sub-video stream marked as valid.

7. A terminal, characterized by comprising: The terminal comprises a memory, a processor, and a breathing state detection program stored on the memory and executable on the processor, and the breathing state detection program, when executed by the processor, implements the steps of the breathing state detection method according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a breathing state detection program, and the breathing state detection program, when executed by the processor, implements the steps of the breathing state detection method according to any one of claims 1-5.

Images