Infant milk choking or milk regurgitation monitoring system and method based on recognition and analysis

Abstract

The invention relates to an infant milk choking or regurgitation monitoring system and method based on recognition and analysis, and the system comprises an image collection module which is used for continuously collecting visual images of an infant; the audio acquisition module is used for acquiring audio signals of the environment where the baby is located; the key point detection module extracts key point information of the head and the abdomen of the baby; the head posture analysis module is used for calculating head posture angle change according to the infant head key point information; the respiration monitoring module is used for calculating respiration amplitude and respiration frequency by analyzing the change of pixel values along with time; and the voiceprint recognition and analysis module is used for extracting voiceprint features from the audio signals, distinguishing infant sounds and adult sounds based on a classification model, and recognizing preset abnormal sound types and the like. The method has the advantages that through deep fusion of visual key point information (head postures and breathing waveforms) and audio voiceprint features, the baby state can be verified in a cross mode from two independent and complementary dimensions of behaviors and sounds.

Description

TECHNICAL FIELD

[0001] The present application relates to a kind of based on identification and analysis baby chokes milk or spits milk monitoring system and monitoring method, belong to baby health monitoring and intelligent sensing field. BACKGROUND

[0002] Baby chokes milk and spits milk are common sudden conditions in feeding process, if not timely handling can cause respiratory obstruction, aspiration pneumonia and other serious complications, pose a threat to baby health.Therefore, it has important practical significance to monitor and early warn baby chokes milk and spits milk behavior in real time and accurately.

[0003] At present, common baby monitoring methods mainly rely on continuous care of parents or nursing staff, but it is difficult to achieve all-weather uninterrupted attention, and there is a monitoring blind area when parents or nursing staff leave temporarily.In existing technology, there are also some auxiliary monitoring schemes with sensing devices, such as using pressure sensor under mattress to detect respiratory frequency change, or through single camera to analyze baby behavior image.However, these methods often have obvious limitations: pressure sensing method cannot effectively distinguish normal breathing from abnormal breathing pattern when choking milk, and is easily disturbed by other baby limb activities;The reliability of the scheme based on ordinary video analysis decreases when light is insufficient, shooting angle is limited or baby is blocked, and it is difficult to integrate sound features for comprehensive judgment, resulting in high false positive or false negative rate.

[0004] In view of the above problems, an intelligent monitoring scheme capable of integrating multi-modal information and having stronger environmental adaptability and judgment accuracy is needed to realize more reliable and timely identification and early warning of baby chokes milk and spits milk risk. SUMMARY

[0005] To overcome the defects of prior art, the present application provides a kind of based on identification and analysis baby chokes milk or spits milk monitoring system and monitoring method, the technical scheme of the present application is: A baby chokes milk or spits milk monitoring system based on key point identification and voiceprint analysis, comprising: An image acquisition module for continuously acquiring visual images of a baby; An audio acquisition module for acquiring audio signals of the environment where the baby is located; A key point detection module based on a deep learning pose estimation model to extract key point information of the baby's head and abdomen from the visual images; A head posture analysis module for calculating head posture angle changes according to the baby's head key point information; A respiration monitoring module for calculating respiration amplitude and respiration frequency by analyzing pixel value changes over time based on the abdomen key point information or abdomen region image sequence; a voiceprint recognition and analysis module for extracting voiceprint features from the audio signal, distinguishing baby voices from adult voices based on a classification model, and identifying preset abnormal sound types; a multi-modal risk judgment module for fusing the head posture angle change, breathing amplitude and frequency information, and the output of the voiceprint recognition and analysis module, calculating the risk level of milk choking / vomiting through a multi-modal fusion algorithm; an alarm module for triggering an alarm notification when the risk level exceeds a preset threshold.

[0006] The deep learning posture estimation model adopted by the key point detection module is a model constructed based on a convolutional neural network or a graph convolutional network, which detects head key points including at least: top of head, lower jaw, left ear, right ear, and mouth key points; and abdominal key points including at least upper and lower boundary key points for defining the abdominal region; The key point detection module realizes key point detection through the following steps: a. Feature extraction and preliminary positioning: input the input baby image I into a feature extraction backbone network constructed based on a convolutional neural network CNN, output multi-scale feature maps {F l | l=1,...,L}; then through a key point prediction head, for each predefined head and abdominal key point i, generate its heat map H i , the peak position of which is the initial coordinate of the key point; b. Key point optimization based on graph model: construct a graph structure G=(V,E) with key points as nodes and baby physiological structure connections as edges; use a graph convolutional network GCN to optimize the initial coordinate set , whose message passing and updating formula can be expressed as: ; wherein, is the feature representation of node i in the kth layer, the initial feature is composed of the initial coordinate of the key point and the corresponding heat map confidence, is the neighbor node set of node i, and is an activation function; finally, the node feature is mapped to the coordinate offset Δp i through the output layer to obtain the optimized coordinate ; c. Time sequence consistency filtering: for a video sequence, let the optimized coordinate of key point i at time t be p i (t), introduce a Kalman filter or a first-order low-pass filter to smooth it to suppress inter-frame jitter, and the updating formula is: ; wherein, For the smoothed coordinates, A is a state transition matrix, H is an observation matrix, K(t) is a Kalman gain matrix; this step is used to realize the continuity of the key point trajectory in the time dimension and the physiological motion rationality.

[0007] The respiratory monitoring module is implemented by locating the abdominal region in consecutive image frames, extracting the pixel intensity or optical flow features of the abdominal region to form a time series, filtering and peak detection on the time series to calculate the respiratory frequency and relative amplitude, specifically: a) Respiratory signal generation: based on the upper and lower boundaries of the abdominal region output by the key point detection module, dynamically intercept the abdominal region of interest in consecutive image frames; by calculating the average intensity value of the pixels in the region on a specific color channel, or calculating the average value of the optical flow vertical component of the region between consecutive frames, an original respiratory-related one-dimensional time series signal Sraw(t) is generated; b) Signal preprocessing: detrend Sraw(t) to eliminate baseline drift, and band-pass filter, with a passband frequency range of [f min ,f max ] Hz matching the infant's respiratory frequency, to obtain a pure respiratory waveform signal R(t); c) Respiratory parameter calculation: Respiratory frequency: by performing spectral analysis or autocorrelation analysis on R(t) in a sliding time window, the main frequency is extracted as the instantaneous respiratory frequency f r ; Respiratory amplitude: calculate the peak-to-peak value or signal energy of R(t) in a sliding time window as a measure of respiratory amplitude Ar; Rhythm abnormality detection: by analyzing the interval time variation of consecutive respiratory cycles, or detecting whether the amplitude of R(t) is continuously below a set threshold for more than a predetermined time, to determine whether there is apnea or rhythm disorder.

[0008] The voiceprint recognition and analysis module includes: a feature extraction unit for extracting mel frequency cepstral coefficients, mel spectrogram or sound spectrogram features from audio signals; a sound classification unit using a trained neural network classifier to classify the extracted features to distinguish between infant cries, infant coughs, adult voices and environmental noise; The voiceprint recognition and analysis module is implemented by the following steps: a) Audio preprocessing and feature extraction: frame and window the collected audio signal, extract the mel frequency cepstral coefficient MFCC feature vector or calculate the mel spectrogram for each frame of audio signal to form a time series feature sequence F(t); b) Sound event classification: input the sequence of time-series features F(t) into a pre-trained neural network classifier; the classifier is based on convolutional neural network or recurrent neural network architecture, output the probability distribution P(t) = [p c (t)] of each time frame belonging to pre-defined classes, where classes c include infant cry, infant choking cough, adult speech and environmental noise; c) Decision and output: determine whether the target infant abnormal sound event occurs in the current audio segment according to the probability distribution P(t), and generate a classification result in combination with context information, and output to the multi-modal risk judgment module.

[0009] The fusion algorithm adopted by the multi-modal risk judgment module is a neural network fusion model based on attention mechanism, a gated fusion network or a gradient boosting decision tree model, which inputs time-series feature vectors from different modalities and outputs discrete values or continuous probability values representing the risk level of choking / milk vomiting; The multi-modal risk judgment module is implemented by the following steps: a) Multi-modal feature alignment and encoding: receiving the respiratory parameter time-series data from the respiratory monitoring module, the posture angle time-series data from the head posture analysis module, and the infant abnormal sound event sequence from the voiceprint recognition and analysis module; aligning the heterogeneous time-series features on the time axis, and converting them into uniform feature vector sequences through feature encoders respectively; b) Feature fusion and risk feature extraction: fuse the encoded multi-modal feature vectors; the fusion method includes feature splicing followed by fully connected network mapping, or weighted fusion based on attention mechanism; extract deep risk features highly related to choking / milk vomiting risk from the fused features through a deep neural network; c) Risk level determination and output: input the extracted deep risk features into a classifier or regressor, calculate the comprehensive risk score S at the current time, and map it to a discrete risk level according to the pre-set threshold interval, or output it as a continuous risk probability value.

[0010] A monitoring method of an infant choking or milk vomiting intelligent monitoring system based on key point recognition and voiceprint analysis, comprising the following steps: S1: continuously acquire visual images of the infant through an image acquisition module, and acquire audio signals of the environment in which the infant is located through an audio acquisition module; S2: use a key point detection module to extract key point information of the infant's head and abdomen from the visual images based on a deep learning pose estimation model; S3: use a head posture analysis module to calculate the change of head posture angle according to the head key point information; S4: Using the respiratory monitoring module, based on the abdominal key point information or the abdominal region image sequence, the respiratory amplitude and the respiratory frequency are calculated by analyzing the change of pixel value over time; S5: Using the voiceprint recognition and analysis module, the voiceprint features are extracted from the audio signal, and the infant voice and the adult voice are distinguished based on a classification model, and a preset abnormal sound type is identified; S6: Using the multi-modal risk judgment module, the head posture angle change information, the respiratory amplitude and frequency information, and the output information of the voiceprint recognition and analysis module are fused, and the choking / milk regurgitation risk level is calculated by a multi-modal fusion algorithm; S7: When the risk level exceeds a preset threshold, an alarm notification is triggered by the alarm module.

[0011] In the step S2, the specific steps of extracting key points based on a deep learning posture estimation model include: a) Feature extraction and preliminary positioning: input the infant image into a feature extraction network based on a convolutional neural network, output multi-scale feature maps; generate a heat map for each predefined head and abdominal key point through a key point prediction head, and the peak position of the heat map is used as the initial coordinates of the key point; b) Key point optimization based on graph model: a graph structure is constructed with key points as nodes and infant physiological structure connections as edges, and the initial coordinates are optimized using a graph convolution network to obtain optimized key point coordinates; c) Time sequence consistency filtering: the coordinates of each key point in the video sequence are time sequence smoothed, and a Kalman filter or a low-pass filter is used to suppress inter-frame jitter to obtain smoothed key point trajectories.

[0012] In the step S4, the specific steps of calculating the respiratory amplitude and the respiratory frequency include: a) Based on the abdominal key point, the abdominal region of interest is dynamically intercepted, the pixel intensity or optical flow features in the region are extracted, and the original respiratory signal is generated; b) The original signal is de-trended and band-pass filtered to obtain a pure respiratory waveform signal; c) The respiratory frequency is extracted by spectral analysis, the respiratory amplitude is calculated, and the respiratory rhythm is detected for abnormality.

[0013] In the step S5, the specific steps of the voiceprint recognition and analysis module include: a) Frame and window processing of the audio signal, extraction of mel frequency cepstral coefficient or mel spectrum feature, formation of time sequence feature sequence; b) Input the temporal feature sequence into a pre-trained neural network classifier and output the probability distribution of each time frame belonging to a preset category, which includes infant crying, infant coughing, adult speech and environmental noise. c) Determine whether an abnormal sound event of the target infant has occurred based on the probability distribution, and output the classification result.

[0014] In step S6, the specific steps of the fusion algorithm include: a) Align and encode the temporal features from respiratory monitoring, head posture analysis, and voiceprint recognition on the time axis, converting them into a sequence of feature vectors of uniform dimension; b) The encoded multimodal features are fused. The fusion methods include feature concatenation followed by mapping through a fully connected network, or weighted fusion based on an attention mechanism. c) Extract deep risk features from the fused features, input them into a classifier or regressor to calculate a comprehensive risk score, and map them to discrete risk levels or continuous risk probability values.

[0015] The risk levels include at least three levels: normal, suspicious, and dangerous; when the risk level is dangerous, an emergency alarm notification is triggered and pushed to the guardian's mobile terminal. After an alarm is triggered, the system automatically saves the image and audio data within a preset time period before and after the trigger, generating an abnormal event playback record.

[0016] The advantages of this invention are: 1. By deeply fusing visual keypoint information (head posture, breathing waveform) with audio voiceprint features, the infant's state can be cross-verified from two independent and complementary dimensions: behavior and sound. This multimodal fusion strategy effectively reduces false negatives and false positives caused by environmental noise, changes in lighting, or misjudgment of a single modality.

[0017] 2. Information is collected solely through a camera and microphone, eliminating the need for any sensors to be attached or worn on the baby's body. This completely avoids potential skin allergies, discomfort, or a feeling of constraint that can occur with contact devices, while also eliminating safety hazards such as tangled cables. This non-invasive design makes the monitoring process natural and seamless, easily accepted by families, and suitable for long-term continuous use.

[0018] 3. It can automatically analyze the infant's condition 24 / 7 without interruption. Once the algorithm identifies a risk level exceeding a threshold, it immediately notifies the caregiver via an alarm module. This greatly compensates for the blind spots and fatigue intervals of human care, providing crucial safety assurance, especially at night or when the caregiver is briefly away, giving the caregiver greater peace of mind.

[0019] 4. The key point detection, voiceprint recognition, and multimodal fusion algorithms employed are all built upon deep learning frameworks such as convolutional neural networks and graph convolutional networks. These models, trained on a large amount of data, are able to adapt to different infant body shapes, postures, lighting conditions, and background environments, demonstrating good generalization ability.

[0020] 5. The tiered early warning mechanism provides caregivers with richer decision support information, helping them to distinguish between high-risk emergencies and suspicious situations that require attention, thereby enabling them to take more appropriate and timely response measures. Attached Figure Description

[0021] Figure 1 This is a structural block diagram of the system of the present invention.

[0022] Figure 2 This is a flowchart of the method of the present invention. Detailed Implementation

[0023] The present invention will be further described below with reference to specific embodiments, and the advantages and features of the present invention will become clearer as a result. However, these embodiments are merely exemplary and do not constitute any limitation on the scope of the present invention. Those skilled in the art should understand that modifications or substitutions can be made to the details and form of the technical solutions of the present invention without departing from the spirit and scope of the present invention, but all such modifications and substitutions fall within the protection scope of the present invention.

[0024] See Figure 1 and Figure 2 This invention relates to an infant choking or vomiting monitoring system based on identification and analysis, comprising: Image acquisition module 1 is used to continuously acquire visual images of the infant; Audio acquisition module 2 is used to acquire audio signals from the environment in which the baby is located; Key point detection module 3, based on a deep learning pose estimation model, extracts key point information of the baby's head and abdomen from the visual image; Head posture analysis module 4 calculates the head posture angle change based on the key point information of the infant's head; The respiratory monitoring module 5 calculates respiratory amplitude and respiratory rate by analyzing the changes in pixel values ​​over time based on the abdominal key point information or abdominal region image sequence. The voiceprint recognition and analysis module 6 is used to extract voiceprint features from the audio signal, distinguish between infant voices and adult voices based on a classification model, and identify preset abnormal sound types. The multimodal risk assessment module 7 is used to integrate the head posture angle change, breathing amplitude and frequency information, and the output of the voiceprint recognition and analysis module to calculate the risk level of choking / vomiting through a multimodal fusion algorithm; Alarm module 8 is used to trigger an alarm notification when the risk level exceeds a preset threshold.

[0025] The system acquires the infant's visual and auditory information in parallel through image and audio acquisition modules, which are then fused and analyzed by subsequent intelligent modules. The keypoint detection module, based on a deep learning model, stably extracts key points from the infant's head and abdomen from the video stream, providing accurate spatial localization for subsequent analysis. The head posture analysis module calculates changes in posture angle in real time based on the head key points, effectively capturing abnormal backward tilting or twisting movements that may occur during choking. The breathing monitoring module uses abdominal key points to locate the breathing area and extracts the breathing waveform and rhythm non-contactly by analyzing micro-motion changes in the image sequence, identifying apnea or irregular breathing. Simultaneously, the voiceprint recognition and analysis module extracts and classifies features from the acquired audio, specifically for identifying abnormal sound events such as infant coughing and crying, eliminating environmental and adult voice interference.

[0026] To achieve more reliable judgment, a multimodal risk assessment module was designed. This module does not simply superimpose data, but uses an attention-based fusion algorithm to temporally align and weight the head posture and breathing parameters from vision with abnormal sound events from hearing, thereby extracting deeper risk features and outputting a comprehensive risk level. Finally, the alarm module issues intelligent warnings based on the risk level.

[0027] Through the collaborative work of the above modules, an intelligent analysis framework with dual-channel perception (visual and auditory) and dual-driven key points and voiceprints is constructed. Its key advantages are: achieving comprehensive, multi-dimensional, and complementary monitoring of infant choking and vomiting behaviors in a completely non-contact manner; enhancing adaptability to complex scenarios through deep learning models; and significantly reducing the possibility of misjudgment by a single sensor through a multi-modal information fusion mechanism, ultimately achieving highly accurate, real-time, and user-friendly monitoring.

[0028] The key point detection module uses a deep learning pose estimation model based on a convolutional neural network or a graph convolutional network. The head key points it detects include at least the following: top of the head, jaw, left ear, right ear, and mouth key points; the abdominal key points include at least the key points used to define the upper and lower boundaries of the abdominal region. The key point detection module performs key point detection through the following steps: a. Feature Extraction and Preliminary Localization: The input infant image I is fed into a feature extraction backbone network built on a convolutional neural network (CNN), which outputs a multi-scale feature map {F}. l | l=1,...,L}; then, the head is predicted by key points, and a heatmap H is generated for each predefined head and abdomen key point i. i The peak position of the heat map , which is the initial coordinate of the key point; b. Keypoint optimization based on graph model: Construct a graph structure G=(V,E) with keypoints as nodes and connections to infant physiological structures as edges; use a graph convolutional network (GCN) to optimize the initial coordinate set. With optimization, its message passing and update formula can be expressed as: ; in, Let i be the feature representation of node i in the k-th layer, and let the initial features be... The confidence level of the key point is determined by its initial coordinates and the corresponding heatmap. Let represent the set of neighboring nodes of node i, and be the activation function; finally, the node features are mapped to coordinate offsets Δp through the output layer. i The optimized coordinates are obtained. ; c. Temporal Consistency Filtering: For a video sequence, let the optimized coordinates of keypoint i at time t be p. i (t) is smoothed using a Kalman filter or a first-order low-pass filter to suppress inter-frame jitter. The update formula is as follows: ; in, The coordinates are smoothed, A is the state transition matrix, H is the observation matrix, and K(t) is the Kalman gain matrix; this step is used to achieve the continuity of the key point trajectory in the time dimension and the rationality of physiological movement.

[0029] The deep learning pose estimation model and its detection steps employed by the key point detection module together constitute an accurate, robust, and physiologically reasonable infant key point perception system. This is reflected not only in the independent design of each step, but also in the closed-loop optimization process formed by their collaborative work.

[0030] 1. Feature Extraction and Preliminary Localization (Step a): A CNN-based multi-scale feature extraction backbone network is employed, enabling it to simultaneously capture the overall contour of the infant's body (high-level semantic features) and the fine texture near key points (low-level detail features). This ensures that even when the infant is wearing different clothing, is in a background, or is under uneven lighting, the system can still generate a heatmap with clear peaks for each key point, obtaining reliable initial coordinates.

[0031] For example, when an infant is lying on their side and part of their face is obscured, it may be difficult to locate the key point of the "right ear" using only single-scale features. However, in a multi-scale feature map, high-level features can understand the general area of ​​the "head," while low-level features can finely search the edge contour of the "ear" within that area. This allows for a reasonable initial estimate of the ear's location even when it is obscured, laying the foundation for subsequent optimization.

[0032] 2. By constructing a graphical model that incorporates the connectivity of the infant's physiological structures and optimizing it using a Graph Convolutional Network (GCN), the system no longer views each keypoint in isolation. GCN, through a message-passing mechanism, allows keypoints to "correct" each other based on their inherent spatial constraints (such as the relatively fixed distance and relative position between the "top of the head" and the "chin"). This effectively eliminates significant outlier errors caused by image noise or temporary occlusion during initial localization.

[0033] For example, suppose that during the initial localization, the heatmap peak of the "mouth" key point shifts due to arm movement obstructing the view, and is located on the cheek. During optimization, GCN receives information from its strongly related neighboring nodes (such as "jaw" and "left / right ear"). The coordinates of these neighboring nodes are relatively accurate, and the constraint information passed through the graph structure prompts the feature representation of the "mouth" node to adjust to its physiologically reasonable position, thereby outputting more accurate optimized coordinates and overcoming the limitations of single-frame image analysis.

[0034] 3. Introducing temporal filters such as Kalman filtering is a crucial supplement to spatial optimization. It combines single-frame image observation with state prediction based on motion models, smoothing out inter-frame coordinate jitter caused by video compression, lighting flicker, or minor fluctuations in the detection model itself. This ensures that the output keypoint trajectories are continuous and smooth in time, better reflecting the coherence and dynamic characteristics of an infant's actual movements.

[0035] Infants naturally exhibit periodic breathing movements during sleep, and the key points on their abdomen (upper / lower boundaries) should show a smooth, undulating trajectory. Without temporal filtering, the detection coordinates may exhibit irregular jumps of several pixels between frames, resulting in the calculated breathing waveform containing a large amount of high-frequency noise. After Kalman filtering smoothing, these random jitters are effectively suppressed, and the extracted abdominal movement trajectory more accurately reflects the slow, periodic displacement caused by breathing, thus providing a cleaner and more reliable input signal for subsequent breathing monitoring modules.

[0036] This keypoint detection scheme achieves high-precision and high-stability tracking of infant keypoints from single-frame images to continuous video streams through progressive processing including multi-scale initial localization, graph structure space optimization, and temporal motion smoothing. This not only directly improves the accuracy of head posture analysis and respiratory monitoring, but also provides a solid and reliable visual perception foundation for the entire multimodal risk assessment system.

[0037] The respiratory monitoring module is implemented as follows: The abdominal region is located in consecutive image frames; the pixel intensity or optical flow features of the abdominal region are extracted to form a time series; and the time series is filtered and peak detected to calculate the respiratory rate and relative amplitude. Specifically: a) Respiratory signal generation: Based on the upper and lower boundaries of the abdominal region output by the key point detection module, the abdominal region of interest is dynamically extracted in consecutive image frames; by calculating the average intensity value of the pixels in the region on a specific color channel, or by calculating the average vertical component of the optical flow in the region between consecutive frames, the original respiratory-related one-dimensional temporal signal Sraw(t) is generated. b) Signal preprocessing: Sraw(t) is detrended to eliminate baseline drift and bandpass filtered, with the passband frequency range set to match the infant's respiratory rate [f]. min ,f max ]Hz, to obtain a pure respiratory waveform signal R(t); c) Calculation of respiratory parameters: Respiratory rate: The dominant frequency of R(t) is extracted as the instantaneous respiratory rate f by performing spectral analysis or autocorrelation analysis on R(t) within a sliding time window. r ; Respiratory amplitude: Calculate the peak-to-peak value or signal energy of R(t) within the sliding time window, as a measure of respiratory amplitude Ar; Rhythm abnormality detection: By analyzing the variation in the interval time of consecutive respiratory cycles, or by detecting whether the amplitude of R(t) is continuously lower than a set threshold for a predetermined time, it can be determined whether there is apnea or rhythm disorder.

[0038] The implementation of the described respiratory monitoring module demonstrates its significant advantages as a non-contact, adaptive, and intelligent method for extracting respiratory parameters and detecting anomalies. Its advantages lie not only in the comfort of non-contact monitoring but also in the algorithm design's targeted solutions to various challenges in real-world application scenarios. A detailed analysis of the specific steps is as follows: 1. Based on the real-time coordinates of the upper and lower abdominal key points provided by the key point detection module, the abdominal monitoring area in the current frame is adaptively determined and extracted. This allows the monitoring area to automatically adjust with changes in the infant's posture (such as lying on their side or rolling over), fundamentally ensuring the accuracy of the signal source and avoiding the problem of fixed-area monitoring failing due to body movement. Simultaneously, two signal generation strategies—pixel intensity method and optical flow method—are provided, forming a redundant and complementary approach. The former is computationally efficient and suitable for most static scenes; the latter is more sensitive to minute movements and has an advantage when textures are indistinct or lighting is uneven.

[0039] When an infant changes from a supine to a side-lying position, the shape and position of the abdomen change. The monitoring area is adjusted based on the new key point coordinates, continuously locking onto the abdomen. If the indoor lighting is dim at this time, the texture of the clothing on the abdomen may be blurred, and the change in pixel intensity may not be obvious. The system can prioritize or use optical flow in parallel, analyzing the pixel motion vectors of the abdominal region in the image sequence to sense the micro-movements caused by breathing, thereby ensuring the generation of effective raw respiratory signals in any posture.

[0040] 2. By combining detrending processing and bandpass filtering, signal variations caused by non-respiratory factors are effectively eliminated. Detrending eliminates baseline drift caused by slow changes in ambient light and the baby's overall movement; bandpass filtering (usually set to 0.1-3Hz, corresponding to the baby's respiratory rate range of 6-180 breaths / minute) filters out high-frequency noise (such as camera noise and limb tremors) and low-frequency interference (such as slow body movement), ultimately resulting in a clean waveform focused on the respiratory frequency band.

[0041] Example: During monitoring, the movement of clouds outside the window causes a slow increase in indoor natural light, resulting in a gradual upward trend in the average intensity value of pixels in the abdominal area (baseline drift). Simultaneously, the infant's occasional kicking movements generate high-frequency pulses. The preprocessing step first removes the slowly rising baseline through detrending, then directly filters out the high-frequency components corresponding to the kicks using a bandpass filter. The final output R(t) will contain almost only the periodic fluctuations caused by breathing, laying the foundation for accurate parameter calculation.

[0042] 3. The dominant frequency is determined using spectral analysis / autocorrelation analysis, a mature method that can adapt to changes in respiratory rate. Peak-to-peak value or energy is calculated as amplitude, directly reflecting respiratory depth. Most importantly, by analyzing the coefficient of variation of the respiratory cycle interval or detecting that the R(t) amplitude is consistently below a threshold, the system can identify dangerous precursors such as apnea or irregular breathing rhythm, which are among the physiological changes that can result from choking on milk.

[0043] In suspected cases of choking on milk, infants may initially exhibit disordered breathing rhythm (rhythm disturbance), which may then progress to brief apnea. This module calculates the interval between consecutive respiratory cycles by analyzing `R(t)` in real time. If a sudden and abnormally prolonged interval (e.g., exceeding 3 seconds) or a significant increase in variability is detected, the system can issue an early warning during the "rhythm anomaly detection" phase, even if the average respiratory rate has not changed drastically. Simultaneously, if the peak-to-peak value of `R(t)` (respiratory amplitude) continuously weakens below the danger threshold, it can directly trigger alarms for shallow breathing or apnea.

[0044] This respiratory monitoring module not only achieves completely non-contact and highly comfortable respiratory monitoring, but also ensures that the extracted respiratory parameters and events are highly reliable in the complex environment of a real home through dynamic adaptation, anti-interference processing and advanced anomaly detection algorithms. This provides crucial and real-time physiological indicators for the entire system to judge the risk of choking / vomiting.

[0045] The voiceprint recognition and analysis module includes: The feature extraction unit is used to extract Mel frequency cepstral coefficients, Mel spectrograms, or acoustic spectrograms from audio signals. The sound classification unit uses a trained neural network classifier to classify the extracted features to distinguish between infant cries, infant coughing sounds, adult speech, and environmental noise. The voiceprint recognition and analysis module is implemented through the following steps: a) Audio preprocessing and feature extraction: The acquired audio signal is segmented and windowed. For each frame of audio signal, the Mel frequency cepstral coefficient (MFCC) feature vector is extracted or its Mel spectrum is calculated to form a time-series feature sequence F(t). b) Sound event classification: The temporal feature sequence F(t) is input into a pre-trained neural network classifier; the classifier is based on a convolutional neural network or recurrent neural network architecture and outputs the probability distribution P(t) = [p...] of each time frame belonging to a preset category. c [(t)], where category c includes baby crying, baby coughing, adult speech and environmental noise; c) Decision and output: Based on the probability distribution P(t), determine whether the target infant's abnormal sound event occurs in the current audio segment, and generate a classification result by combining the context information, and output it to the multimodal risk judgment module.

[0046] The voiceprint recognition and analysis module can not only accurately separate the voice of the target infant from complex acoustic environments, but also perform semantic-level recognition of risk-related events (such as choking), thereby providing a reliable auditory evidence chain for multimodal judgment.

[0047] The feature extraction unit uses Mel-frequency cepstral coefficients (MFCC) or Mel spectrograms as core features, a biomimetic design based on the characteristics of human hearing. The Mel scale is more sensitive to low frequencies, effectively capturing the essential sound features rich in mid-to-low frequency energy, such as infant cries and coughs, while naturally suppressing some high-frequency environmental noise. This feature extraction method provides highly discriminative and low-redundancy input for subsequent classification.

[0048] In an environment where both a baby's high-frequency, sharp crying and low-frequency air conditioning noise are present, MFCC features compress the harmonic structure and formants representing the baby's cry, while relatively weakening the smooth low-frequency noise from the air conditioner, making it easier for the classifier to focus on the key patterns of the target sound.

[0049] The sound classification unit can learn complex, non-linear decision boundaries between different sound categories. CNNs excel at capturing local spatiotemporal patterns in Mel spectrograms (such as the short-term burst texture of a cough), while RNNs excel at modeling the temporal dependencies of sound events (such as the fluctuating rhythm of a cry). This design enables the module to distinguish easily confused acoustic events with high accuracy, such as infant coughing versus swallowing, and adult coughing versus speech.

[0050] The "choking" sound of an infant and the "hiccup" sound may be similar in the short-time spectrum. A well-trained CNN classifier can learn that choking sounds are usually accompanied by a more abrupt onset and a wider frequency band energy distribution, thus effectively separating them from hiccups in the feature space and significantly reducing misclassification.

[0051] Decision and Output Logic: By incorporating contextual information, the clinical rationality of the event assessment is ensured. The final decision is not simply based on the maximum probability of a single frame, but rather on a comprehensive interpretation that integrates contextual information (such as event duration, probability continuity, and evolution trends). For example, a genuine case of choking related to milk typically manifests as multiple consecutive, short-interval coughing episodes. The decision logic analyzes the clusters of consecutive high-probability "choking sound" categories in P(t) and determines whether their duration and frequency match a choking pattern. If only one isolated, extremely short-lived high-probability frame is detected, it may be classified as random noise rather than a risky event, thus significantly improving the accuracy and reliability of the alarm.

[0052] It not only solved the question of "whether there is a baby's voice", but also more accurately answered the question of "what kind of sound does the baby make, and does that sound mean risk".

[0053] Its output of "abnormal infant sound events" (such as "high-confidence coughing event lasting 2 seconds") is a structured risk clue at the same semantic level as the "abnormal posture" and "breathing disorder" signals output by the visual analysis module, and can be directly used for advanced fusion judgment. This avoids the common problem of multimodal systems awkwardly splicing raw data or low-level features.

[0054] The multimodal risk assessment module uses a fusion algorithm based on an attention mechanism, a gated fusion network, or a gradient boosting decision tree model. Its input is a temporal feature vector from different modalities, and its output is a discrete value or a continuous probability value representing the risk level of choking / vomiting. The multimodal risk assessment module is implemented through the following steps: a) Multimodal feature alignment and encoding: Receives time-series respiratory parameter data from the respiratory monitoring module, time-series posture angle data from the head posture analysis module, and a sequence of abnormal infant voice events from the voiceprint recognition and analysis module; aligns the heterogeneous time-series features on the time axis and converts them into a unified feature vector sequence through a feature encoder. b) Feature fusion and risk feature extraction: The encoded multimodal feature vectors are fused; the fusion method includes feature concatenation followed by mapping to a fully connected network, or weighted fusion based on an attention mechanism; deep risk features highly correlated with the risk of choking / vomiting are extracted from the fused features using a deep neural network; c) Risk level determination and output: Input the extracted deep risk features into the classifier or regressor, calculate the comprehensive risk score S at the current time, and map it to a discrete risk level according to the preset threshold range, or output it as a continuous risk probability value.

[0055] This step addresses the primary challenge of multi-source information fusion: heterogeneity and temporal asynchrony. Breathing parameters (such as frequency and amplitude), head posture angles (such as pitch angle), and sound events (such as "coughing") are signals with different physical meanings, data rates, and temporal starting points. Through time axis alignment and feature encoders, the system transforms them into a sequence of feature vectors with uniform dimensions, uniform rates, and time synchronization, providing a comparable and computable foundation for subsequent deep fusion.

[0056] Suppose that at time t, the respiratory monitoring module detects a respiratory arrest, the head posture module detects a violent head tilt, and the voiceprint module identifies a coughing sound t+0.3 seconds later. Without alignment, these three events are isolated. Through feature alignment, the system calibrates them into a unified temporal framework, forming a combined feature vector characterized as "within a close time window, 'respiratory arrest + head tilt + coughing' occurred," which is a more significant risk indicator than any single event.

[0057] Employing attention-based weighted or gated fusion networks allows the model to assign different importance weights to information from different modalities at different times, dynamically focusing on evidence most relevant to the current risk. For example, in a quiet environment, the weight of sound events may increase; while when an infant is frequently wriggling, the weights of posture and breathing may be more critical. Deep neural networks further extract deeper risk association patterns from the fused features, going beyond simple rules.

[0058] In the early stages of a risk event, there may be only slight respiratory rhythm disturbances and occasional head movements, but no obvious coughing. Attention-based fusion models can automatically assign higher weights to these subtle visual cues and extract deeper features of the "early respiratory-postural incoordination pattern." Conversely, if there is only a brief cough but breathing and posture are completely normal, the model will reduce the weight of the vocal modality to avoid overreacting. This capability enables the system to identify risks earlier and more accurately, reducing false negatives and missed positives.

[0059] The risk level assessment and output ultimately provides a discrete risk level (e.g., normal, watch out, dangerous) or a continuous risk probability value, rather than a simple "yes / no" alarm. This provides caregivers with a basis for tiered responses, aligning with clinical decision-making logic. Furthermore, this design allows the system to balance sensitivity and specificity through threshold adjustments, adapting to the needs of different infants or families.

[0060] The system may output "Risk Level: Attention (Probability 65%)", indicating that the caregiver has noticed some abnormal signs in the infant (such as slightly rapid breathing accompanied by slight wriggling) and needs to be observed closely. If the situation develops into "Danger (Probability 92%)" (such as apnea, persistent backward tilting accompanied by continuous coughing), it indicates that immediate intervention is required.

[0061] This invention also relates to a monitoring method for an intelligent monitoring system for infant choking or vomiting based on key point recognition and voiceprint analysis, comprising the following steps: S1: The image acquisition module continuously acquires visual images of the baby, and the audio acquisition module acquires audio signals of the baby's environment. S2: Using the key point detection module, based on the deep learning pose estimation model, extract key point information of the baby's head and abdomen from the visual image; S3: Using the head posture analysis module, calculate the head posture angle change based on the head key point information; S4: Using the respiratory monitoring module, based on the abdominal key point information or abdominal region image sequence, calculate the respiratory amplitude and respiratory rate by analyzing the changes in pixel values ​​over time; S5: Using the voiceprint recognition and analysis module, extract voiceprint features from the audio signal, and distinguish between infant voices and adult voices based on a classification model, as well as identify preset abnormal sound types; S6: Using the multimodal risk assessment module, the head posture angle change information, breathing amplitude and frequency information, and the output information of the voiceprint recognition and analysis module are integrated to calculate the risk level of choking / vomiting through a multimodal fusion algorithm; S7: When the risk level exceeds the preset threshold, an alarm notification is triggered through the alarm module.

[0062] Step S1, which involves the simultaneous acquisition of visual and auditory signals, ensures the consistency of timestamps, making subsequent multimodal temporal alignment and fusion possible. This parallel data acquisition method guarantees the integrity and complementarity of information from the source.

[0063] Steps S2 to S4 constitute a progressive visual analysis chain. S2 (key point detection) is the cornerstone, providing precise spatial positioning for S3 (posture analysis) and S4 (respiration monitoring). The advantage of this chain is that it achieves completely non-contact acquisition of physiological and behavioral parameters (posture, respiration), ensuring the comfort and safety of the infant while avoiding the interference and limitations of contact sensors.

[0064] The S5 step is independent of the visual flow and focuses solely on processing audio information. Its advantage lies in its ability to not only differentiate sound sources (infant / adult) but also to identify unusual sound types that pose specific risks (such as coughing sounds). This provides strong, independent auditory evidence for judgment, particularly useful in situations where vision is obstructed but sound can still be heard.

[0065] S6 is the core and essence of this method. It fuses and jointly analyzes the heterogeneous information (posture angle, breathing parameters, sound events) generated in the preceding steps. Its greatest advantage is that it achieves "information cross-validation" and "contextual comprehensive understanding." For example, a single head tilt back may be considered extension, but if it is accompanied by apnea and coughing sounds, the risk of choking on milk is extremely high. This fusion decision-making mechanism greatly reduces the risk caused by misjudgment (false alarm) or insufficient sensitivity (false negative) due to single pieces of information, making the system's judgment closer to the comprehensive reasoning of a human caregiver.

[0066] Step S7 transforms the intelligent analysis results into specific action instructions (alarms), completing a full closed loop from "perception-analysis-decision-action". Its advantage lies in achieving 24 / 7 uninterrupted automatic monitoring and real-time early warning. After quantitatively assessing the risk, it can proactively notify caregivers at critical moments, effectively compensating for the attention gaps and fatigue blind spots of human monitoring.

[0067] In step S2, the specific steps for extracting key points based on the deep learning pose estimation model include: a) Feature extraction and preliminary localization: The infant image is input into a feature extraction network based on a convolutional neural network, which outputs a multi-scale feature map; a heat map is generated for each predefined head and abdominal key point through key point prediction, and the peak position of the heat map is used as the initial coordinate of the key point; b) Keypoint optimization based on graph model: Construct a graph structure with keypoints as nodes and connections between infant physiological structures as edges, and use a graph convolutional network to optimize the initial coordinates to obtain the optimized keypoint coordinates; c) Temporal consistency filtering: The coordinates of each key point in the video sequence are temporally smoothed. Kalman filters or low-pass filters are used to suppress inter-frame jitter and obtain the smoothed key point trajectory.

[0068] In step S2, the three-step process of key point extraction (feature extraction and preliminary localization, graph model-based optimization, and temporal consistency filtering) together constitute a progressive optimization system that is "from coarse to fine and from static to dynamic".

[0069] By leveraging multi-scale feature extraction based on convolutional neural networks, we can simultaneously capture global semantic information (such as torso contours) and local detail features (such as ear edges) of an infant's body, ensuring that effective heatmap responses are generated for each keypoint under various lighting conditions, clothing, and partial occlusion. The mechanism of using heatmap peaks as initial coordinates is more robust to positional changes than directly regressing coordinate values, providing stable and comprehensive initial keypoint guesses.

[0070] When an infant is wearing loose clothing or is in a sidelit environment, the abdominal boundary may be blurred. Multi-scale feature extraction can combine contextual information (such as torso position) and local texture to generate a heatmap region that, while not precise, is roughly located on the abdomen. This provides a reliable starting point for subsequent optimization and avoids tracking loss due to complete failure of initial localization.

[0071] By constructing a graph structure that incorporates the physiological connections of the infant and optimizing it using a Graph Convolutional Network (GCN), the system no longer views each keypoint in isolation. GCN allows keypoints to exchange information and coordinate corrections based on predefined physiological structures (such as rigid connections between the top of the head, jaw, and neck, and relative positional relationships between the shoulders and abdomen). This effectively corrects outlier errors in the initial localization and ensures that the overall pose estimation results conform to ergonomic principles.

[0072] If an infant's hand briefly covers their ear, causing the initial heatmap peak of the "right ear" keypoint to drift to the cheek, the node will receive information from its strongly connected neighbors (such as "top of head" and "right shoulder") during GCN optimization. These unoccluded keypoints have accurate coordinates, and the constraints they convey guide the features of the "right ear" node to adjust to the correct physiological position, thus largely overcoming the occlusion problem in a single frame and outputting more reasonable optimized coordinates.

[0073] By introducing time-series estimators such as Kalman filters, the image observations of each frame are combined with motion model-based predictions, which smooths out inter-frame coordinate jitter caused by image noise, lighting flicker, or minor fluctuations in the detection model itself. This ensures that the final output keypoint trajectory is smooth, continuous, and conforms to the laws of inertial motion, providing high-quality data for subsequent analysis modules that require stable input (such as head pose analysis and respiration monitoring).

[0074] When an infant is sleeping peacefully, their abdomen rises and falls slowly with breathing. Without temporal filtering, the coordinates of key points on the abdomen may exhibit irregular pixel jumps between frames, resulting in the extracted respiratory signal containing a large amount of high-frequency noise. After Kalman filtering, these random jitters are effectively suppressed, resulting in a smooth trajectory that accurately reflects the slow displacement caused by breathing, greatly improving the input signal quality of the respiratory monitoring module.

[0075] In step S4, the specific steps for calculating respiratory amplitude and respiratory rate include: a) Dynamically extract the region of interest in the abdomen based on key points on the abdomen, extract the pixel intensity or optical flow features in the region, and generate the original respiratory signal. b) Perform detrending processing and bandpass filtering on the original signal to obtain a clean breathing waveform signal; c) Extract respiratory frequency through spectrum analysis, calculate signal peak-to-peak value or extract respiratory amplitude, and detect whether the respiratory rhythm is abnormal.

[0076] By combining "dynamic adaptive tracking, strong anti-interference signal purification and multi-dimensional intelligent analysis", high-precision, non-contact extraction of infant respiratory parameters and early risk identification are achieved in complex real-world scenarios.

[0077] Dynamic adaptive capture (corresponding to step a) ensures that the monitoring area is always accurately locked on the abdomen, regardless of the baby's movements, thus guaranteeing the reliability of the signal source from the source and solving the problem of monitoring failure caused by body movement.

[0078] Strong anti-interference signal processing (corresponding to step b): By detrending and bandpass filtering, interference from changes in ambient light, overall movement and high-frequency noise is effectively removed, and pure physiological waveforms that only reflect respiratory movements are extracted, laying the foundation for accurate calculation.

[0079] Multidimensional intelligent analysis and early warning (corresponding to step c) not only extracts the regular respiratory rate and amplitude, but also innovatively integrates the detection of abnormal respiratory rhythms (such as pauses and disturbances), and can identify early physiological signs that may be caused by choking on milk, earlier than obvious changes in respiratory rate.

[0080] In step S5, the specific steps of the voiceprint recognition and analysis module include: a) Perform frame segmentation and windowing on the audio signal, extract Mel frequency cepstral coefficients or Mel spectrogram features, and form a time-series feature sequence; b) Input the temporal feature sequence into a pre-trained neural network classifier and output the probability distribution of each time frame belonging to a preset category, which includes infant crying, infant coughing, adult speech and environmental noise. c) Determine whether an abnormal sound event of the target infant has occurred based on the probability distribution, and output the classification result.

[0081] Transforming raw audio into credible and semantically clear anomalous sound events enables intelligent auditory perception to accurately identify key audio evidence of choking risk from complex acoustic environments.

[0082] In step S6, the specific steps of the fusion algorithm include: a) Align and encode the temporal features from respiratory monitoring, head posture analysis, and voiceprint recognition on the time axis, converting them into a sequence of feature vectors of uniform dimension; b) The encoded multimodal features are fused. The fusion methods include feature concatenation followed by mapping through a fully connected network, or weighted fusion based on an attention mechanism. c) Extract deep risk features from the fused features, input them into a classifier or regressor to calculate a comprehensive risk score, and map them to discrete risk levels or continuous risk probability values.

[0083] The risk levels include at least three levels: normal, suspicious, and dangerous; when the risk level is dangerous, an emergency alarm notification is triggered and pushed to the guardian's mobile terminal. After an alarm is triggered, the system automatically saves the image and audio data within a preset time period before and after the trigger, generating an abnormal event playback record.

[0084] Breathing, posture, and sound data differ in physical meaning, frequency, and start time. This step ensures their comparability in the time dimension by aligning them along the time axis, and then converts them into a unified feature vector through encoding.

[0085] Suppose the system detects a breathing pause (from the breathing monitoring module), a sudden head tilt (from the head posture analysis module), and a baby coughing sound (from the voiceprint recognition module). Without alignment, these might be considered three separate events. By aligning them, the system can accurately determine that these three events occurred densely within the same time window, significantly improving the reliability of the judgment.

[0086] Attention-based adaptive fusion (step b): This simulates the focusing process of human judgment, dynamically evaluating the weight of different pieces of evidence. In quiet nighttime environments, the weight of abnormal sounds (such as coughing) is automatically increased by the attention mechanism because auditory signals are clear and less interfered with. Conversely, in noisy daytime environments or when infants are actively moving, the weight of visual signals (such as abnormal breathing waveforms or sudden changes in posture) increases, while auditory signals may be partially deweighted due to environmental noise. This dynamic focusing capability allows the system to maintain robustness in judgments in complex and changing environments, effectively reducing false alarms and false negatives.

[0087] The tiered risk output and closed-loop data preservation (step c and subsequent steps) provide guardians with clear response guidelines. Simultaneously, audio and video data from before and after an alarm is triggered are automatically saved for a period of time.

[0088] For example: 1. Tiered response: When the system determines something is "suspicious," it may only push a mild notification to the guardian's mobile app, reminding them to observe carefully and avoid unnecessary stress. However, when it determines something is "dangerous," it will immediately trigger an emergency audible and visual alarm and push a strong notification, requiring immediate intervention.

[0089] 2. Data traceability: After a "danger" alarm, the saved video and audio recordings allow caregivers to review what happened (e.g., how the baby choked on milk and how long it lasted). This not only helps in taking the correct first aid measures, but can also be used for post-incident review to understand whether there were any problems in feeding or care.

[0090] This fusion algorithm not only achieves more accurate risk identification in terms of technology, but also builds a powerful closed-loop advantage in terms of user experience and system evolution.

[0091] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A keypoint recognition and voiceprint analysis based infant choking or spitting up monitoring system, characterized in that, The method comprises the following steps: An image acquisition module is used to continuously acquire visual images of the baby; An audio acquisition module is used to acquire audio signals of the environment in which the baby is located; A key point detection module is used to extract key point information of the baby's head and abdomen from the visual images based on a deep learning pose estimation model; A head posture analysis module is used to calculate the change in head posture angle based on the key point information of the baby's head; A breathing monitoring module is used to calculate the breathing amplitude and breathing frequency by analyzing the change in pixel value over time based on the key point information of the abdomen or the image sequence of the abdominal region; A voiceprint recognition and analysis module is used to extract voiceprint features from the audio signals, and to distinguish between baby voices and adult voices based on a classification model, and to identify pre-set abnormal sound types; A multi-modal risk judgment module is used to fuse the head posture angle change, breathing amplitude and frequency information, and the output of the voiceprint recognition and analysis module, and to calculate the risk level of choking or spitting based on a multi-modal fusion algorithm; An alarm module is used to trigger an alarm notification when the risk level exceeds a pre-set threshold.

2. The keypoint recognition and voiceprint analysis based infant choking or spitting-up monitoring system according to claim 1, wherein, The deep learning pose estimation model used by the key point detection module is a model constructed based on a convolutional neural network or a graph convolutional network, which detects at least the following head key points: crown, chin, left ear, right ear, and mouth key points; and at least the following abdominal key points: upper boundary and lower boundary key points for defining the abdominal region. The key point detection module realizes key point detection through the following steps: a.Feature extraction and preliminary positioning: input the input baby image I into the feature extraction backbone network constructed based on the convolutional neural network CNN, output multi-scale feature maps {F l | l=1,...,L};Subsequently, through the key point prediction head, for each pre-defined head and abdominal key point i, generate its heat map H i , the peak position of the heat map , that is, the initial coordinates of the key point; b.Keypoint optimization based on graph model: a graph structure G=(V, E) is constructed, in which the key points are nodes and the connections between the physiological structures of the baby are edges; the initial coordinate set is optimized by using graph convolution network GCN, and the message passing and update formula can be expressed as: . . wherein, is the initial feature of node i in the k-th layer, is composed of the preliminary coordinates of the key points and the corresponding heat map confidence, represents the neighbor node set of node i, is an activation function; finally, the node features are mapped to coordinate offsets Δp i by the output layer, and the optimized coordinates are obtained; c. Timing consistency filtering: For video sequences, let p i (t), a Kalman filter or a first-order low-pass filter is introduced to smooth it to suppress inter-frame jitter, whose update formula is: ； wherein, is the smoothed coordinate, A is the state transition matrix, H is the observation matrix, and K(t) is the Kalman gain matrix; this step is used to realize the continuity of the key point trajectory in the time dimension and the rationality of the physiological movement.

3. The keypoint recognition and voiceprint analysis based monitoring system for infant choking or spitting up according to claim 1 or 2, characterized in that, The breathing monitoring module is realized by the following method: locating the abdominal region in consecutive image frames, extracting the pixel intensity or optical flow features of the abdominal region to form a time sequence, and filtering and peak detecting the time sequence to calculate the breathing frequency and relative amplitude, specifically: a) Breathing signal generation: based on the upper and lower boundaries of the abdominal region output by the key point detection module, dynamically intercepting the abdominal region of interest in consecutive image frames; by calculating the average intensity value of the pixels in a specific color channel in this region, or calculating the average value of the vertical component of the optical flow of the region between consecutive frames, an original breathing-related one-dimensional time sequence signal Sraw(t) is generated; b) Signal preprocessing: detrending Sraw(t) to eliminate baseline drift and band-pass filtering with passband frequency range set to [f min ,f max ] Hz to obtain a clean respiratory waveform signal R(t); c) Breathing parameter calculation: Breathing rate: By performing a spectral analysis or an autocorrelation analysis on R(t) in a sliding time window, the dominant frequency thereof is extracted as the instantaneous breathing rate f r ; Breathing amplitude: calculating the peak-to-peak value or signal energy of R(t) within a sliding time window as a measure of breathing amplitude Ar; Rhythm abnormality detection: by analyzing the interval time variation of consecutive breathing cycles, or detecting whether the amplitude of R(t) is continuously below a set threshold for more than a predetermined time, to determine whether there is apnea or rhythm disorder.

4. The keypoint recognition and voiceprint analysis based infant choking or spitting up monitoring system of claim 1, wherein, The voiceprint recognition and analysis module comprises: A feature extraction unit is used to extract Mel-frequency cepstral coefficients, Mel spectrogram or sound spectrogram features from the audio signals; A sound classification unit uses a trained neural network classifier to classify the extracted features to distinguish between baby cries, baby choking coughs, adult voices and environmental noise; The voiceprint recognition and analysis module is realized by the following steps: a) Audio preprocessing and feature extraction: the collected audio signal is framed and windowed, and the mel-frequency cepstral coefficient (MFCC) feature vector or the mel-spectrogram of each frame of audio signal is extracted to form a time series of features F(t); b) Sound event classification: input the sequence of temporal features F(t) into a pre-trained neural network classifier; the classifier is based on a convolutional neural network or recurrent neural network architecture, outputting a probability distribution P(t) = [p c (t)] for each time frame belonging to pre-defined classes c, where classes c include infant cry, infant cough, adult speech, and environmental noise; b) Sound event classification: input the sequence of temporal features F(t) into a pre-trained neural network classifier; the classifier is based on a convolutional neural network or recurrent neural network architecture, outputting a probability distribution P(t) = [p c (t)] for each time frame belonging to pre-defined classes c, where classes c include infant cry, infant cough, adult speech, and environmental noise; c) Decision and output: based on the probability distribution P(t), it is determined whether the target infant abnormal sound event occurs in the current audio segment, and the classification result is generated in combination with the context information and output to the multi-modal risk judgment module.

5. The keypoint recognition and voiceprint analysis based infant choking or spitting up monitoring system according to claim 1, wherein, The fusion algorithm used by the multi-modal risk judgment module is a neural network fusion model based on an attention mechanism, a gated fusion network, or a gradient boosting decision tree model, which inputs time series feature vectors from different modalities and outputs discrete values or continuous probability values representing the risk level of choking or spitting milk; The multi-modal risk judgment module is implemented by the following steps: a) Multi-modal feature alignment and encoding: receiving the respiratory parameter time series data from the respiratory monitoring module, the posture angle time series data from the head posture analysis module, and the infant abnormal sound event sequence from the voiceprint recognition and analysis module; aligning the heterogeneous time series features on the time axis, and converting them into uniform feature vector sequences through feature encoders respectively; b) Feature fusion and risk feature extraction: the encoded multi-modal feature vectors are fused; the fusion method includes feature splicing followed by fully connected network mapping, or weighted fusion based on an attention mechanism; deep neural networks are used to extract deep risk features highly related to choking or spitting milk risk from the fused features; c) Risk level determination and output: the extracted deep risk features are input into a classifier or regressor to calculate the comprehensive risk score S at the current time, and the score is mapped to a discrete risk level or output as a continuous risk probability value according to a pre-set threshold interval.

6. A monitoring method of the infant choking or spitting-up intelligent monitoring system based on key point recognition and voiceprint analysis according to any one of claims 1 to 5, characterized in that, The method comprises the following steps: S1: continuously acquiring visual images of the infant through an image acquisition module, and acquiring audio signals of the environment in which the infant is located through an audio acquisition module; S2: using a key point detection module to extract key point information of the infant's head and abdomen from the visual images based on a deep learning pose estimation model; S3: using a head posture analysis module to calculate the change in head posture angle based on the head key point information; S4: using a respiratory monitoring module to calculate the respiratory amplitude and frequency based on the abdominal key point information or the sequence of abdominal region images by analyzing the change in pixel value over time; S5: using a voiceprint recognition and analysis module to extract voiceprint features from the audio signals, and distinguishing between infant sounds and adult sounds based on a classification model, as well as identifying pre-set abnormal sound types; S6: using a multi-modal risk judgment module to fuse the head posture angle change information, respiratory amplitude and frequency information, and output information from the voiceprint recognition and analysis module, and calculating the risk level of choking or spitting milk through a multi-modal fusion algorithm; S7: when the risk level exceeds a pre-set threshold, triggering an alarm notification through an alarm module.

7. The monitoring method according to claim 6, characterized in that, In the step S2, the specific steps of extracting key points based on a deep learning pose estimation model include: a) Feature extraction and preliminary positioning: input the infant image into the feature extraction network based on the convolutional neural network, output multi-scale feature maps; generate a heat map for each pre-defined head and abdomen key point through the key point prediction head, and the peak position of the heat map is taken as the initial coordinate of the key point; b) Key point optimization based on graph model: construct a graph structure with key points as nodes and infant physiological structure connections as edges, and use the graph convolution network to optimize the initial coordinates to obtain the optimized key point coordinates; c) Temporal consistency filtering: perform temporal smoothing processing on the coordinates of each key point in the video sequence, use Kalman filter or low-pass filter to suppress inter-frame jitter, and obtain the smoothed key point trajectory.

8. The monitoring method of claim 6, wherein, In the step S4, the specific steps of calculating the respiratory amplitude and respiratory frequency include: a) Based on the dynamic extraction of the abdomen key point, the abdomen region of interest is extracted, the pixel intensity or optical flow features in the region are extracted, and the original respiratory signal is generated; b) The original signal is de-trended and band-pass filtered to obtain a pure respiratory waveform signal; c) The respiratory frequency is extracted through spectral analysis, the respiratory amplitude is calculated, and the respiratory rhythm is detected for abnormality.

9. The monitoring method of claim 6, wherein, In the step S5, the specific steps of the voiceprint recognition and analysis module include: a) Frame and window processing of the audio signal, extraction of mel-frequency cepstral coefficient or mel-frequency spectrum feature, formation of time sequence feature sequence; b) Input the time sequence feature sequence into the pre-trained neural network classifier, output the probability distribution of each time frame belonging to the pre-set category, and the category includes infant crying, infant coughing, adult voice and environmental noise; c) According to the probability distribution, it is judged whether the target infant abnormal sound event occurs, and the classification result is output.

10. The monitoring method of claim 6, wherein, In the step S6, the specific steps of the fusion algorithm include: a) Align and encode the time sequence features from respiratory monitoring, head posture analysis and voiceprint recognition on the time axis, and convert them into a uniform dimension feature vector sequence; b) Fuse the encoded multi-modal features, including feature splicing and mapping through a fully connected network, or weighted fusion based on an attention mechanism; c) Extract deep risk features from the fused features, input the classifier or regressor to calculate the comprehensive risk score, and map it to discrete risk levels or continuous risk probability values; The risk level at least includes three levels: normal, suspicious and dangerous; when the risk level is dangerous, trigger an emergency alarm notification and push it to the guardian's mobile terminal; After triggering the alarm, automatically save the image and audio data within a preset time before and after triggering the alarm to generate an abnormal event playback record.

Images