A method and system for monitoring infant sleep by integrating visual and audio analysis

Abstract

This invention provides an infant sleep monitoring method and system integrating visual and audio analysis, belonging to the fields of smart home and image processing technology. It includes the following steps: S1: Acquire the original video and audio of the infant's sleep, perform preprocessing to obtain encrypted facial feature information and voiceprint information; S2: Identify the infant's key point coordinates using the YOLOv8-Pose network model, classify the infant's posture, and classify the infant's audio using the MiniResNetV2 model; S3: Output alarm information to a remote monitoring terminal using a multimodal risk analysis model. This invention enables 24 / 7, low-latency, and highly privacy-protected infant sleep monitoring, improving monitoring reliability while reducing the burden on caregivers.

Description

Technical Field

[0001] This manual relates to the fields of smart home and edge computing technology, and in particular to a method and system for monitoring infant sleep that integrates visual and audio analysis. Background Technology

[0002] Infants and young children may face serious risks such as suffocation, abnormal breathing, or even sudden death during sleep, and parents cannot monitor them continuously 24 hours a day. Smart monitoring devices have therefore become a key tool for ensuring family safety, seamlessly covering the needs of day and night monitoring.

[0003] Existing products typically rely on cloud analytics, enabling only single-modal recognition of video or crying sounds. This poses a risk of data loss during network outages and uploads private audio and video to the cloud, increasing the possibility of leakage and misuse. Relying solely on a single modality can also lead to false alarms or missed alarms, lacking the ability to intelligently fuse and judge multi-source data (video + audio).

[0004] Therefore, there is an urgent need in this field for a 24 / 7 edge computing monitoring system that can achieve real-time judgment and trigger multi-channel alarms through continuous multimodal data acquisition and local AI inference, so as to significantly improve the timeliness, accuracy and privacy protection of infant sleep monitoring. Summary of the Invention

[0005] This specification provides an embodiment of an infant sleep monitoring method and system that integrates visual and audio analysis, aiming to solve the problems of existing home monitoring systems that rely on cloud inference, only support single-modal recognition, and suffer from monitoring interruptions and privacy leaks due to network instability, thereby improving the timeliness, accuracy, all-weather reliability, and data security of real-time monitoring.

[0006] In some embodiments, an infant sleep monitoring method integrating visual and audio analysis includes the following steps:

[0007] S1: Acquire the original video and audio of the infant's sleep, perform preprocessing, and obtain encrypted facial feature information and voiceprint information features;

[0008] S2: Use the YOLOv8-Pose network model to identify the key point coordinates of infants and toddlers, classify the infants' and toddlers' postures, and use the MiniResNetV2 model to classify the infants' and toddlers' audio.

[0009] S3: Output alarm information to the remote monitoring terminal through the multimodal risk analysis model.

[0010] Using the aforementioned technical methods, the YOLOv8-Pose network model is employed to identify the coordinates of key points (such as head, shoulders, knees, etc.) of infants in the incoming video stream in real time. Based on the location of the key points and their spatial relationships, the posture recognition is classified into states such as: face occlusion (e.g., face covered by a blanket or lying face down), blanket covering the area below the neck, blanket covering the lower limbs, blanket being kicked off, and infants lying on their side.

[0011] Further, the preprocessing includes obtaining encrypted facial features by re-synthesizing the video using 3D modeling technology, and obtaining encrypted voiceprint information features by converting the original audio into predefined prompts.

[0012] In order to effectively reduce the risk of privacy leakage during remote monitoring and ensure that guardians receive timely notifications of important events, the real-time video stream is re-synthesized using 3D modeling technology, and audio events are converted into predefined notification sounds to avoid leakage of infants' facial and voiceprint information.

[0013] Furthermore, the YOLOv8-Pose network model is used to infer 2D pose key points in the video stream in real time, and these key points are then input into the PoseLifter network for processing to generate absolute 3D pose key points relative to the camera coordinate system. The generated 3D pose key points are then pushed to a remote monitoring terminal. The received 3D pose key points are used to reconstruct the human skeleton model, which is then rendered and displayed. The data is packaged into a message and sent via the lightweight communication protocol MQTT.

[0014] Using the aforementioned technical means, the YOLOv8-Pose network model identifies the coordinates of key points (such as head, shoulders, knees, etc.) of infants in the incoming video stream in real time. Based on the location of the key points and their spatial relationships, the posture recognition is classified into states such as: face occlusion (e.g., face covered by a blanket or lying face down), blanket covering the area below the neck, blanket covering the lower limbs, blanket kicked off, and infant lying on their side.

[0015] Furthermore, a 1-second sliding window with a sampling rate of 16kHz and a frame shift of 20ms was obtained and used as the input to the MiniResNetV2 model. The specific feature shape is 64 Mel frequency bands and 50 frames of time axis.

[0016] Using the aforementioned techniques, the system employs a lightweight neural network, MiniResNetV2, to convert the incoming audio into a Mel spectrogram, which is then used as input to the model for audio event detection. The categories covered include infant crying, infant laughter, coughing, and the sound of objects falling. Through transfer learning and fine-tuning of the pre-trained model, and by using a self-collected infant audio dataset, the system enhances its recognition accuracy, achieving precise classification of audio events based on spectral features. Audio data first enters a sliding buffer and is then fed into the model for infant crying detection. When the model determines that crying is present, the system backtracks to the buffer, extracts the corresponding audio segment, and calculates its RMS volume and dominant frequency features to enhance the interpretability and effectiveness of the crying events. Optionally, the system also analyzes the rhythmic features of the crying (such as the interval between cries) to dynamically adjust the playback frequency and rhythm of the prompts.

[0017] Furthermore, including the MTCNN network model, by calculating the aspect ratio of both eyes... ,

[0018]

[0019] In the formula, BF and CE are the vertical distances between the upper and lower eyelids, respectively, and AD is the horizontal distance between the eyes; when both eyes When the average value is greater than a preset threshold, it is determined that both eyes are open.

[0020] An infant sleep monitoring system integrating visual and audio analysis, with an edge microcontroller including multiple recognition units and a multimodal risk analysis model, preprocesses the original video and original audio, and pushes the recognition results, privatized video streams and privatized audio events to a remote monitoring terminal;

[0021] The video capture module is used to acquire real-time video streams of infants and young children;

[0022] The audio acquisition module is used to simultaneously collect ambient noise and the sound information of infants and young children;

[0023] The remote alarm module triggers an alarm signal based on the output results of the multimodal risk analysis model, and pushes the alarm information to a preset remote monitoring terminal through the communication module.

[0024] Furthermore, the remote monitoring terminal is an APP set on a mobile terminal, which can obtain alarm signals, private video streams and private audio events associated with the account after logging in.

[0025] Furthermore, including the MTCNN network model, by calculating the aspect ratio of both eyes... ,

[0026]

[0027] In the formula, BF and CE are the vertical distances between the upper and lower eyelids, respectively, and AD is the horizontal distance between the eyes; when both eyes When the average value is greater than a preset threshold, it is determined that both eyes are open.

[0028] The beneficial effects of this invention are:

[0029] 1. Based on a posture recognition model (such as YOLOv8-Pose), the system processes video images and determines the infant's current sleeping posture. When the recognition result is an abnormal posture (such as sleeping on one's stomach, face obscured, etc.), the system immediately triggers an alarm.

[0030] 2. Protect the facial and vocal privacy of infants and young children under 24 / 7 monitoring to avoid leaking information about guardians or infants and young children.

[0031] 3. 24 / 7 local inference, improving monitoring reliability: The system deploys video / audio acquisition, key point recognition and event judgment functions on the edge microcontroller to achieve 24-hour real-time monitoring and has the ability to alarm locally when the network is offline, which significantly solves the network blind spot problem of existing systems that rely on cloud inference.

[0032] 4. Before performing 3D modeling, encrypt the original video stream to prevent the video stream on the device from being copied to the cloud. Attached Figure Description

[0033] This specification will be further described by way of exemplary embodiments, which will be described in detail with reference to the accompanying drawings. These embodiments are not limiting; in these embodiments, the same reference numerals denote the same structures, wherein:

[0034] Figure 1 This is a schematic diagram of a sleep monitoring system according to some embodiments of this specification;

[0035] Figure 2 This is a schematic diagram illustrating the working principle according to some embodiments of this specification;

[0036] Figure 3 This is a schematic diagram of key eye points according to some embodiments of this specification. Detailed Implementation

[0037] To more clearly illustrate the technical solutions of the embodiments in this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely some examples or embodiments of this specification. For those skilled in the art, these drawings can be applied to other similar scenarios without creative effort. Unless obvious from the context or otherwise specified, the same reference numerals in the drawings represent the same structures or operations.

[0038] It should be understood that the terms “system,” “device,” “unit,” and / or “module” used herein are one way to distinguish different components, elements, parts, sections, or assemblies at different levels. However, if other terms can achieve the same purpose, they may be replaced by other expressions.

[0039] As indicated in this specification and claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" do not specifically refer to the singular and may also include the plural. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of expressly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements.

[0040] Flowcharts are used in this specification to illustrate the operations performed by the system according to embodiments of this specification. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, the steps can be processed in reverse order or simultaneously. Furthermore, other operations can be added to these processes, or one or more steps can be removed from them.

[0041] Example:

[0042] Please refer to Figure 1 This embodiment includes an edge microcontroller, a video acquisition module, an audio acquisition module, a remote alarm module, and a remote terminal APP module.

[0043] The edge microcontroller, video acquisition module, audio acquisition module, and remote alarm module are usually integrated together and fixedly installed on the baby bed. The position is usually adjusted so that a clear picture of the baby can be captured. The processor is then started to complete the networking and parameter initialization of each module.

[0044] The video acquisition module is used to capture video streams of infants and young children while they are sleeping. The acquired video streams are preprocessed and then sent to the edge microcontroller. Similarly, the audio acquisition module is used to capture sounds that infants and young children may make while sleeping or awake, as well as loud environmental noises, and then send them to the edge microcontroller for processing.

[0045] In particular, the edge microcontroller is equipped with multiple AI deep learning models. This monitoring system uses the YOLOv8-Pose network model to identify the coordinates of key points (such as head, shoulders, knees, etc.) of infants in the incoming video stream in real time. Based on the location of key points and their spatial relationships, the posture recognition is classified into states such as: face occlusion (e.g., face covered by a blanket or lying face down), blanket covering the area under the neck, blanket covering the lower limbs, blanket kicked off, and infant lying on their side.

[0046] Furthermore, the system employs the lightweight neural network MiniResNetV2 to convert the input audio into a Mel spectrogram, which is then used as input to the model for audio event detection. The categories covered include: infant crying, infant laughter, coughing, and the sound of objects falling. Through transfer learning and fine-tuning of the pre-trained model, and using a self-collected infant audio dataset, the model's recognition accuracy is enhanced, achieving precise classification of audio events based on spectral features.

[0047] To effectively reduce the risk of privacy leaks during remote monitoring and ensure that guardians receive timely notifications of important events, real-time video streams are re-synthesized using 3D modeling technology, and audio events are converted into predefined notification sounds to avoid the leakage of infants' facial and voiceprint information.

[0048] This privacy protection mechanism ensures guardians are promptly informed of unusual events while maximizing the privacy and safety of infants and toddlers. Furthermore, to meet temporary needs in specific situations, the system allows guardians to actively choose to view the original video or audio within their terminal app. However, this function is entirely initiated by the guardian; the system will record the request process and grant access only when permitted by the protection mechanism, thus achieving the design principle of "privacy first by default, transparency only on demand."

[0049] This also includes encrypting the raw data during video capture or preprocessing stages (such as before frame extraction and encoding) to ensure data security from the source. This involves block encryption of the pixel data of video frames (e.g., using the AES-256 algorithm), dividing each frame into fixed block sizes (e.g., 16×16 pixels), and encrypting the block data with a symmetric key. The encryption key can be generated and stored through a Hardware Security Module (HSM) or a Trusted Execution Environment (TEE, such as a mobile phone's Secure Enclave) to avoid exposing the key in plaintext. Key frames (e.g., I-frames) are heavily encrypted, while non-key frames (P-frames / B-frames) can use lightweight encryption (e.g., XOR operation combined with a dynamic key), balancing security and processing efficiency (suitable for real-time video processing scenarios). Encrypted file systems (e.g., Linux's eCryptfs, Windows' BitLocker) or dedicated encryption tools (e.g., VeraCrypt) are used to encrypt the entire video file before storage. Decryption requires authorization via password, hardware key (e.g., USB key), or biometric authentication (fingerprint / face recognition). After the historical video streams are processed, they are encrypted and stored as described above, making it convenient for guardians to retrieve the images of infants and young children later.

[0050] Furthermore, during the audio acquisition phase, this system employs a 16kHz sampling rate and a 20ms frame shift to form a sliding window of approximately 1 second in length, used as input to the MiniResNetV2 model. Its feature shape is 64 Mel frequency bands × 50 frames time axis. Audio data first enters the sliding buffer and is then fed into the model for infant cry detection. When the model determines that crying behavior is present, the system backtracks to the buffer, extracts the corresponding audio segment, and calculates its RMS volume and dominant frequency characteristics to enhance the interpretability and effectiveness of the crying event. Optionally, the system also analyzes the rhythmic characteristics of the crying (such as the interval between cries) to dynamically adjust the playback frequency and rhythm of the prompt tone.

[0051] Furthermore, the system uses the YOLOv8-Pose model to infer 2D pose keypoints in the video stream in real time, and then inputs them into the PoseLifter network for processing to generate absolute 3D pose keypoints relative to the camera coordinate system. The system then pushes the generated 3D pose keypoint data to a remote cloud or app. On the remote end, the received 3D pose keypoint data is used to reconstruct the human skeletal model and render it for display.

[0052] To reduce power consumption on the local device, the above information is packaged into a message and transmitted via the lightweight communication protocol MQTT. At the remote end, the received 3D pose keypoint data is used to reconstruct the human skeletal model and render it. A "beep beep" notification sound is resynthesized based on the audio information to provide real-time feedback.

[0053] Furthermore, the system deploys an MTCNN network to perform facial detection and key facial feature recognition for infants, including the eyes, nose, and corners of the mouth, and further determines whether the infant is awake by opening their eyes using the following methods:

[0054] Please refer to Figure 3 Calculate the aspect ratio of both eyes ( ): Where BF and CE are the vertical distances between the upper and lower eyelids, and AD is the horizontal distance; when When the average value exceeds a preset threshold, the system determines that the baby's eyes are open, thus identifying that the baby is awake. This recognition mechanism implements a complete process from facial detection to eye status determination and can respond to the baby's waking signals in real time.

[0055] The decision-making mechanism for multimodal fusion judgment in the remote alarm module (multimodal risk analysis model): Please refer to... Figure 2 ,

[0056] (1) Based on the posture recognition model (e.g., YOLOv8-Pose), the video image is processed and the current sleeping posture of the infant is determined. When the recognition result is an abnormal posture (e.g., lying on the stomach, face occlusion, etc.), the system immediately triggers a level one alarm.

[0057] (2) Real-time analysis of audio data; when an infant crying event is detected, a level one alarm is also triggered.

[0058] (3) The system identifies whether the infant is awake based on the MTCNN model. If the infant is awake during the set sleep period, the system enters the second-level warning state and continuously monitors the duration of the state. If the preset threshold is exceeded, the system will automatically upgrade to the first-level alarm.

[0059] (4) When all three modal judgments are in normal condition, the system maintains normal working mode, only records and displays data, and does not perform any alarm actions.

[0060] Furthermore, based on the aforementioned decision-making mechanism, a decision priority order is also designed. When a sound is detected from an infant, an alarm message is directly sent to the remote terminal. An alarm message is also triggered when the posture cannot identify complete facial features. These two situations are ranked as first priority and second priority, respectively. When neither of the above two situations occurs, posture recognition and sound monitoring continue. The advantage of this embodiment compared to the prior art is that the guardian can prioritize the monitored events on the remote terminal APP, listing various behaviors and sounds related to the infant as separate features. The guardian can then prioritize these features after logging into their account and arrange the alarm order according to the ranking.

[0061] Furthermore, the remote terminal APP module enables complete remote interactive control of the system through a dedicated application APP installed on a smartphone or tablet: Real-time monitoring: The APP receives the video stream (3D skeletal model or binarized image) and prompts processed by the edge microcontroller, and displays the image interface in real time. Users can switch between sound and image modes via the "Audio and Video" tab; Two-way voice reassurance: After the user clicks the "Speak" button, the recorded voice is sent to the system device via the network, and the system plays the voice content through the speaker to achieve remote reassurance function;

[0062] Camera PTZ control: The APP controls the camera's tilt and horizontal movement using the directional keys on the interface; Multi-guardian management: The main account can grant monitoring permissions to family members or guardians' accounts through the "Device Sharing" function; Each account can view video, voice interaction, and alarm records after logging in;

[0063] Network outage warning mechanism: When the device detects a Wi-Fi or cellular network outage, it immediately triggers an offline reminder push notification on the APP; if an abnormality is detected during the network outage (such as crying), the device's local speaker will emit a beeping alert, and the APP will record the event and push a notification after the connection is restored.

[0064] In summary, this application protects privacy while fulfilling functionality: it uses 3D skeletal reconstruction or binarization to replace the original video and converts infant voices into prompts, pushing only anonymized data to protect facial and voice privacy while ensuring that caregivers can still obtain valid information through prompts. It also offers flexible remote interaction and access control: the remote terminal APP module supports real-time monitoring, two-way reassurance, PTZ control, multi-account sharing, and network outage alerts, balancing monitoring needs with device security management and improving user experience. Furthermore, it is low-cost, easy to deploy, and adaptable to home environments: the system utilizes lightweight AI models (MiniResNetV2, YOLOv8-Pose, MTCNN), which are low-cost and energy-efficient, facilitating rapid installation and long-term operation in home environments.

[0065] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device, and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0066] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0067] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application.

[0068] The embodiments described in this application are preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Therefore, all equivalent changes made in accordance with the structure, shape and principle of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A method for monitoring infant sleep integrating visual and audio analysis, characterized in that, Includes the following steps: S1: Acquire the original video and audio of the infant's sleep, perform preprocessing, and obtain encrypted facial feature information and voiceprint information features; S2: The YOLOv8-Pose network model identifies key point coordinates of infants and toddlers, classifying their poses. It then uses the YOLOv8-Pose network model to infer 2D pose key points in the video stream in real time, inputting these into the PoseLifter network for processing to generate absolute 3D pose key points relative to the camera coordinate system. These generated 3D pose key points are pushed to a remote monitoring terminal. The received 3D pose key points are used to reconstruct the human skeletal model, render it, package it into a message, and send it via the lightweight communication protocol MQTT. Additionally, the MiniResNetV2 model, including the MTCNN network model, is used to classify infant and toddler audio by calculating the aspect ratio of each eye. , ； In the formula, BF and CE are the vertical distances between the upper and lower eyelids, respectively, and AD is the horizontal distance between the eyes; when both eyes The system determines that both eyes are open when the average value exceeds a preset threshold. S3: Output alarm information to the remote monitoring terminal through the multimodal risk analysis model.

2. The infant sleep monitoring method integrating visual and audio analysis as described in claim 1, characterized in that, Preprocessing includes obtaining encrypted facial features by re-synthesizing the video using 3D modeling technology, and obtaining encrypted voiceprint information features by converting the original audio into predefined prompts.

3. The infant sleep monitoring method integrating visual and audio analysis as described in claim 2, characterized in that, A 1-second sliding window with a sampling rate of 16kHz and a frame shift of 20ms was used as the input to the MiniResNetV2 model. The specific feature shape was 64 Mel frequency bands and 50 frames of time axis.

4. An infant sleep monitoring system integrating visual and audio analysis, used to execute the infant sleep monitoring method integrating visual and audio analysis as described in claim 3, characterized in that, Also includes The edge microcontroller, including multiple recognition units and a multimodal risk analysis model, preprocesses the original video and audio, and pushes the recognition results, privatized video streams, and privatized audio events to the remote monitoring terminal. The video capture module is used to acquire real-time video streams of infants and young children; The audio acquisition module is used to simultaneously collect environmental noise and the sound information of infants and young children; The remote alarm module triggers an alarm signal based on the output results of the multimodal risk analysis model, and pushes the alarm information to a preset remote monitoring terminal through the communication module.

5. The infant sleep monitoring system integrating visual and audio analysis as described in claim 4, characterized in that, The remote monitoring terminal is an app set up on a mobile terminal. After logging into the account, it can obtain alarm signals, private video streams, and private audio events associated with the account.

6. The infant sleep monitoring system integrating visual and audio analysis as described in claim 5, characterized in that, Including the MTCNN network model, which calculates the aspect ratio of both eyes. , ； In the formula, BF and CE are the vertical distances between the upper and lower eyelids, respectively, and AD is the horizontal distance between the eyes; when both eyes When the average value is greater than a preset threshold, it is determined that both eyes are open.

Images