Child intelligent development detection bracelet based on multi-mode mouth shape analysis

Abstract

The invention discloses an intelligent child development detection bracelet based on multi-modal mouth shape analysis. The bracelet comprises a wearable bracelet main body, the multi-mode sensor module comprises a visual sensor used for capturing mouth shape dynamic images and an audio sensor used for collecting voice; the data processing unit is used for preprocessing sensor data and executing multi-mode mouth shape analysis; the intelligent development evaluation module is used for generating a development score and a risk index according to the analysis result; the user interface module is used for displaying information and providing feedback; according to the invention, a high-resolution visual sensor and a multi-microphone array are integrated on the bracelet, so that multi-modal data collaborative acquisition of dynamic images and synchronous voices of mouth shapes of children is realized, and fusion analysis is carried out by adopting a model based on a convolutional neural network and a recurrent neural network. The time sequence relevance among the lip motion trail, the mouth opening amplitude and the voice frequency spectrum feature can be extracted, and therefore the problem of one-sidedness of single-mode monitoring is solved.

Description

Technical Field

[0001] This invention relates to the field of smart bracelet technology, and in particular to a smart development detection bracelet for children based on multimodal lip reading analysis. Background Technology

[0002] Currently, there are various child development monitoring solutions based on wearable devices. These solutions mainly use motion sensors to monitor children's gross motor development or analyze language abilities through single-modal audio acquisition. Some high-end devices also attempt to combine simple visual observation functions by recording children's daily behavior through miniature cameras worn on the head.

[0003] The aforementioned single-modal monitoring methods are insufficient to comprehensively assess fine developmental indicators related to mouth shape. For example, relying solely on audio analysis cannot distinguish whether unclear pronunciation is due to oral motor dysfunction or vocal cord problems. Furthermore, existing visual monitoring devices are mostly head-mounted, which are uncomfortable and easily resisted by children, resulting in incomplete data collection. In addition, traditional devices lack the ability to fuse and analyze multimodal data, making it impossible to establish the correlation between mouth shape movements and speech output, leading to a high misjudgment rate. Moreover, existing solutions generally lack real-time feedback mechanisms, failing to provide timely corrective guidance for children.

[0004] Therefore, in response to the problems mentioned above, this invention proposes a smart development detection bracelet for children based on multimodal lip-reading analysis. Summary of the Invention

[0005] To overcome the problems of existing children's development monitoring devices, such as lack of multimodal data in lip shape analysis, insufficient wearing comfort, and lack of real-time feedback, this invention proposes a children's intelligent development monitoring bracelet based on multimodal lip shape analysis. By integrating an optimized array of visual and audio sensors on the bracelet, combined with a multimodal data fusion algorithm and a real-time feedback mechanism, the developmental monitoring and assessment of children's lip shape can be realized.

[0006] The technical solution of this invention is: a children's intelligent development detection bracelet based on multimodal lip-reading analysis, comprising: The main body of the wearable bracelet is made of medical-grade silicone material; A multimodal sensor module, integrated into the wearable bracelet body, is used to collect real-time multimodal data related to children's mouth shapes. The multimodal sensor module includes: The vision sensor uses a miniature camera with 1080P resolution and 30fps sampling rate, equipped with a 120° wide-angle lens and an 850nm wavelength infrared fill light unit. The fill light intensity can be automatically adjusted according to the ambient light (adjustable from 1-100 lux) to ensure clear capture of lip-sync images under various indoor and outdoor lighting conditions. An audio sensor is used to synchronously collect children's voice data. The audio sensor uses a four-microphone array and includes adaptive beamforming and noise reduction algorithms to isolate environmental noise and focus on children's voice. The data processing unit, integrated into the main body of the wearable bracelet, is used for: A1 preprocesses the dynamic image sequence captured by the vision sensor, including image enhancement, frame alignment and motion compensation, to extract the lip region; A2 preprocesses the speech data collected by the audio sensor, including audio segmentation, feature extraction, and noise reduction. A3 performs multimodal lip-reading analysis, extracting lip-reading feature parameters by fusing visual and audio data, including lip movement trajectory, mouth opening amplitude, speech spectrum features, and temporal synchronization. The multimodal lip-reading analysis uses a deep learning model, which is jointly trained based on a CNN-BiLSTM hybrid neural network with an attention mechanism to identify the correspondence between lip-reading and speech. The model takes into account the optical flow features of the image sequence (extraction dimension: 256 dimensions) and the MFCC features of the audio (extraction dimension: 128 dimensions), and outputs a lip-reading-speech synchronization score (0-100 points) and abnormal pattern recognition. The intelligent development assessment module, integrated into the data processing unit, compares extracted lip shape feature parameters with a pre-stored database of children's intelligent development standards. This database covers standard lip shapes for children aged 3-12 months and includes over 200 developmental milestone indicators. Based on the comparison results, a developmental score and a risk index are generated. The risk index is used to identify potential developmental delays or disorders. The module also includes a personalized calibration unit, which adjusts the thresholds of the pre-stored database of children's intelligent development standards according to the child's individual characteristics, including age, gender, and baseline developmental level. The user interface module includes a touch screen and LED indicators, which are integrated into the wearable wristband body. The touch screen has a graphical user interface for displaying real-time visualization of lip shape analysis, developmental trend graphs, and interactive games to encourage children to participate in the testing. The LED indicators use color changes to indicate the developmental risk level. The communication module, integrated into the main body of the wearable bracelet, supports wireless communication protocols including Bluetooth and Wi-Fi, and is used to transmit the collected data and evaluation results to external devices or cloud servers; The power management module is used to power the entire bracelet; The real-time feedback module, integrated into the data processing unit, is used to generate instant interactive feedback through the user interface module when abnormal mouth shape or developmental risks are detected. This includes displaying standard mouth shape animations on the screen and playing voice prompts through the built-in speaker to guide children in corrective training. The data storage module, integrated into the main body of the wearable wristband, is used for local caching of collected multimodal data and evaluation results.

[0007] Preferably, the vision sensor also includes a near-infrared light source with a wavelength range of 700nm to 1000nm, used to enhance the contrast of the lip region under non-invasive conditions.

[0008] Preferably, the multi-microphone array in the audio sensor includes at least three microphones arranged in a spatial distribution and is equipped with a real-time sound source localization algorithm to accurately track the position of the child's mouth and improve the speech signal-to-noise ratio.

[0009] The beneficial effects of this invention are: 1. This invention integrates a high-resolution visual sensor and a multi-microphone array on a wristband to achieve multimodal data collaborative acquisition of dynamic images of children's mouth shapes and synchronous speech. It also employs a fusion analysis based on convolutional neural networks and recurrent neural networks to extract the temporal correlation between lip movement trajectory, mouth opening amplitude, and speech spectrum features. This solves the problem of the one-sidedness of single-modal monitoring and significantly improves the accuracy and reliability of identifying developmental abnormalities such as oral motor dysfunction and unclear pronunciation.

[0010] 2. By integrating a real-time feedback module, this invention can instantly display a standard mouth shape correction animation on a touchscreen and play synchronized voice prompts via a miniature speaker when abnormal mouth shape or developmental risks are detected. This provides children with intuitive, fun, and immediate guidance and interactive training, transforming traditional passive monitoring into active intervention. It helps to correct bad pronunciation habits in a timely manner and promotes the positive development of their language and oral motor abilities.

[0011] 3. This invention, through its built-in database covering a large number of developmental milestone indicators for each age group from 3 to 12 months, and combined with the personalized calibration unit and dynamic time warping algorithm in the intelligent development assessment module, can establish a personal developmental profile for each child and conduct personalized growth trend analysis and risk assessment. Attached Figure Description

[0012] Figure 1 The diagram shown is a schematic representation of the system framework of this invention. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0014] Please see Figure 1 This invention provides an embodiment of a children's intelligent development detection bracelet based on multimodal lip-reading analysis: In this embodiment, the main body of the wearable bracelet is made of medical-grade liquid silicone material, which has excellent biocompatibility and ensures safety during prolonged contact with children's skin. The bracelet size is designed for the wrist circumference of children aged 3-12.

[0015] In this embodiment, the multimodal sensor module will be described in detail: The module's visual sensor uses a 1080P resolution CMOS image sensor, paired with an optical lens with a 120° wide-angle field of view. With a sensor frame rate of 30fps, it can fully capture the dynamic changes in a child's lip movements. To cope with different lighting conditions, it integrates an 850nm wavelength infrared illumination unit, whose illumination intensity can be automatically adjusted within the range of 1-100 lux.

[0016] Furthermore, an anti-reflective coating can be applied to the lens surface to reduce glare and ghosting in bright light. The sensor also integrates a 940nm wavelength VCSEL infrared light source array to ensure clear imaging even at night or in low light without harming children's eyes.

[0017] The audio sensor employs a four-microphone array design, with the microphones spaced 12mm apart to form an equilateral triangle layout. Each microphone has a sampling frequency of 16kHz, a bit depth of 24bit, and a dynamic range of 110dB. Through hardware-level acoustic structure optimization, a signal-to-noise ratio exceeding 70dB is achieved.

[0018] In this embodiment, the data processing unit will be described in detail: In the data preprocessing stage, for visual data, a Haar-feature-based hierarchical classifier is first used to quickly locate the mouth region. Then, image distortion correction is performed using a Brown-Conrad model to correct lens distortion. Finally, contrast-limited adaptive histogram equalization is applied to enhance image details. Simultaneously, for synchronously acquired audio data, a generalized sidelobe canceller algorithm is first used for sound source separation. Combined with a reference microphone signal, this achieves environmental noise suppression of over 15dB. Echo cancellation is then performed using a normalized least mean square adaptive filter. Finally, 128-dimensional Mel-frequency cepstral coefficient features are extracted.

[0019] In the multimodal lip-reading analysis stage, the system employs a CNN-BiLSTM hybrid neural network model based on an attention mechanism for fusion and recognition. The model's input includes 256-dimensional optical flow features calculated from image sequences and 128-dimensional MFCC features extracted from audio. The network structure first extracts local spatiotemporal features of the lip-reading image through a CNN containing 5 layers of deep separable convolutions. Then, it connects to a bidirectional LSTM network with 2 layers and a hidden dimension of 512 to capture the long-term temporal dependency between lip movements and speech output. An 8-head self-attention mechanism is introduced to enable the model to automatically focus on important features at key time steps. Finally, the output layer uses a multi-task learning framework to simultaneously output classification results for 10 basic lip shapes, recognition results for 200 core words, and a lip-reading-speech synchronization score from 0 to 100. The model is trained using over 100,000 carefully labeled lip-reading-speech correspondence samples, covering data from children of different ages, genders, and pronunciation characteristics.

[0020] In this embodiment, the intelligent development assessment module is described in detail: This module includes a built-in database of oral development standards for children aged 3-12 months, containing 236 finely segmented developmental milestones. The evaluation algorithm employs an improved dynamic time warping algorithm, which introduces a weight adjustment mechanism to assign different weights based on the importance of different developmental indicators.

[0021] The assessment process includes: first, collecting mouth shape data from children for 7 consecutive days to establish a personalized developmental baseline profile; then, performing weighted dynamic time-normalized distance calculations on the real-time collected mouth shape feature parameters and standard patterns in the database to calculate a developmental score and a risk index identifying potential developmental delays or disorders; and finally, using the Bootstrap method to calculate the 95% confidence interval of the assessment results to avoid absolute judgments. This module also includes a personalized calibration unit, allowing parents or professionals to set high, medium, and low assessment sensitivities based on individual differences in children, thereby achieving personalized developmental trend analysis and risk assessment.

[0022] In this embodiment, the real-time feedback module will be described in detail: The real-time feedback module integrates a touchscreen and a miniature speaker, and provides two main feedback methods: (1) Display 20 standard mouth shapes in 3D animation via touch screen. The animation frame rate is kept at 60fps to ensure smoothness. The animation adopts a gamified design, such as designing the correct mouth shape imitation as a "Pac-Man" game, thereby increasing children's participation.

[0023] (2) Play 50 pre-stored pronunciation training contents through a miniature speaker. The audio uses binaural recording technology to ensure the sound is natural and real. The system will dynamically adjust the training difficulty according to the child's performance to achieve personalized training programs.

[0024] The graphical user interface is designed using the principles of color psychology, and risk warnings use a gradient color scheme, with green for normal, yellow for attention, and red for warning.

[0025] In this embodiment, the communication module supports Bluetooth 5.2 and Wi-Fi 6 protocols, with a maximum transmission rate of 150Mbps. The power management module uses a lithium polymer battery, providing up to 8.5 hours of battery life under typical usage scenarios and supporting 2-hour fast charging. The data storage module is equipped with 64GB of storage and employs an intelligent data filtering strategy, permanently saving abnormal data and retaining the most recent 32 days of monitoring data for normal data.

[0026] This invention provides a comparative example: This comparative study recruited 360 children aged 3-8 years, divided into three groups of 120 children each: 3-4 years, 5-6 years, and 7-8 years. The study included four groups of participants: The embodiments are those of the present invention described above; Comparative Example 1: A smart bracelet containing only an audio sensor; Comparative Example 2: A head-mounted device containing only a visual sensor.

[0027] This study employs a double-blind, randomized, controlled design, with each group of children randomly assigned to use different devices. The experimental period lasts for 3 months, with the first 7 days serving as the baseline establishment period. Subsequent periodic assessments are conducted to verify the lip-speech synchronization recognition scores of each group.

[0028]

[0029] As shown in the table above, the recognition accuracy of this invention is significantly higher than that of the comparative examples in all age groups. Comparative examples 1 and 2, due to using only single-modal data, are difficult to comprehensively evaluate the lip-speech synchronization.

[0030]

[0031] As shown in the table above, this invention performs best in terms of comfort and wearing compliance, while Comparative Example 2, being a head-mounted device, has the worst comfort and the highest refusal-to-wear rate. This indicates that the wristband design of this invention is more suitable for children's wearing needs.

[0032]

[0033] As shown in the table above, the multimodal feedback of this invention is significantly better than single-modal feedback in terms of children's cooperation, imitation accuracy, and sustained attention, indicating that multimodal feedback can more effectively promote children's participation in training.

[0034] This invention provides Embodiment 1: A four-year-old boy was initially observed to have problems with unclear pronunciation of velar consonants such as "g" and "k". After a week of baseline monitoring using the device of this invention, the system used the CNN-BiLSTM hybrid neural network model in the multimodal mouth shape analysis module to perform deep fusion analysis on the synchronously acquired dynamic mouth shape image sequence and speech data. The quantitative assessment showed that the mouth shape-speech synchronization score was only 62 points (far below the average level of children of the same age). The system also identified key abnormal patterns in his attempts to pronounce velar consonants, such as insufficient mouth shape movement amplitude, insufficient tongue root elevation, and insufficient soft palate elevation.

[0035] After identifying the root cause of the problem, the intelligent developmental assessment module immediately generated a personalized training plan, while the real-time feedback module immediately initiated the intervention process. Through a touchscreen, the module demonstrated a three-dimensional standard mouth shape animation specifically for correcting velar consonants to the child. The animation, in a gamified manner, showed the complete process of raising the root of the tongue, raising the soft palate, and forming an obstruction with the back of the soft palate step by step in slow motion. At the same time, clear and amplified standard pronunciation audio of "g" and "k" was played synchronously through a miniature speaker to guide the child to observe and imitate. When the child attempted to pronounce the sounds, the multimodal sensor captured the child's mouth shape and speech in real time, immediately providing specific voice prompts and animated encouragement such as "Raise the root of the tongue a little higher" or "Very good, that's it." This interactive approach greatly stimulated the child's participation and interest in imitation.

[0036] After six weeks of consistent, 20-minute daily training, a comparison of data before and after training revealed that the boy's tongue root elevation when pronouncing "g" and "k" improved by approximately 75%, and his mouth-speech synchronicity score increased from a baseline of 62 to 85. His parents also reported that their child's pronunciation of related syllables in daily communication became clearer and more understandable. This demonstrates that the personalized training program implemented through precise identification and real-time feedback in this invention has a very good effect on improving specific pronunciation disorders.

[0037] This invention provides Embodiment 2: A 5-year-old girl did not exhibit obvious pronunciation errors during daily observation, but her speech fluency showed subtle pauses. This invention quantified the transition time between different mouth shape units during a series of consecutive syllable pronunciations using an intelligent developmental assessment module and a dynamic time warping algorithm. Specific data showed that her average mouth shape transition time reached 420 milliseconds, which exceeded the average of children of the same age and gender in the database by 2.3 standard deviations. This identified that her core problem was insufficient coordination and agility of mouth movements, rather than articulation errors of specific phonemes.

[0038] After diagnosing that the child's motor coordination at the exit point was a weakness, the system intervened through interactive games in the real-time feedback module. For example, the "lip-syncing express" game requires children to follow the rapidly changing visual cues on the screen (such as quickly changing from a "u" shape to an "i" shape) and simultaneously utter the corresponding sounds. The system uses a multimodal sensor to capture the speed and accuracy of each attempt at lip-syncing in real time and immediately provides positive feedback on the screen in the form of animated rewards (such as train acceleration) and score increases. This method can effectively maintain the child's interest and participation, and encourage them to keep trying to improve their motor speed.

[0039] After three months of regular gamified training, five times a week for 15 minutes each time, a reassessment of the girl showed that her oral motor coordination had significantly improved. Her average mouth shape switching time was significantly reduced from 420 milliseconds before training to 310 milliseconds, and the gap with the average of children of the same age narrowed from 2.3 standard deviations to 0.8 standard deviations.

[0040] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments that can be applied to other fields. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A child intelligence development detection bracelet based on multi-modal mouth shape analysis, characterized in that, The application relates to a wearable bracelet for real-time assessment of children's oral motor development, comprising: a wearable bracelet body made of flexible material; a multi-modal sensor module integrated on the wearable bracelet body for real-time collection of children's oral motor related multi-modal data, wherein the multi-modal sensor module comprises: a visual sensor for capturing dynamic image sequences of children's mouth, the visual sensor comprising a miniature camera; an audio sensor for synchronous collection of children's speech data, the audio sensor comprising a multi-microphone array with adaptive beamforming and noise reduction algorithms to isolate environmental noise and focus on children's speech; a data processing unit integrated in the wearable bracelet body for: A1. pre-processing dynamic image sequences captured by the visual sensor, including image enhancement, frame alignment and motion compensation, to extract oral motor regions; A2. pre-processing speech data collected by the audio sensor, including audio segmentation, feature extraction and noise reduction; A3. performing multi-modal oral motor analysis by fusing visual and audio data to extract oral motor feature parameters, including lip movement trajectory, mouth opening amplitude, speech spectrum features and timing synchronization, wherein the multi-modal oral motor analysis uses a deep learning model based on a CNN-BiLSTM hybrid neural network trained jointly based on an attention mechanism to identify the correspondence between oral motor and speech and detect abnormal patterns; an intelligent development assessment module integrated in the data processing unit for: B1. comparing the extracted oral motor feature parameters with a pre-stored children's intelligent development standard database, wherein the database includes normal oral motor development milestones and common development disorder indicators based on age segmentation; B2. generating development scores and risk indexes based on the comparison results, the risk indexes being used to identify potential development delays or disorders; a user interface module comprising a touch display screen and LED indicator lights integrated on the wearable bracelet body for real-time display of development assessment results, alerts and suggestion information; a communication module integrated in the wearable bracelet body supporting wireless communication protocols including Bluetooth and Wi-Fi for transmitting collected data and assessment results to external devices or cloud servers; a power management module for powering the entire bracelet.

2. The child intelligence development detection bracelet based on multi-modal mouth shape analysis according to claim 1, characterized in that: The visual sensor further comprises a near-infrared light source with a wavelength range of 700-1000 nm for enhancing the contrast of the oral motor region under non-invasive conditions. 3.The child intelligence development detection bracelet based on multi-modal mouth shape analysis of claim 1, wherein: The multi-microphone array in the audio sensor comprises at least three microphones arranged in a spatial distribution and provided with a real-time sound source positioning algorithm for accurate tracking of children's mouth position and improving speech signal-to-noise ratio.

4. The child intelligence development detection bracelet based on multi-modal mouth shape analysis according to claim 1, characterized in that: The deep learning model used in the multi-modal oral motor analysis is trained through a multi-task learning framework, simultaneously optimizing oral motor classification and speech recognition tasks, wherein the model input includes optical flow features of image sequences and mel-frequency cepstral coefficients of audio, and the output includes oral motor movement patterns and speech content matching degree.

5. The child intelligence development detection bracelet based on multi-modal mouth shape analysis according to claim 1, characterized in that: The intelligent development assessment module further comprises a personalized calibration unit for adjusting the threshold of the pre-stored children's intelligent development standard database according to individual characteristics of children including age, gender and baseline development level. 6.The child intelligence development detection bracelet based on multi-modal mouth shape analysis of claim 1, wherein: The wearable bracelet body further comprises a real-time feedback module integrated in the data processing unit, for generating instant interactive feedback through the user interface module when an abnormal mouth shape or developmental risk is detected, including displaying standard mouth shape animation through the display screen and playing voice prompts through the built-in speaker, for guiding the child to perform corrective training.

7. The child intelligence development detection bracelet based on multi-modal mouth shape analysis according to claim 1, characterized in that: The touch display screen of the user interface module has a graphical user interface for displaying mouth shape analysis real-time visualization, developmental trend chart and interactive games for encouraging the child to participate in the detection, and the LED indicator light indicates the developmental risk level through color change. 8.The child intelligence development detection bracelet based on multi-modal mouth shape analysis of claim 1, wherein: The communication module also supports secure synchronization with a parent mobile device or a medical professional's remote server. 9.The child intelligence development detection bracelet based on multi-modal mouth shape analysis of claim 1, wherein: The bracelet further comprises a data storage module integrated in the wearable bracelet body, for locally caching the collected multi-modal data and evaluation results. 10.The child intelligence development detection bracelet based on multi-modal mouth shape analysis of claim 1, wherein: The miniature camera comprises a wide-angle lens and an infrared light supplementing unit, to enhance the image quality in low-light environments and ensure stable capture of mouth shape changes in the child's daily activities.

Images