Children oral and facial muscle training compliance monitoring system based on multi-modal sensing

Abstract

The invention relates to the technical field of biological rehabilitation engineering and intelligent medical equipment, and discloses a children oral and facial muscle training compliance monitoring system based on multi-modal sensing, which is used for solving the problems of subjectivity, poor compliance and difficulty in remote monitoring of traditional training. The system comprises a wearable multi-mode sensing unit for acquiring surface myoelectricity, occlusal force, inertial measurement and image data; the AI recognition and compliance evaluation module processes data through a deep learning model to realize action recognition, multi-dimensional compliance scoring and personalized self-learning so as to objectively quantify the training quality; the feedback interaction module provides real-time guidance and instant feedback for the user to improve the degree of participation; and the remote management module is used for doctors to monitor data, adjust schemes and obtain analysis reports. Through objective and quantitative data analysis and intelligent interaction, the standardization level and interestingness of child oral and facial muscle training are effectively improved, efficient remote monitoring and personalized intervention are achieved, and the rehabilitation effect is remarkably improved.

Description

Technical Field

[0001] This invention relates to the fields of biorehabilitation engineering and intelligent medical equipment technology, specifically a children's oral and facial muscle training compliance monitoring system based on multimodal sensing. Background Technology

[0002] Orofacial myofunctional function is fundamental to maintaining normal vital functions such as breathing, swallowing, chewing, and speech. For orofacial myofunctional abnormalities that occur during childhood development, such as abnormal swallowing and mouth breathing, orofacial myofunctional training is a commonly used and effective non-invasive corrective method in clinical practice.

[0003] Traditional orofacial muscle function training primarily relies on one-on-one instruction between therapists and patients (mostly children) in person. This model not only consumes significant medical resources but also imposes inconvenience and financial burden on families requiring long-term training. The shortcomings of existing technologies have become particularly apparent as training has expanded from clinics to the home environment.

[0004] Currently, the evaluation of training effectiveness relies heavily on therapists' visual observation and subjective judgment, lacking unified and quantifiable objective standards. When children train independently at home, the lack of professional supervision makes it difficult to guarantee the accuracy of their movements, and incorrect movement patterns may become ingrained, even leading to compensatory behaviors. Furthermore, the monotonous and repetitive nature of the training process often results in low child participation and compliance. Simultaneously, therapists cannot obtain effective training data from patients in non-clinical settings, making it difficult to conduct continuous and accurate follow-up assessments and adjust training plans in a timely manner, which greatly limits the overall effectiveness of rehabilitation therapy. Summary of the Invention

[0005] The purpose of this invention is to provide a multimodal sensing-based system for monitoring compliance in children's oral and facial muscle training, in order to solve the technical problems existing in the current myofunctional training process, such as subjective motion assessment, difficulty in quantifying training compliance, and untimely synchronization of doctor-patient information.

[0006] To achieve the above objectives, the present invention provides the following technical solution: A multimodal sensing-based system for monitoring children's oral and facial muscle training compliance includes: A wearable multimodal sensing unit is used to collect motion data during children's oral and facial muscle function training. The multimodal sensing unit is based on a neck brace structure. An AI recognition and compliance assessment module is used to receive the action data, recognize the training actions, and calculate and generate a training compliance score; A feedback interaction module, connected to the AI ​​recognition and compliance assessment module, is used to present training guidance and real-time feedback to the child. It also includes a remote management module, which is connected to the AI ​​recognition and compliance assessment module, for remote monitoring and intervention by doctors.

[0007] In one specific embodiment, the multimodal sensing unit includes: A tongue position and lip movement image acquisition module, which is installed on the neck brace, is used to acquire image data of tongue position and lip movement; Multiple surface electromyography (EMG) sensors are connected via wiring to a collector mounted on the neck brace for collecting facial muscle activity data. Multiple bite force micro-pressure array sensors are connected via wires to a collector mounted on the neck brace to collect force distribution data of the dental arch; It also includes an IMU (Inertial Measurement Unit) for acquiring head and jaw posture and motion trajectory data.

[0008] Preferably, the neck brace has connecting straps at both the top and bottom openings, and each set of connecting straps is connected by a snap fastener to adjust the tightness of the fit.

[0009] Furthermore, the tongue position and lip movement image acquisition module includes a camera, which is mounted on the neck brace via a carbon steel spring. The carbon steel spring is used to position the camera in front of the child's mouth. Infrared auxiliary lighting LEDs are provided on both sides of the camera to provide auxiliary lighting for image acquisition.

[0010] In a preferred embodiment, the AI ​​recognition and compliance assessment module is implemented through a deep learning system configured to perform the following processing flow: First, for the image data, a convolutional neural network (CNN) is used to extract spatial features representing the position of the tongue and the shape of the lips; for the surface electromyography data, occlusal force data and IMU data, a one-dimensional convolutional neural network (1D-CNN) is used to extract temporal signal features.

[0011] Second, multimodal feature fusion is performed on the extracted spatial features and the temporal signal features. In one specific implementation, this fusion is accomplished through a cross-modal attention mechanism, which dynamically assigns importance weights to each modal feature at each time point to generate a fused feature vector containing cross-modal contextual information. The implementation of this mechanism includes: For any modality's feature vector, a query vector, key vector, and value vector are generated by projecting them through a learnable transformation matrix. Calculate the similarity score between the query vector of one modality and the key vectors of all other modalities, and obtain the attention weights through normalization. Using the calculated attention weights, the value vectors of all modalities are weighted and summed to generate a new feature vector after adjustment. Finally, all the newly generated feature vectors are concatenated to form a fused feature sequence.

[0012] Third, the fused feature sequence is input into a bidirectional gated recurrent unit (BiGRU) network for temporal modeling to learn long-distance temporal dependencies in the action process and output a comprehensive feature sequence of user actions.

[0013] Fourth, the matching degree between the comprehensive feature sequence and a preset standard children's oral and facial muscle training action template library is calculated using the Dynamic Time Warping (DTW) algorithm. The action corresponding to the template with the highest similarity is identified as the training action performed by the user. This matching degree serves as one of the core bases for generating the training compliance score.

[0014] In one embodiment, the system further includes a motion artifact filtering module configured to: Receive head motion data collected by the IMU inertial measurement unit, including triaxial angular velocity data and triaxial linear acceleration data; Two intensity indicators for head motion are calculated: the square root of the sum of squares of the three-axis angular velocity components (i.e., the magnitude of the angular velocity) and the square root of the sum of squares of the three-axis linear acceleration components (i.e., the magnitude of the linear acceleration). When any intensity index exceeds a preset threshold, a time window near that time point is identified as a non-training action period. During the non-training period, the corresponding surface electromyography data and occlusal force data are automatically removed or marked.

[0015] The AI ​​recognition and compliance assessment module also includes a compliance scoring model, which calculates a comprehensive compliance score based on a set of preset scoring indicators. The calculation process involves multiplying the quantified values ​​of completion rate, duration accuracy, movement standardization, electromyographic coordination, and rhythm and stability by their respective preset weighting coefficients, and then summing them.

[0016] To achieve personalized recognition optimization, in a preferred embodiment, the AI ​​recognition and compliance assessment module is configured with a personalized adjustment module, which includes: A dynamic growth adaptation submodule is used to receive input physiological parameters of children (including age and dental development stage) and dynamically adjust the benchmark threshold used to evaluate the standard of action within the AI ​​recognition and compliance assessment module according to preset mapping rules. And a self-learning submodule, which is used to retrain the deep learning system in the AI ​​recognition and compliance assessment module based on the specific child's historical training data, in order to fine-tune the model parameters.

[0017] Preferably, the feedback interaction module includes a child-friendly interface, which is equipped with a cartoon-style training guidance system and a task planning and reward mechanism to present training guidance and provide immediate feedback to the child through gamified interaction.

[0018] In one embodiment, the remote management module includes a doctor-side remote monitoring and intervention interface, which allows doctors to remotely customize or modify training plans and push the training plans to the feedback interaction module; at the same time, the interface is also used to display training compliance scores and training trend analysis generated by the AI ​​recognition and compliance assessment module.

[0019] Furthermore, the remote management module is also equipped with an automatic report generation engine, which is used to automatically generate weekly or monthly training reports that include training compliance trend curves and action standardization on a regular basis.

[0020] This invention provides a multimodal sensing-based system for monitoring children's oral and facial muscle training compliance. It offers the following advantages: 1. This invention sets up a wearable multimodal sensing unit to simultaneously collect multi-dimensional data such as surface electromyography, tongue position images, head posture, and biting force. It also uses the compliance scoring model in the AI ​​recognition and compliance assessment module to perform quantitative calculations, thereby achieving an objective and standardized evaluation of training actions. This method overcomes the subjectivity and inconsistency caused by traditional reliance on manual observation and provides quantifiable and traceable data support for training effects.

[0021] 2. This invention, by setting up a feedback interaction module, especially the child-friendly interface, is equipped with a cartoon-style training guidance system and a task planning and reward mechanism. This gamified interaction transforms tedious training tasks into fun challenges, and uses instant feedback and virtual rewards to stimulate children's training motivation, thereby significantly improving children's initiative and compliance in oral and facial muscle function training.

[0022] 3. This invention constructs a closed-loop management system connecting the patient's end and the doctor's end by setting up a remote management module. Doctors can remotely view detailed training data reports and compliance trend analysis through this module, and can adjust and push personalized training plans in real time, realizing efficient remote monitoring and dynamic intervention. This solves the problems of lagging doctor-patient communication and inability to adjust the plan in a timely manner in the traditional training mode, and improves the accuracy of intervention and treatment efficiency. Attached Figure Description

[0023] Figure 1 This is a perspective view of the wearable multimodal sensing unit of the present invention; Figure 2 This is a schematic diagram of the snap fastener structure of the present invention; Figure 3 for Figure 2 Enlarged view of point A in the middle; Figure 4 for Figure 2 Enlarged view at point B in the middle; Figure 5 This is a system composition block diagram according to an embodiment of the present invention; Figure 6 This is a schematic diagram of the IMU inertial measurement unit of the present invention.

[0024] The components include: 1. Neck brace; 2. Carbon steel spring; 3. Camera; 4. Infrared auxiliary lighting LED; 5. Collector 1; 6. Circuit 1; 7. Biting force micro-pressure array sensor; 8. Collector 2; 9. Circuit 2; 10. Surface electromyography sensor; 11. Connecting strap; 12. Snap fastener. Detailed Implementation

[0025] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] Please refer to the appendix. Figure 5 , Figure 5 This is a system composition block diagram according to an embodiment of the present invention. The present invention provides a children's oral and facial muscle training compliance monitoring system based on multimodal sensing, which may include: a wearable multimodal sensing unit, an AI recognition and compliance assessment module, a feedback interaction module, and a remote management module.

[0027] In the system, wearable multimodal sensing units are used to collect motion data from children during oral and facial muscle function training. An AI recognition and compliance assessment module connects to the wearable multimodal sensing units to receive and process the motion data. A feedback interaction module connects to the AI ​​recognition and compliance assessment module to present feedback information to the user. A remote management module also connects to the AI ​​recognition and compliance assessment module to provide data monitoring and management functions to doctors.

[0028] The workflow of this invention is as follows: First, the child user wears a wearable multimodal sensing unit. When performing muscle function training tasks, the unit collects multi-dimensional motion data in real time, including surface electromyography, images, inertial parameters and biting force.

[0029] Next, the wearable multimodal sensing unit transmits the collected motion data to the AI ​​recognition and compliance assessment module via wireless communication.

[0030] After receiving action data, the AI ​​recognition and compliance assessment module identifies, matches, and quantifies the data, and calculates a training compliance score. This score is based on a set of preset scoring indicators and is calculated using the following formula to obtain a comprehensive compliance score ( ): ; in, The final output is a comprehensive compliance score; This is the quantified value of the completion rate indicator; This is the quantified value of the duration accuracy index; This is the quantified value of the action standardization index; This is a quantitative value for the electromyographic synergy index; This refers to the quantitative value of the rhythm and stability index; , , , , These are the preset weighting coefficients for each corresponding indicator.

[0031] After the calculation is completed, the AI ​​recognition and compliance assessment module will output the assessment results, including the training compliance score, through two data paths: the first path will send simplified information for immediate feedback to the feedback interaction module for child users to view in real time; the second path will send detailed information, including the score, raw data, and analysis results, to the remote management module.

[0032] Finally, doctors can view detailed training data reports and compliance trend analyses through the remote management module, and remotely adjust the training plan based on the analysis results. The adjusted plan is then distributed through the remote management module and synchronized to the child's interface in the feedback interaction module via the AI ​​recognition and compliance assessment module, thus forming a data-driven closed-loop intervention process.

[0033] Please refer to the appendix. Figure 1 and attached Figure 2The wearable multimodal sensing unit uses a neck brace 1 as its basic support and fixation structure. The neck brace 1 is designed according to the ergonomic characteristics of a child's neck, and its main structure is an open or semi-open ring for easy wearing and removal.

[0034] The neck brace 1 is made of medical-grade flexible polymer material or silicone material to ensure biocompatibility when in contact with the skin. The surface of the neck brace 1 has pre-drilled interfaces and positions for installing collector 1 5, collector 2 8, and carbon steel spring 2.

[0035] Please refer to the appendix. Figure 2 Connecting straps 11 are attached to the openings at both the top and bottom of the neck brace 1. The ends of each set of connecting straps 11 are connected by snap fasteners 12. By adjusting the fit of the snap fasteners 12, the effective length of the connecting straps 11 can be changed, thereby adjusting the tightness of the neck brace 1 to securely fix it around the child's neck.

[0036] See attached document Figure 3 The multimodal sensing unit includes a tongue position and lip movement image acquisition module. This module includes a camera 3, a carbon steel spring 2, and an infrared auxiliary illumination LED 4.

[0037] Camera 3 is mounted on the front of neck brace 1 via carbon steel spring 2. Carbon steel spring 2 positions and holds camera 3 at a preset distance and angle in front of the child user's mouth, ensuring the field of view fully covers the range of motion of the lips and part of the tongue. The elastic structure of carbon steel spring 2 allows for fine-tuning during wear and provides cushioning against vibrations caused by user movements.

[0038] Infrared auxiliary illumination LEDs 4 are installed on both sides of camera 3. The infrared auxiliary illumination LEDs 4 are used to provide a stable supplementary light source when the ambient light is insufficient or changes, thereby ensuring that the acquired image data has consistent brightness and contrast and reducing the impact of ambient light on image quality.

[0039] Camera 3 is a miniature digital camera that transmits the collected continuous image frame data through internal circuitry to the processing circuit or data transceiver unit integrated on the neck brace 1 for subsequent processing or transmission.

[0040] Please refer to the appendix. Figure 3 Appendix Figure 4 and attached Figure 6 The multimodal sensing unit also includes multiple surface electromyography sensors 10, multiple bite force micro-pressure array sensors 7, and an IMU inertial measurement unit.

[0041] The surface electromyography (EMG) sensor 10 includes multiple electrode patches for attaching to the skin surface of specific muscle groups on the user's face, such as the orbicularis oris or mentalis muscle, during training to collect corresponding muscle electrical activity signals. The EMG sensor 10 is electrically connected to a collector 8 via a line 2 9, which is fixedly mounted on the structure of the neck brace 1. The collector 8 integrates signal conditioning circuitry for amplifying and filtering the received weak EMG signals.

[0042] The bite force micro-pressure array sensor 7 is a flexible thin-film sensor with multiple micro-pressure sensing points distributed on it. It is placed on the occlusal surface of the user's maxilla or mandible during training to collect data on the force distribution of the dental arch during biting. The bite force micro-pressure array sensor 7 is electrically connected to a collector 5 via a line 6, which is also fixedly mounted on the structure of the neck brace 1. The collector 5 is used to receive and process electrical signals from the sensor array.

[0043] The IMU (Inertial Measurement Unit) is encapsulated and embedded within the main structure of the neck brace 1. This unit includes a three-axis accelerometer and a three-axis gyroscope, used to measure in real-time acceleration and angular velocity data generated by the user's head movements during training. This data can be used to calculate the head and jaw posture and trajectory.

[0044] Collector 5 and Collector 8 serve as data acquisition nodes, collecting the sensor data after their respective conditioning and preliminary processing, along with the data from the IMU inertial measurement unit and the image data acquired by the camera 3, together at the center set on the neck brace 1.

[0045] After receiving motion data from the wearable multimodal sensing unit, the AI ​​recognition and compliance assessment module is first processed by its internal feature extraction subsystem. This subsystem converts the raw multimodal data stream into structured feature vectors that can be used for subsequent recognition and assessment.

[0046] The workflow of this subsystem begins with data preprocessing. Since surface electromyography, occlusal force, IMU, and image data are generated by different sensors, the system first performs time alignment on these heterogeneous data streams based on the timestamp information inherent in each data frame or data point to form a unified time series.

[0047] After time alignment, the system performs noise reduction processing on each data stream. For the timing signals acquired by the surface electromyography sensor 10 and the IMU inertial measurement unit, a digital filter, such as a fourth-order Butterworth low-pass filter, is used to filter out power frequency interference and high-frequency motion artifacts. For the image data acquired by the camera 3, image filtering algorithms, such as Gaussian filtering, are applied to reduce image noise.

[0048] The final step in preprocessing is data normalization. The amplitude of all data streams is scaled to a uniform preset range, such as [0,1], to eliminate differences in units and numerical ranges between different types of sensor data and ensure their weighted balance in subsequent model processing.

[0049] After preprocessing, the feature extraction subsystem uses a deep learning model to extract high-dimensional features. For tongue position and lip movement image data, the system inputs a continuous sequence of image frames into a convolutional neural network (CNN). This CNN extracts spatial feature vectors that characterize lip shape, degree of opening, and tongue position frame by frame through its convolutional and pooling layers.

[0050] For time-series data such as surface electromyography (EMG) signals, bite force distribution data, and IMU signals, the system inputs them into multiple one-dimensional convolutional neural networks (1D-CNNs). The convolutional kernels of these 1D-CNNs slide along the time axis to capture local temporal patterns, trends, and frequency domain features in the signals.

[0051] Finally, at each synchronized time point, the feature extraction subsystem concatenates the spatial feature vector extracted from the image data with the time-domain and frequency-domain feature vectors extracted from various types of time-series data, combining them into a single, higher-dimensional fused feature vector. Thus, the original multimodal data stream is transformed into a multidimensional temporal feature sequence, which is then transmitted to the action recognition and matching subsystem for further processing.

[0052] This invention performs a data preprocessing method before inputting the collected data into the AI ​​recognition and compliance assessment module to filter out artifact data generated by non-training actions. The method includes the following steps: Step 1: Acquire multi-source synchronized data. The system synchronously acquires the following data streams from the wearable multimodal sensing unit: Head motion data from the IMU (Inertial Measurement Unit), specifically triaxial angular velocity data and triaxial linear acceleration data.

[0053] Surface electromyography (sEMG) data from multiple surface electromyography sensors 10.

[0054] Biting force data from multiple bite force micro-pressure array sensors.

[0055] Step 2: Identification of non-training action periods.

[0056] This step is performed by quantification and comparison with a threshold.

[0057] Data and parameter definitions: definition t This refers to the sampling time point.

[0058] Head motion data from the IMU (Inertial Measurement Unit) is defined as follows: Triaxial angular velocity vector t ); Triaxial linear acceleration vector a ( t ) ; The system has two preset numerical thresholds: angular velocity threshold ; Linear acceleration threshold ; These two thresholds together define the intensity of head movements that are defined as non-training actions.

[0059] Motion intensity quantification calculation: The data processing module calculates at each time point. Above, the Euclidean norm of the motion vector (i.e., its scalar magnitude) serves as a quantitative indicator of motion intensity. .

[0060] angular velocity magnitude The calculation formula is: ;(Formula 1); Magnitude of linear acceleration The calculation formula is: ;(Formula 2) in, , , Three-axis angular velocity components , , Triaxial linear acceleration components Threshold comparison and time period marking: The data processing module determines the time point based on the following logical conditions. Did any non-training movements occur? ;(Formula 3) in," " represents a logical OR. If the condition in Formula 3 is true, then the event point will be used as the reference." A time period centered on [the training action] is identified and labeled as a non-training action period. The time period can be defined as […]. t - ,in The preset time window length (e.g., (ms, forming a 200ms time window).

[0061] Step 3: Processing artifact data.

[0062] The system's data filtering module receives the non-training action time period markers generated in step two.

[0063] For each marked non-training action period, the module locates the surface electromyography data points and occlusal force data points that are synchronously collected during that period.

[0064] Perform one of the following operations on the located data points: a) Data removal: Data points within this time period are directly removed from the original surface electromyography data stream and occlusal force data stream to form a discontinuous data stream.

[0065] b) Data tagging: Preserve the position of the data point in the time series, but modify its value to a predefined invalid tag value, such as 0 or NaN (Not a Number).

[0066] Step 4: Output the processed data. The data filtering module outputs the surface electromyography (EMG) data stream and occlusal force data stream processed in Step 3, along with the unprocessed image data stream, to the AI ​​recognition and compliance assessment module. As a result, the data input to the AI ​​module for subsequent feature extraction and analysis no longer includes EMG and occlusal force data that occurred simultaneously with head movements exceeding the movement threshold.

[0067] The AI ​​recognition and compliance assessment module is the core technology of this invention. It receives raw data collected by wearable multimodal sensing units and outputs accurate recognition results and quantified compliance scores for training actions. In a preferred embodiment of this invention, this module is implemented through an end-to-end deep learning system integrating feature extraction, multimodal fusion, temporal modeling, and template matching. Its specific processing flow is as follows: Step 1: Parallel multimodal feature extraction. This step uses an optimized network structure for parallel processing of data from different sources.

[0068] For image data: For the video stream acquired by the tongue position and lip movement image acquisition module, the system first decomposes it into consecutive image frames. Then, a pre-trained convolutional neural network (CNN) is used to process each frame to extract spatial feature vectors that can represent key information such as tongue morphology, tongue tip position, lip opening and closing degree and shape.

[0069] For time-series signal data: For sEMG signals, pressure distribution signals, and posture motion signals acquired by the surface electromyography sensor 10, the bite force micro-pressure array sensor 7, and the IMU inertial measurement unit, the system employs a one-dimensional convolutional neural network (1D-CNN). The 1D-CNN can effectively capture local patterns and abrupt changes in these time-series signals, such as the burst points of electromyography signals, peak patterns of bite force, or specific motion trajectory segments of the head, and generate corresponding time-series feature sequences.

[0070] Step 2: Feature fusion based on cross-modal attention mechanism To address the vulnerability of single-sensor data to interference in specific contexts (e.g., minor facial tremors introducing noise into electromyography signals), and to enable the model to autonomously learn the intrinsic correlations between data from different modalities, this invention employs a cross-modal attention mechanism to replace simple feature concatenation. This mechanism dynamically calculates and assigns importance weights to each modal feature at each time point. Its workflow is as follows: Feature vector input: at any time point The mechanism receives parallel feature vectors from the first step, including: Image spatial feature vector ; Surface electromyography temporal feature vector ; Temporal eigenvector of bite force ; IMU motion time-series feature vector ; Query, Key, Value Projection: For each input modal feature vector (in ∈{img, semg, bite, imu}), the system uses three independent and learnable linear transformation matrices ( , , This is then projected to generate query, key, and value vectors: ; ; ; Attention weight calculation: The system calculates the similarity score between a query vector of one modality and the key vectors of all other modalities. For example, calculating surface electromyography features ( ) for all other features ( When a person's attention is focused, their attention score is calculated as follows: ; in, This is the dimension of the key vector, used for scaling to prevent the gradient from becoming too small. Then, these scores are normalized using the Softmax function to obtain the final attention weights. : ; Weighted feature generation: Using the calculated attention weights, the value vectors of all modalities are processed. Weighted summation is performed to generate a new surface electromyography feature vector after attention mechanism regulation. : ; This process is repeated for each modality, ultimately generating a new set of feature vectors that contain cross-modal contextual information. .

[0071] Example of specific implementation effect: When surface electromyography characteristics When exhibiting a high-intensity, rhythmic burst pattern (corresponding to a single effective muscle contraction), the attentional weights calculated by this mechanism will... Characteristics of bite force Increased attention (weight). Conversely, if If it presents as a high-frequency, low-amplitude noise pattern (possibly caused by skin tremors), then its effect on... The weight given to these will decrease accordingly.

[0072] Final fused feature sequence output: The newly generated, attention-weighted feature vectors from each modality { The features are concatenated to form a single, higher-dimensional fused feature vector. F ( t (From all points in time) F ( t The sequence composed of these features is the final fusion feature sequence.

[0073] Step 3: Long-term temporal modeling. The fused feature sequence output from step 2 is input into a bidirectional gated recurrent unit (BiGRU) network. BiGRU can learn the long-distance temporal dependencies in the entire action process from both forward and backward dimensions, and finally output a comprehensive feature sequence that can fully and dynamically represent the complete action performed by the user.

[0074] Step 4: Action Recognition and Matching Based on Dynamic Time Warping. The system has a pre-installed standard action template library. This library stores standard examples of at least five core children's oral and facial muscle training actions (e.g., cheek puffing, tongue lifting, teeth-showing smile, alternating "ao" sounds, etc.). These standard examples are performed and recorded by professional therapists and processed using the aforementioned CNN, 1D-CNN, and BiGRU networks to generate comprehensive feature sequences. When the system obtains a comprehensive feature sequence of a user's action, it uses the Dynamic Time Warping (DTW) algorithm to calculate the similarity between it and each standard action template in the library. The DTW algorithm effectively measures the similarity between two sequences of unequal time durations. After calculation, the action corresponding to the template with the highest similarity (i.e., the smallest DTW distance) is identified as the training action currently being performed by the user. Simultaneously, this calculated minimum DTW distance value is directly used as the core quantitative indicator for evaluating "action standardization."

[0075] Step 5: Generate a comprehensive compliance score. Based on the movement standard score obtained from the DTW algorithm, and combined with other data such as training completion rate, difference between movement duration and standard duration, coordination pattern of electromyographic signals, and movement rhythm stability recorded by the system, the compliance score model finally calculates a comprehensive training compliance score and transmits it to the feedback interaction module and remote management module.

[0076] The compliance scoring model receives output data from the action recognition and matching subsystem, which includes the identified training action categories and the similarity scores obtained after matching the action with a standard template. The model's function is to quantify the performance quality of the identified actions based on a set of preset scoring metrics.

[0077] The model calculates the following five indicators separately: completion rate ( ), duration accuracy ( ), standard of movement ( ), electromyographic synergy ( ) and rhythm and stability ( ).

[0078] Completion rate ( The calculation method is as follows: divide the number of valid action repetitions successfully identified by the action recognition and matching subsystem within a training unit by the total target number of repetitions set by the doctor within that training unit to obtain a ratio value.

[0079] Duration accuracy ( The calculation method is as follows: First, measure the actual duration of a single identified action from start to finish. Then, calculate the absolute difference between the actual duration and the standard duration defined in the standard training action template. Finally, map the difference to a score through a preset function. The smaller the difference, the higher the score.

[0080] Movement standard ( The similarity score is calculated as follows: directly or through function transformation, using the Dynamic Time Warping (DTW) algorithm in the action recognition and matching subsystem to calculate the similarity score between the user action sequence and the standard template sequence. The higher the similarity score, the more standard the action, and the higher the score of this indicator.

[0081] electromyographic synergy ( The calculation method is as follows: During the time period in which the identified action occurs, synchronous surface electromyography data is extracted, the integrated electromyography value or root mean square value of the electromyography signal of the target muscle group (such as the orbicularis oris) is calculated, and compared with the corresponding electromyography signal value of the non-target compensatory muscle group (such as the mentalis). The accuracy of muscle exertion is quantitatively evaluated based on the ratio between the two.

[0082] Rhythm and stability ( The calculation method for the index is as follows: For multiple consecutive effective repetitions of an action, measure the time interval between the start points of each repetition, and calculate the standard deviation or variance of this series of time interval values. The smaller the standard deviation or variance, the more stable the action rhythm, and the higher the score of this index.

[0083] After calculating the scores of the above indicators, the compliance scoring model uses the aforementioned weighted summation formula to assign each indicator score to its corresponding preset weight coefficient. , , , , The products are multiplied and then summed to output a single, quantitative comprehensive compliance score.

[0084] To address the technical problem of deviations in standardized assessments caused by individual physiological differences and developmental stages among children, this invention incorporates a personalized adjustment module within its AI recognition and compliance assessment module. This module comprises two components: a dynamic growth adaptation submodule and a self-learning submodule, which work together to achieve individualized optimization of the assessment model.

[0085] The dynamic growth adaptation submodule is used to make prior adjustments to the assessment benchmarks based on the child's objective physiological developmental stage. Its workflow includes the following steps: Physiological parameter acquisition: The system receives and stores the input physiological parameters of the child through the user profile configuration interface. These physiological parameters include at least: Age: A numerical value expressed in years and months.

[0086] Dental development stage: Select from a preset category, which includes, but is not limited to: primary dentition stage, mixed dentition stage, and early permanent dentition stage.

[0087] Benchmark Threshold Adjustment: This submodule internally stores a parameter mapping rule set. This rule set defines a deterministic mapping relationship between physiological parameters and benchmark thresholds for the evaluation model. Upon receiving the physiological parameters, the submodule automatically selects or calculates a corresponding set of benchmark thresholds based on the rule set and updates them to the AI ​​recognition and compliance assessment module. The adjusted benchmark thresholds may specifically include: Occlusal force threshold: If the input dentition development stage is "mixed dentition period", the pressure value range used to determine whether the occlusal action meets the standard will be set to be lower than the value range corresponding to "permanent dentition period".

[0088] Electromyography (sEMG) amplitude range: Adjust the surface electromyography (sEMG) amplitude benchmark used to assess the activity intensity of related muscles such as the orbicularis oris or buccinator muscle based on the input age parameter.

[0089] The self-learning submodule is used to perform posterior fine-tuning of the evaluation model by learning the behavioral patterns of specific individuals, based on dynamic growth and adaptation. Its workflow includes the following steps: Personalized Dataset Construction: The system establishes an independent database for each registered child user. This database is used to continuously store the data generated during each training session, including: Multimodal sensing data collected by wearable multimodal sensing units and filtered for motion artifacts.

[0090] The labels are manually corrected by doctors through a remote management module based on the recognition results.

[0091] Retraining process triggering: The retraining process is initiated when any of the following conditions are met: Data volume condition: When the number of training data samples accumulated in the individualized database reaches a preset value (e.g., 1000 valid action samples).

[0092] Command condition: When the doctor issues a retraining command through the remote management module.

[0093] Model parameter fine-tuning: When the retraining process is triggered, this submodule calls upon the individualized dataset for the corresponding child. This dataset is used as incremental training samples to update the weight parameters of the pre-trained deep learning system (including convolutional neural networks, one-dimensional convolutional neural networks, and bidirectional gated recurrent unit networks) in the AI ​​recognition and compliance assessment module. This update process performs additional training iterations based on the existing model weights through backpropagation and gradient descent algorithms.

[0094] Personalized model deployment: After retraining, the newly model with fine-tuned parameters will be saved and associated with the user's identity identifier. In subsequent training sessions, when the system recognizes the user, it will automatically call this personalized optimized model to perform action recognition and compliance assessment tasks, thereby improving the accuracy of action recognition for that user.

[0095] The feedback interaction module connects with the AI ​​recognition and compliance assessment module, and its function is to present training guidance information and real-time feedback results to children's users.

[0096] The feedback interaction module includes a child-friendly user interface running on a terminal device (such as a tablet or smartphone). This interface features a cartoon-style training guidance system. This system includes a preset cartoon character that demonstrates standard training movements set by the doctor through videos or animations. Simultaneously, the system also outputs voice commands through this cartoon character, guiding the user through the training steps.

[0097] The user interface for children will also visualize the training plans received from the remote management module in the form of a task list or level map. Parameters such as the name of each training task, the target number of repetitions, and the required duration are clearly displayed on the interface.

[0098] During training, the feedback interaction module receives compliance scores in real time, calculated by the AI ​​recognition and compliance assessment module. These scores are then transformed into visual elements on the interface, such as the increase of a progress bar or a change in a score, providing users with immediate quantitative feedback on the quality of their action completion.

[0099] The interface also features a task scheduling and reward mechanism. This reward mechanism is triggered when the system determines that the user has completed a training action or a set of training tasks. Upon triggering, the system automatically accumulates a preset score on the interface and, when certain conditions are met, awards the user a pre-designed virtual badge. This badge is stored and displayed in the user's personal interface.

[0100] The remote management module connects with the AI ​​recognition and compliance assessment module, and its function is to provide doctors with an operating platform for remote monitoring, data analysis and intervention.

[0101] The remote management module includes a remote monitoring and intervention interface that runs on a doctor's terminal device (such as a personal computer or tablet). This interface receives and displays detailed training data sent by the AI ​​recognition and compliance assessment module. Doctors can access and view a specific user's training records through this interface, including synchronized playback of tongue and lip movement videos, surface electromyography (EMG) atlases, occlusal force distribution maps, and historical trend charts of compliance scores.

[0102] The remote monitoring and intervention interface includes a training plan editing and push function. Doctors can use this function to create or modify training plans for designated users. Editable parameters include the type of training movement, the number of repetitions per set, the standard duration of the movement, and the daily training frequency. Once the doctor has finished editing and saved the plan, the function sends the updated training plan data to the AI ​​recognition and compliance assessment module, which then synchronizes the new plan to the feedback interaction module on the child's end.

[0103] The remote management module also includes an automatic report generation engine. This engine is used to periodically integrate the training data of designated users. For example, at the end of each calendar week or month, the engine automatically extracts all key indicators such as compliance scores, completion rates, and movement accuracy within that time period and generates a structured PDF training report. This report visually presents the user's compliance trends and training progress, allowing doctors to conduct periodic assessments.

[0104] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A multimodal sensing-based system for monitoring children's oral and facial muscle training compliance, characterized in that, include: A wearable multimodal sensing unit is used to collect motion data during the training of children's oral and facial muscle functions. The multimodal sensing unit is based on a neck brace (1). The AI ​​recognition and compliance assessment module is used to receive the action data, recognize the training actions, and calculate and generate a training compliance score. The feedback interaction module, connected to the AI ​​recognition and compliance assessment module, is used to present training guidance and real-time feedback to the child. The remote management module is connected to the AI ​​recognition and compliance assessment module and is used for remote monitoring and intervention by doctors.

2. The multimodal sensing-based system for monitoring children's oral and facial muscle training compliance according to claim 1, characterized in that, The multimodal sensing unit includes: A tongue position and lip movement image acquisition module is installed on the neck brace (1). The tongue position and lip movement image acquisition module is used to acquire image data of tongue position and lip movement. Multiple surface electromyography (10) sensors (10) are connected via line 2 (9) to collector 2 (8) mounted on the neck brace (1), and the surface electromyography (10) sensors (10) are used to collect facial muscle activity data; Multiple bite force micro-pressure array sensors (7) are connected to collector (5) installed on the neck brace (1) via line 1 (6). The bite force micro-pressure array sensors (7) are used to collect force distribution data of the dental arch. An IMU (Inertial Measurement Unit) is used to acquire posture and motion trajectory data of the head and jaw.

3. The children's oral and facial muscle training compliance monitoring system based on multimodal sensing according to claim 1, characterized in that, The neck brace (1) has connecting straps (11) at both the upper and lower ends, and each set of connecting straps (11) is connected by a snap fastener (12).

4. The children's oral and facial muscle training compliance monitoring system based on multimodal sensing according to claim 2, characterized in that, The tongue position and lip movement image acquisition module includes a camera (3), which is mounted on the neck brace (1) by a carbon steel spring (2). The carbon steel spring (2) is used to position the camera (3) in front of the child's mouth. Infrared auxiliary lighting LEDs (4) are provided on both sides of the camera (3).

5. The children's oral and facial muscle training compliance monitoring system based on multimodal sensing according to claim 1, characterized in that, The AI ​​recognition and compliance assessment module is specifically implemented through a deep learning system, which is used for: A convolutional neural network is used to extract spatial features from the image data; A one-dimensional convolutional neural network was used to extract temporal signal features from the surface electromyography data, occlusal force data, and IMU data. The spatial features and the temporal signal features are input into a bidirectional gated recurrent unit network for multimodal feature fusion and temporal modeling to generate a comprehensive feature sequence of user actions; The algorithm uses dynamic time warping to calculate the matching degree between the comprehensive feature sequence and a preset template library containing at least five types of standard children's oral and facial muscle training movements, so as to complete the movement recognition and serve as the core basis for generating the training compliance score.

6. The multimodal sensing-based system for monitoring children's oral and facial muscle training compliance according to claim 2, characterized in that, It also includes a motion artifact filtering module, which is used for: Receive head motion data collected by the IMU inertial measurement unit; When the head movement data exceeds a preset threshold, non-training action periods are identified; During the non-training action period, the corresponding surface electromyography data and occlusal force data are automatically removed to prevent them from interfering with the AI ​​recognition and compliance assessment module.

7. The children's oral and facial muscle training compliance monitoring system based on multimodal sensing according to claim 1, characterized in that, The feedback interaction module includes a child-friendly interface, which is equipped with a cartoon-style training guidance system and a task planning and reward mechanism to enhance the child's initiative in training through gamified interaction.

8. The children's oral and facial muscle training compliance monitoring system based on multimodal sensing according to claim 1, characterized in that, The remote management module includes a doctor-side remote monitoring and intervention interface, which allows doctors to remotely customize or modify training plans, push the training plans to the feedback interaction module, and view training compliance scores and training trend analyses generated by the AI ​​recognition and compliance assessment module.

9. The multimodal sensing-based system for monitoring children's oral and facial muscle training compliance according to claim 1, characterized in that, The remote management module is equipped with an automatic report generation engine, which is used to automatically generate weekly or monthly training reports that include training compliance trend curves and action standardization.

10. The children's oral and facial muscle training compliance monitoring system based on multimodal sensing according to claim 1, characterized in that, The AI ​​recognition and compliance assessment module is equipped with a personalized adjustment module, which is used to achieve individualized recognition optimization. The personalized adjustment module includes: A dynamic growth adaptation submodule is used to receive input physiological parameters of the child, including age and dental development stage, and dynamically adjust the benchmark threshold used to evaluate the standard of action within the AI ​​recognition and compliance assessment module based on the physiological parameters. A self-learning submodule is used to retrain the deep learning system in the AI ​​recognition and compliance assessment module based on the specific child's historical training data.

Images