Tongue body motion form reconstruction and neuromuscular lesion identification method and system

Abstract

The invention relates to the technical field of biomedical engineering, and provides a tongue body motion form reconstruction and neuromuscular lesion recognition method, which comprises the following steps: S1, acquiring a multi-modal original signal by using a flexible electrode array attached to a lingual surface, and recording a system timestamp; s2, performing targeted filtering, artifact removal and feature extraction processing on the multi-mode original signal; s3, reconstructing a local dynamic curved surface of the lingual surface by adopting a preset model, and inferring an overall position and posture sequence of the tongue body in combination with an IMU signal; s4, a joint optimization objective function is constructed, and a tongue body three-dimensional dynamic form sequence is obtained through iterative solution; s5, performing motion unit action potential MUAP decomposition and feature extraction on the surface electromyogram signal sEMG; and S6, constructing lesion feature vectors based on the multi-dimensional abnormal features, and analyzing and generating a tongue lesion risk map TDM through a machine learning model. The defects that real motion reconstruction, form inference and neuromuscular lesion positioning of the tongue cannot be achieved through an existing tongue detection technology are overcome.

Description

Technical Field

[0001] This invention relates to the technical field of biomedical engineering, and more particularly to a method and system for morphological reconstruction of tongue movement and identification of neuromuscular lesions. Specifically, it is applied in the fields of computational neuroscience and artificial intelligence, and is a three-dimensional reconstruction of tongue movement and morphology based on the fusion of multimodal signals from the tongue surface, as well as a technique for analyzing neuromuscular lesions. This falls under the direction of oral function testing and digital medical diagnostic technology. Background Technology

[0002] As a core functional organ in the human oral cavity, the tongue's motor coordination and the physiological activity of its muscles directly govern a series of key physiological processes, including swallowing, pronunciation, chewing, and taste formation. The normal functioning of these processes is essential for ensuring adequate nutrient intake, language communication, and quality of life. The hypoglossal nerve and lingual nerve, as core neural pathways controlling the tongue muscles, are particularly vulnerable. Damage to their integrity, along with the integrity of the muscles they control, can directly lead to various motor dysfunctions such as slowed tongue movement, deviation, muscle weakness, and fasciculations. In severe cases, it can even cause complications such as difficulty swallowing and unclear speech, significantly impacting the patient's physical and mental health.

[0003] In current clinical practice, the methods for detecting and assessing neuromuscular lesions of the tongue are relatively limited, mainly relying on the following approaches: First, subjective visual examination, where physicians observe the tongue's movements such as protrusion, curling, and lateral swaying to determine the presence of abnormalities such as tongue deviation, fasciculations, and muscle atrophy. This method is highly dependent on the physician's clinical experience, subjective, and difficult to quantify the severity of the lesion. Second, imaging examinations, including ultrasound and magnetic resonance imaging (MRI). Ultrasound imaging is limited by resolution and does not clearly display the deep tongue muscle structure, while MRI, although providing clearer anatomical images, is time-consuming, expensive, and requires strict adherence to the patient's posture, making it unsuitable for patients with metal implants. Third, traditional electrophysiological monitoring, which involves collecting electromyographic signals of the tongue muscles using electrodes. This method can only record the temporal changes in electromyographic activity and cannot correlate the signals with the spatial movement of the tongue.

[0004] However, the aforementioned existing detection technologies all have significant technical shortcomings and are difficult to meet the needs of accurate clinical diagnosis: (1) Imaging detection methods such as ultrasound and MRI are essentially static or quasi-static detection methods. They cannot capture the movement of the tongue in natural physiological activities (such as continuous pronunciation, eating and swallowing) in real time. Furthermore, their clinical applicability and popularity are greatly limited due to detection costs, equipment conditions and individual patient differences. (2) Traditional electrophysiological detection technology has a relatively simple function. It can only collect and record electromyographic activity, and cannot simultaneously reflect the real three-dimensional movement trajectory, spatial posture and surface morphology dynamic changes of the tongue during the movement process. This makes it difficult for clinicians to fully assess the integrity of tongue muscle function and the impact of lesions on motor function. (3) At present, no technology can simultaneously integrate multi-dimensional information such as tongue electrophysiological signals, local tongue movement information, tongue geometric deformation data and three-dimensional anatomical prior structure of tongue muscles to form a complete functional assessment system, thereby accurately determining the specific location, extent and nature of the lesion.

[0005] Furthermore, there are significant technological gaps in the current field: on the one hand, there is a lack of a multimodal fusion reconstruction model specifically for the tongue, which needs to integrate three-dimensional motion information, surface electromyography signals, geometric deformation data, and prior anatomical knowledge to achieve a comprehensive depiction of the tongue's functional state; on the other hand, there is no effective technical solution that can regionally locate (i.e., accurately pinpoint the damaged tongue muscle groups) and identify lesion types (such as neurological lesions and myogenic lesions) of the tongue based on the aforementioned multimodal fusion reconstruction results. Existing technologies are mostly single-dimensional detections, with each detection result independent and lacking correlation, failing to achieve complementary verification of electrophysiological, motor, morphological, and anatomical information. This limits the accuracy and reliability of diagnosis, only allowing for a qualitative judgment of "the presence of lesions," making it difficult to support the precise formulation of clinical treatment plans and the quantitative evaluation of rehabilitation effects.

[0006] Therefore, to address the aforementioned pain points in clinical diagnosis and meet the core needs of precision medicine for "early detection, precise localization, and quantitative assessment" of tongue neuromuscular lesions, a novel digital framework for tongue function analysis is urgently needed. This framework would achieve deep integration and mutual corroboration of information on tongue movement, tissue morphology changes, and electrophysiological activity. Simultaneously, by combining prior information on tongue muscle anatomy, it would accurately locate and identify the type of tongue neuromuscular lesions, providing scientific and objective quantitative evidence for clinical diagnosis and treatment, thus overcoming the shortcomings of existing technologies. Summary of the Invention

[0007] To address the aforementioned problems, the present invention aims to provide a method and system for morphological reconstruction of tongue movement and identification of neuromuscular lesions. Overcoming the limitations of existing tongue detection technologies in achieving accurate reconstruction of tongue movement, morphological inference, and localization of neuromuscular lesions, this invention proposes a multimodal fusion method for three-dimensional tongue movement-morphology reconstruction based on tongue electrophysiological signals (sEMG), inertial measurement unit (IMU) motion signals, tongue surface geometric deformation signals, and a priori three-dimensional anatomical model of the tongue muscles. This method, combined with regional functional analysis, enables automatic identification of tongue neuromuscular lesions. The key challenges addressed are: first, how to convert multimodal signals from the tongue surface into interpretable three-dimensional tongue movement trajectories and morphological sequences; second, how to establish a correspondence between the reconstruction results and the anatomical structure of the tongue muscles; and third, how to infer neuromuscular lesions of specific tongue muscle groups from electrophysiological, motor, and deformation characteristics.

[0008] The above-mentioned objective of this invention is achieved through the following technical solutions: A method for morphological reconstruction of tongue movement and identification of neuromuscular lesions includes the following steps: S1: Perform multimodal signal acquisition, use a flexible electrode array attached to the tongue surface to acquire multimodal raw signals and record system timestamps. The multimodal raw signals include surface electromyography (sEMG) signals, inertial measurement unit (IMU) signals, tongue surface geometric deformation signals and tongue-palatal contact pressure event sequences. S2: Perform signal preprocessing and temporal feature extraction. Targeted filtering, artifact removal, and feature extraction are performed on the original multimodal signal. Cross-modal time axis alignment is achieved through multimodal time synchronization to obtain the electromyographic temporal coupling feature set of the multimodal preprocessed signal set and the tongue movement. S3: Perform joint estimation of tongue surface deformation and tongue body posture. Based on the processing results of step S2, reconstruct the local dynamic surface of the tongue surface using a preset model, and infer the overall pose sequence of the tongue body by combining IMU signals. S4: Perform dynamic fusion reconstruction of the three-dimensional tongue body, introduce a prior anatomical model of the tongue body containing multiple anatomical regions and constraints, construct a joint optimization objective function by combining tongue surface deformation data and pose information, and obtain the three-dimensional dynamic morphological sequence of the tongue body through iterative solution; S5: Perform MUAP analysis and construct regional functional maps. Perform motor unit action potential (MUAP) decomposition and feature extraction on surface electromyography (sEMG) signals. Map the extracted features and tongue movement-related data to the tongue anatomical model to generate a tongue regional functional map (TFM). S6: Perform lesion feature extraction and diagnostic reasoning, construct lesion feature vectors based on multi-dimensional abnormal features, generate tongue lesion risk map TDM through machine learning model analysis, and output the potentially damaged tongue muscle areas, nerve branches and lesion types.

[0009] Further, in step S1, the inertial measurement unit (IMU) signal includes triaxial acceleration and triaxial angular velocity; the tongue surface geometric deformation signal is obtained by the distance between electrode nodes, the stretching matrix or the stretching sensor array inside the flexible patch, and the distance measurement unit between electrodes; the surface electromyography (sEMG) signal is a surface electromyography potential sequence with several channels; the tongue and palate contact pressure event sequence is used to indicate the motion phase.

[0010] Furthermore, in step S2, the targeted processing specifically includes: bandpass filtering and artifact removal of the surface electromyography (sEMG) signal, low-pass filtering of the inertial measurement unit (IMU) signal, smoothing of the tongue surface geometric deformation signal, and threshold detection of the tongue-palatal contact pressure event signal. The artifact removal employs independent component analysis (ICA) or time-frequency domain denoising methods. The temporal features extracted include EMG-onset, MOV-onset, peak time, duration of movement, acceleration integral, tongue surface deformation center, curvature features, and local maximum deformation point. The multimodal time synchronization method includes using specific action events as synchronization anchors, using cross-correlation to calculate the optimal alignment offset, introducing system clock offset compensation, and further employing a motion phase segmentation method with tongue and palate pressure events as start and end markers.

[0011] Furthermore, in step S3, the preset model is a thin plate spline TPS or a finite element model, and the finite element model is based on triangular mesh or tetrahedral mesh modeling; the inference method for the overall pose sequence of the tongue is to use the acceleration and gyroscope signals of the IMU combined with extended Kalman filtering, or to achieve real-time inference through a deep inference network.

[0012] Further, in step S4, the anatomical region of the tongue anatomy prior model includes longitudinal muscles, transverse muscles, vertical muscles, genioglossus, hyoidoglossus, and stylohyoidus, and further includes the direction of tongue fibers, muscle bundle orientation, and regional elastic parameters; the constraints include continuity constraints, impenetrability constraints, movement range constraints of each anatomical region, and structural stress-deformation relationship constraints; the joint optimization objective function includes deformation field matching error, pose prediction error, anatomical structure deviation, temporal continuity constraints, and EMG activation intensity sub-terms for some regions; the spatial topological relationship of the electrode fixing points is incorporated during the iterative solution process.

[0013] Further, in step S5, MUAP decomposition uses sparse coding, blind source separation, convolutional sparse coding, or deep neural networks to perform dealiasing on sEMG. The extracted features include motor unit recruitment patterns, firing frequency, conduction velocity, unit synchronicity, and neural drive intensity estimation. The tongue movement-related data includes local deformation amplitude and regional movement trajectory. The tongue region functional map (TFM) includes regional activation intensity, regional deformation amplitude, regional movement path, and regional stress-deformation coupling curve, and the electromyographic activation intensity and regional deformation amplitude are fused into a comprehensive functional index according to weights.

[0014] Further, in step S6, the multidimensional abnormal features include MUAP abnormality, regional motion abnormality, deformation abnormality, EMG-motor temporal coupling abnormality, and anatomical mapping abnormality. Among them, MUAP abnormality includes neural and myogenic abnormalities, specifically including low activation, reduced conduction velocity, and abnormal synchronization; regional motion abnormality includes limited motion amplitude and deviation; deformation abnormality includes contralateral asymmetry and local collapse; EMG-motor temporal coupling abnormality includes increased delay and lack of correspondence; and anatomical mapping abnormality includes mismatch between activated area and expected area. The lesion feature vectors include EMG-motion coupling delay (EMD), left-right symmetry index (SI), and local deformation stiffness changes; the machine learning models include graph convolutional networks and Bayesian inference models; in the output results, the tongue muscle region includes the genioglossus and hyoidoglossus muscles, the nerve branches include the hypoglossal nerve or the lateral branches of the lingual nerve, and the lesion types include neurological, myogenic, and postoperative structural lesions; the method ultimately generates a three-dimensional dynamic model of the tongue, a functional map of the tongue region (TFM), and a risk map of tongue lesions (TDM), which are used for clinical diagnostic report output; If a decrease in left / right tongue movement and a reduction in MUAP conduction velocity are detected, it is diagnosed as possible damage to one branch of the hypoglossal nerve; if a significant reduction in local deformation is detected but MUAP activity is normal, it is diagnosed as a risk of structural or fibrotic lesions of the tongue; if MUAP activation is weakened and deformation is almost gone, it is judged as a myogenic lesion or postoperative scar traction.

[0015] A tongue movement morphology reconstruction and neuromuscular lesion identification system for performing the above-described tongue movement morphology reconstruction and neuromuscular lesion identification method includes: The multimodal signal acquisition module is used to acquire multimodal signals. It uses a flexible electrode array attached to the tongue surface to acquire multimodal raw signals and record system timestamps. The multimodal raw signals include surface electromyography (sEMG) signals, inertial measurement unit (IMU) signals, tongue surface geometric deformation signals, and tongue-palatal contact pressure event sequences. The preprocessing and feature extraction module is used to perform signal preprocessing and temporal feature extraction. It performs targeted filtering, artifact removal and feature extraction on the multimodal raw signal, and achieves cross-modal time axis alignment through multimodal time synchronization to obtain the electromyographic temporal coupling feature set of multimodal preprocessed signal set and tongue movement. The tongue surface deformation and posture estimation module is used to jointly estimate the tongue surface deformation and tongue body posture. Based on the processing results of step S2, the module reconstructs the local dynamic surface of the tongue surface using a preset model, and infers the overall posture sequence of the tongue body by combining IMU signals. The 3D tongue dynamic reconstruction module is used to perform 3D tongue dynamic fusion reconstruction. It introduces a tongue anatomical prior model containing multiple anatomical regions and constraints, and constructs a joint optimization objective function by combining tongue surface deformation data and pose information. The 3D dynamic morphological sequence of the tongue is obtained through iterative solution. The MUAP analysis TFM construction module is used to perform MUAP analysis and regional functional map construction. It performs motor unit action potential (MUAP) decomposition and feature extraction on surface electromyography (sEMG) signals, maps the extracted features and tongue movement-related data to the tongue anatomical model, and generates tongue regional functional maps (TFM). The lesion feature extraction and diagnosis module is used to extract lesion features and make diagnostic inferences. It constructs lesion feature vectors based on multi-dimensional abnormal features, and generates a tongue lesion risk map (TDM) through machine learning model analysis, outputting the potentially damaged tongue muscle areas, nerve branches, and lesion types.

[0016] A computer device, characterized in that it includes a memory and one or more processors, wherein the memory stores computer code, and when the computer code is executed by the one or more processors, causes the one or more processors to perform the method as described above.

[0017] A computer-readable storage medium, characterized in that the computer-readable storage medium stores computer code, which, when executed, is performed as described above.

[0018] Compared with the prior art, the present invention has at least one of the following beneficial effects: (1) The first dynamic fusion reconstruction technology of tongue three-dimensional motion and morphology has overcome the technical bottleneck that traditional visual observation can only capture surface appearance and single-channel sensors cannot restore spatial motion. It realizes real-time and accurate depiction of the tongue’s real motion trajectory and morphological changes, filling the technical gap of tongue dynamic multi-dimensional representation.

[0019] (2) The tongue electrophysiological signal (sEMG), motion signal (IMU) and geometric deformation signal are innovatively modeled together. Through cross-validation and complementary verification of multimodal data, the limitations and errors of a single data dimension are effectively avoided, and the accuracy, stability and reliability of the diagnostic results are significantly improved, providing a more comprehensive quantitative basis for clinical decision-making.

[0020] (3) By relying on the three-dimensional anatomical prior structure of the tongue muscles to locate regional lesions, it is possible to accurately pinpoint the specific tongue muscle groups (such as the genioglossus muscle, hyoidoglossus muscle, etc.) or branches of the tongue nerve (such as the mandibular nerve and hypoglossal nerve branches) involved in the lesion. This core diagnostic capability is beyond the reach of traditional electromyography (EMG) which can only record electrical activity, and imaging methods such as ultrasound / MRI which are difficult to associate with function and anatomical structure, thus providing direct guidance for precision treatment.

[0021] (4) It has strong clinical adaptability and can be widely used in various scenarios such as dynamic monitoring of postoperative rehabilitation effects of tongue-related diseases, graded assessment of swallowing dysfunction, early screening of neurological tongue neuromuscular lesions, and auxiliary diagnosis of speech dysfunction, covering the entire clinical chain from disease screening and treatment guidance to rehabilitation assessment.

[0022] (5) Adopting a software and hardware decoupling design architecture, the core algorithm and specific hardware devices are independent of each other, with good compatibility and scalability, which can easily realize cross-brand and cross-system integrated applications, greatly reducing the cost of equipment updates and technology adaptation in medical institutions, and laying a solid foundation for the large-scale promotion and industrialization of the technology. Attached Figure Description

[0023] Figure 1 This is an overall flowchart of the method for morphological reconstruction of tongue movement and identification of neuromuscular lesions of the present invention; Figure 2 This is a structural diagram of the tongue movement morphology reconstruction and neuromuscular lesion recognition system of the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0026] This invention proposes a multimodal fusion method for three-dimensional reconstruction of the tongue and diagnosis of neuromuscular disorders based on electrophysiological signals, inertial measurement signals, tongue surface geometric deformation signals, and prior anatomical information of the tongue. Existing technologies for detecting tongue dysfunction mainly rely on MRI, ultrasound, or subjective clinical assessment, which cannot simultaneously acquire information on local electrophysiological activity, overall movement trajectory, local deformation behavior, and muscle anatomical differentiation of the tongue under dynamic conditions. Therefore, it is difficult to achieve quantitative analysis of deep functional changes in the tongue. To address this technological gap, this invention establishes a dynamic tongue reconstruction framework with high spatiotemporal resolution, anatomical consistency, and functional interpretation capabilities through multi-source data fusion.

[0027] The technical features and innovations of this invention are reflected in the following aspects: (1) A method for acquiring multimodal signals by combining a flexible electrode array on the tongue surface with an inertial measurement unit and a deformation sensing structure is proposed for the first time. This method can simultaneously obtain the surface electromyographic activity, local configuration changes and overall spatial motion of each region of the tongue, providing a basis for real-time capture of the biomechanical behavior of the tongue.

[0028] (2) Tongue surface deformation reconstruction is achieved by using thin plate splines or finite element models, and a three-dimensional tongue dynamic model is established by combining IMU attitude estimation and tongue body anatomy priors, which effectively solves the problem that traditional methods cannot simultaneously capture the complex morphology of the tongue body and the deep muscle driving mechanism.

[0029] (3) A method for mapping the functional areas of the tongue body based on the action potential analysis of motor units is proposed, which maps the electromyographic activation, deformation amplitude and local movement trajectory to the anatomical area of ​​the tongue muscle to form a functional map of the tongue body area with anatomical significance.

[0030] (4) A lesion feature vector integrating MUAP abnormality, motor abnormality, deformation abnormality and anatomical deviation was established, and an automated neuromuscular lesion reasoning model was constructed, which can locate the tongue muscle group or nerve branch that may be damaged, providing a quantitative basis for the determination of the type of neurogenic, myogenic and postoperative structural lesions.

[0031] In summary, this invention achieves multi-level data fusion of the tongue, encompassing electrical activity, kinematics, geometry, and anatomical structure, overcoming the technical limitations of traditional tongue function assessment techniques in achieving dynamic, high spatiotemporal resolution, and anatomically driven comprehensive diagnosis. This invention not only provides an effective tool for the detection of clinical neuromuscular diseases but also offers a new technological foundation for the fields of tongue dynamics, speech movement research, and oral rehabilitation.

[0032] The technical solution of this invention mainly includes: (1) Multimodal signal acquisition and preprocessing The following tongue surface signals were acquired and synchronized with noise reduction: Surface electromyography (sEMG) signals; IMU acceleration and angular velocity signals; Electrode deformation sensing signal; tongue and palate pressure event sequence; Timestamp sequence.

[0033] After synchronization, filtering, artifact removal, and normalization, a multimodal preprocessed signal set is obtained.

[0034] (2) Multimodal temporal feature extraction Extract from the preprocessed signal: EMG-onset time; Motion start time (MOV-onset); Peak time, duration of exercise; Local deformation initiation point and amplitude.

[0035] The tongue movement-electromyography temporal coupling feature set was obtained.

[0036] (3) Reconstruction of tongue surface geometric deformation Utilizing the nodal displacement variations of the flexible electrode array, the following approach is adopted: Thin-Plate Spline (TPS) or finite element deformation model Construct a local dynamic surface on the tongue.

[0037] (4) Three-dimensional movement of the tongue body - morphological fusion reconstruction The three-dimensional morphology of the tongue is constructed based on the following information: Local geometric deformation of the tongue surface; IMU provides orientation and pose changes; Three-dimensional anatomical mesh model of the tongue muscles; Motion constraints (continuity, impenetrability); The spatial topological relationship of the electrode fixing points.

[0038] A structural constraint optimization model was established to obtain the three-dimensional dynamic morphological sequence of the tongue.

[0039] (5) Muscle power source analysis and motor unit feature extraction By using blind source separation, convolutional sparse coding, or deep learning models to decompose sEMG into Motor Unit Action Potential (MUAP), we obtain: Recruitment Mode Frequency characteristics Conduction speed Unit Synchronization Neural drive strength estimation (6) Functional mapping of tongue muscle regions Projecting MUAP features into anatomical space achieves the following: Tongue muscle group region matching Regional activation intensity reconstruction Functional connectivity inference Local muscle stress / deformation inference The tongue functional map (TMM) was obtained.

[0040] (7) Identification of neuromuscular lesions of the tongue Construct a lesion identification model and conduct comprehensive analysis: MUAP abnormalities (neurological vs. myogenic) Abnormal range of motion in the region Regional deformation asymmetry or abnormal path Anatomical mapping abnormality Timing Coupling Anomaly Output a Tongue Dysfunction Map (TDM) and indicate the most likely areas of tongue muscle group or nerve branch lesions.

[0041] The following is an illustration through specific examples: First Embodiment like Figure 1 As shown, this embodiment provides a method for morphological reconstruction of tongue movement and identification of neuromuscular lesions, including the following steps: S1: Perform multimodal signal acquisition, use a flexible electrode array attached to the tongue surface to acquire multimodal raw signals and record system timestamps. The multimodal raw signals include surface electromyography (sEMG) signals, inertial measurement unit (IMU) signals, tongue surface geometric deformation signals and tongue-palatal contact pressure event sequences.

[0042] In step S1, the inertial measurement unit (IMU) signal includes triaxial acceleration and triaxial angular velocity; the tongue surface geometric deformation signal is obtained by the distance between electrode nodes, the stretching matrix or the stretching sensor array inside the flexible patch, and the distance measurement unit between electrodes; the surface electromyography (sEMG) signal is a series of surface electromyographic potentials with several channels; the tongue and palate contact pressure event sequence is used to indicate the motion phase.

[0043] Step S1, as the basic data acquisition step of the entire technical solution, has the core objective of acquiring raw data that can comprehensively characterize the physiological function of the tongue through a highly adaptable and multi-dimensional signal acquisition strategy, so as to provide high-quality data support for subsequent multimodal fusion modeling, three-dimensional reconstruction and lesion identification.

[0044] Among them, the triaxial acceleration and triaxial angular velocity signals of the inertial measurement unit (IMU) can capture the overall motion trend and posture changes of the tongue in three-dimensional space in real time, including the rate and direction information of actions such as tongue extension, tongue curling, and left and right swinging, providing core kinematic data for the reconstruction of the overall motion trajectory of the tongue; the geometric deformation signal of the tongue surface is acquired through multiple methods. The measurement of the distance between electrode nodes and the stretching matrix can directly reflect the stretching or contraction state of local tissues on the tongue surface, while the stretching sensor array and the distance measurement unit built into the flexible patch further improve the sensitivity and spatial resolution of deformation detection, adapting to the irregular and dynamically changing surface of the tongue. Physiological characteristics; the surface electromyography (sEMG) signal is designed with a multi-channel acquisition mode, which can simultaneously record the electrophysiological activities of different tongue muscle groups. Each channel corresponds to the muscle fiber discharge status of a specific region, providing rich electrophysiological evidence for subsequent motor unit action potential (MUAP) decomposition and muscle group function analysis; the tongue-palatal contact pressure event sequence can not only accurately indicate the key phases of tongue movement (such as the start and end times of tongue elevation and contact with the palate), but also serve as an important reference anchor point for multimodal signal time synchronization, ensuring the consistency of different types of signals such as electrophysiology, movement, and deformation in the time dimension, laying the foundation for subsequent temporal feature extraction and cross-modal coupling analysis.

[0045] The entire acquisition process relies on a flexible electrode array attached to the tongue surface. This array has good biocompatibility and deformation adaptability, can closely fit the tongue surface without affecting the tongue's natural physiological movement, ensuring the authenticity and integrity of the original signal acquisition, and effectively avoiding data distortion caused by interference from the acquisition equipment.

[0046] S2: Perform signal preprocessing and temporal feature extraction. Targeted filtering, artifact removal, and feature extraction are performed on the original multimodal signals. Cross-modal time axis alignment is achieved through multimodal time synchronization to obtain the electromyographic temporal coupling feature set of the multimodal preprocessed signal set and the tongue movement.

[0047] In step S2, the targeted processing specifically includes: bandpass filtering and artifact removal of the surface electromyography (sEMG) signal, low-pass filtering of the inertial measurement unit (IMU) signal, smoothing of the tongue surface geometric deformation signal, and threshold detection of the tongue-palatal contact pressure event signal. The artifact removal employs independent component analysis (ICA) or time-frequency domain denoising methods. The temporal features extracted include EMG-onset, MOV-onset, peak time, duration of movement, acceleration integral, tongue surface deformation center, curvature features, and local maximum deformation point. The multimodal time synchronization method includes using specific action events as synchronization anchors, using cross-correlation to calculate the optimal alignment offset, introducing system clock offset compensation, and further employing a motion phase segmentation method with tongue and palate pressure events as start and end markers.

[0048] Step S2, as the core intermediate link connecting the original signal acquisition with subsequent 3D reconstruction and lesion analysis, aims to transform multi-source heterogeneous original signals into standardized, high-value, and time-synchronized effective data through integrated processing of "signal purification, feature extraction, and temporal alignment," thus providing a solid foundation for subsequent cross-modal fusion modeling and accurate diagnosis.

[0049] The targeted processing strategies are designed based on the physical characteristics and physiological significance of different signals: bandpass filtering is used for surface electromyography (sEMG) signals to accurately retain the effective frequency band of muscle fiber discharge (usually 20-500Hz), filter out noise such as power frequency interference and baseline drift, and remove motion artifacts such as swallowing movements and electrode displacement by combining independent component analysis (ICA) or time-frequency domain denoising methods to ensure the purity of the EMG signal; low-pass filtering is used for inertial measurement unit (IMU) signals to smooth high-frequency jitter noise in acceleration and angular velocity, restoring the smooth trajectory of the tongue's overall movement; smoothing of tongue surface geometric deformation signals can eliminate local fluctuations caused by electrode measurement errors and accurately reflect the true deformation trend of the tongue surface tissue; threshold detection of tongue-palatal contact pressure event signals can accurately screen out key event points of contact / separation between the tongue and palate, providing a clear basis for motion phase division.

[0050] The temporal feature extraction stage focuses on mining key physiological information from the signal: the extraction of electromyographic onset time (EMG-onset) and motor onset time (MOV-onset) can quantify the temporal coupling relationship between electromyographic activity and mechanical movement, providing a basis for judging neuromuscular conduction function; features such as peak time, duration of movement, and acceleration integral can objectively characterize the intensity, speed, and range of tongue movement; tongue surface deformation center, curvature features, and local maximum deformation points can accurately capture the core area and degree of tongue surface morphological changes. The tongue movement-electromyographic temporal coupling feature set constituted by these features is the core data source for subsequent pathological feature analysis.

[0051] Multimodal time synchronization is crucial for ensuring the validity of cross-modal data: by using specific motion events (such as rapid upward movement) as synchronization anchors, initial time references for each modality signal can be established; the optimal alignment offset calculated using the cross-correlation method can correct for minor time delays during signal transmission; introducing system clock offset compensation can solve the clock error problem between different acquisition modules; and further, using the palatal pressure event as the start and end marker for motion phase segmentation can refine the synchronization accuracy to within a single motion cycle, ensuring that different types of signals such as electrophysiological, motion, and deformation are completely matched in the time dimension, providing time consistency assurance for subsequent joint modeling and cross-validation of multimodal data.

[0052] The entire processing logic of step S2 revolves around "improving data quality, mining effective information, and ensuring time consistency." It avoids noise interference and time deviation in the original signal and extracts core features with clear physiological significance. This provides standardized and highly reliable data support for subsequent tongue surface geometric deformation reconstruction, three-dimensional motion-morphology fusion, and neuromuscular lesion identification, and is a key guarantee for achieving the accuracy of the entire technical solution.

[0053] S3: Perform joint estimation of tongue surface deformation and tongue posture. Based on the processing results of step S2, use a preset model to reconstruct the local dynamic surface of the tongue, and combine IMU signals to infer the overall pose sequence of the tongue.

[0054] In step S3, the preset model is a thin plate spline TPS or a finite element model, and the finite element model is based on triangular mesh or tetrahedral mesh modeling; the inference method for the overall pose sequence of the tongue is to use the acceleration and gyroscope signals of the IMU combined with extended Kalman filtering, or to achieve real-time inference through a deep inference network.

[0055] Step S3, as a key link connecting signal preprocessing and three-dimensional dynamic fusion reconstruction, aims to simultaneously acquire the surface morphological changes and spatial motion state of the tongue through the joint processing of "refined reconstruction of local deformation + accurate estimation of overall posture". This provides core data support for the subsequent construction of a complete and consistent three-dimensional dynamic model of the tongue, and achieves the coordinated representation of local details and overall motion.

[0056] The selection of preset models is well-suited to the physiological characteristics of the tongue and the accuracy requirements of detection: the thin plate spline (TPS) model has a strong ability to fit flexible curved surfaces, and can flexibly restore the irregular local deformation trend of the tongue surface based on the displacement data of the electrode array nodes, which is especially suitable for capturing the dynamic surface changes of flexible parts such as the tip and edge of the tongue; while the finite element model, through triangular mesh or tetrahedral mesh modeling, divides the tongue surface into discrete units, which can accurately calculate the stress and deformation relationship of each unit, and is more suitable for scenarios that require quantitative analysis of the mechanical properties of tongue surface tissues. The optional design of the two models allows the technical solution to adapt to different clinical needs and hardware conditions. Regardless of which model is used, the standardized signal processed in step S2 is used as input to ensure the accuracy and reliability of deformation reconstruction.

[0057] The inference of the overall tongue pose sequence focuses on reconstructing the tongue's trajectory and posture changes in three-dimensional space: IMU acceleration signals can be used to calculate the tongue's displacement and acceleration changes, while gyroscope signals can capture the tongue's rotation angle and angular velocity in real time. Combined with the extended Kalman filter algorithm, the advantages of both signals can be effectively integrated, suppressing measurement noise and achieving real-time, stable pose estimation, meeting the timeliness requirements of dynamic detection. Meanwhile, the deep inference network, by learning from a large amount of tongue motion sample data, can extract more complex motion features from the IMU signals, further improving the accuracy of pose estimation, especially suitable for scenarios with complex motion patterns. The design of these two inference methods ensures both the real-time performance of the technical solution and meets the high-precision requirements.

[0058] The entire S3 process, through the combined processing of local deformation and overall posture, breaks through the limitations of the traditional technique of separating the representation of local and overall motion, ensuring the consistency of the local dynamic curved surface of the tongue and the overall pose sequence in the temporal and spatial dimensions. This lays a solid foundation for the subsequent steps of integrating the prior model of tongue muscle anatomy to construct a complete three-dimensional dynamic morphological sequence of the tongue, and is the core technical support for achieving accurate reconstruction of tongue motion and morphology.

[0059] S4: Perform dynamic fusion reconstruction of the three-dimensional tongue body, introduce a prior anatomical model of the tongue body containing multiple anatomical regions and constraints, construct a joint optimization objective function by combining tongue surface deformation data and pose information, and obtain the three-dimensional dynamic morphological sequence of the tongue body through iterative solution.

[0060] In step S4, the anatomical region of the tongue anatomy prior model includes the longitudinal muscles, transverse muscles, vertical muscles, genioglossus, hyoidoglossus, and stylohyoidus muscles, and further includes the direction of tongue fibers, muscle bundle orientation, and regional elastic parameters; the constraints include continuity constraints, impenetrability constraints, movement range constraints of each anatomical region, and structural stress-deformation relationship constraints; the joint optimization objective function includes deformation field matching error, pose prediction error, anatomical structure deviation, temporal continuity constraints, and EMG activation intensity sub-terms for some regions; the spatial topological relationship of the electrode fixing points is incorporated during the iterative solution process.

[0061] Step S4 is the core link in the whole technical solution to realize the integrated representation of the tongue's "morphology-movement-anatomy". The core objective is to organically combine local deformation, overall posture and anatomical structure through deep fusion and constraint optimization of multi-source information, and generate a three-dimensional dynamic morphological sequence of the tongue that has physiological authenticity, spatial accuracy and dynamic continuity, so as to provide complete three-dimensional model support for subsequent functional analysis and lesion localization.

[0062] The introduction of a priori anatomical model of the tongue is key to ensuring the physiological rationality of the reconstruction results: the anatomical regions covered, such as longitudinal muscles, transverse muscles, vertical muscles, genioglossus, hyoidoglossus, and styloglossus, provide a precise structural framework for reconstruction, ensuring that the model is consistent with the actual anatomical structure of the human tongue muscles; while the additional tongue fiber direction, muscle bundle orientation, and regional elasticity parameters further endow the model with biomechanical properties, making the deformation and movement of the reconstruction conform to the physiological and mechanical laws of the tongue muscle tissue, avoiding false shapes that deviate from the actual anatomical structure.

[0063] The design of constraints limits the rationality boundary of the model from multiple dimensions: continuity constraints ensure that the tongue tissue does not break or separate during movement, which conforms to the integrity of biological tissue; impermeability constraints prevent the tongue from physically penetrating itself or surrounding tissues, which conforms to the physiological environment; the range of motion constraints of each anatomical region limit the activity limits of different tongue muscle groups, preventing abnormal postures that exceed the normal physiological range of motion; and structural stress-deformation relationship constraints link tongue deformation with muscle stress, making morphological changes more biomechanically logical.

[0064] The construction of the joint optimization objective function realizes the collaborative calibration of multi-source data: deformation field matching error ensures that the reconstructed tongue surface deformation is consistent with the local deformation data in step S3, ensuring the accuracy of surface morphology; pose prediction error makes the overall posture match the pose sequence inferred by the IMU, ensuring the accuracy of spatial motion; anatomical structure deviation constraint model does not deviate from the anatomical prior of the tongue muscle, avoiding structural distortion; temporal continuity constraint makes the morphological changes of adjacent frames in the dynamic sequence transition smoothly without jumps or abrupt changes; the EMG activation intensity sub-item in some regions associates electromyographic activity with morphological motion, so that the reconstruction results reflect the physiological logic of "electromyographic-driven movement", realizing the coupling of function and morphology.

[0065] Incorporating the spatial topological relationship of the electrode fixed points into the iterative solution process is an important detail for improving reconstruction accuracy: by keeping the relative spatial position relationship between the electrode nodes unchanged, a precise spatial correspondence between the signal acquisition points and the reconstruction model is established, ensuring that the calculation of deformation and motion always uses the measured signal as the anchor point, effectively suppressing accumulated errors, and making the three-dimensional dynamic morphological sequence both conform to physiological laws and highly consistent with the original acquisition data.

[0066] The core logic of the entire S4 step is "multi-source information fusion + multi-constraint calibration". By giving the model a physiological basis through anatomical priors, by limiting reasonable boundaries through constraints, and by optimizing the objective function to coordinate multi-dimensional data, the model is finally transformed from scattered local deformation and overall posture data to a complete, realistic, and dynamic three-dimensional tongue model. This lays a key foundation for the subsequent association of morphological movement with electromyographic function and anatomical structure.

[0067] S5: Perform MUAP analysis and construct regional functional maps. Perform motor unit action potential (MUAP) decomposition and feature extraction on surface electromyography (sEMG) signals. Map the extracted features and tongue movement-related data to the tongue anatomical model to generate a tongue regional functional map (TFM).

[0068] In step S5, MUAP decomposition uses sparse coding, blind source separation, convolutional sparse coding, or deep neural networks to de-alias the sEMG. The extracted features include motor unit recruitment patterns, firing frequency, conduction velocity, unit synchronicity, and neural drive intensity estimation. The tongue movement-related data includes local deformation amplitude and regional movement trajectory. The tongue region functional map (TFM) includes regional activation intensity, regional deformation amplitude, regional movement path, and regional stress-deformation coupling curve. The electromyographic activation intensity and regional deformation amplitude are fused into a comprehensive functional index according to weights.

[0069] Step S5, as the core link connecting in-depth analysis of electromyographic signals and visualization of tongue function, aims to transform abstract electrophysiological signals and concrete motion and deformation data into anatomically significant functional atlases through "precise analysis of motor units + multi-dimensional data anatomical mapping," providing a three-in-one quantitative basis of "electrophysiology-motor-anatomy" for subsequent lesion localization.

[0070] Among these techniques, MUAP decomposition is key to uncovering deeper physiological information from electromyography (EMG) signals. The raw sEMG signal is essentially a mixture of electrical activity from multiple motor units. Dealiasing using sparse coding, blind source separation, convolutional sparse coding, or deep neural networks enables precise separation of action potentials from individual motor units, overcoming the limitations of traditional EMG analysis at the "overall signal level." The extracted motor unit recruitment patterns reflect the number and activation sequence of motor units during muscle activation, directly correlated with muscle contraction intensity. Discharge frequency reflects the excitability of motor units and is positively correlated with muscle movement strength. Conduction velocity reflects the conduction function of nerve fibers and is a core indicator for assessing nerve damage. Unit synchronicity reflects the coordinated activation state of multiple motor units, reflecting muscle functional coordination. Neural drive strength estimation quantifies the central nervous system's ability to drive muscles, providing direct evidence for assessing the integrity of the neuromuscular conduction pathway.

[0071] The joint mapping of tongue movement-related data with MUAP features is the core logic for achieving functional and anatomical correlation: the amplitude of local deformation and the trajectory of regional movement are derived from the reconstruction results of previous steps, representing the mechanical movement state of the tongue; by projecting them together with MUAP features onto the tongue muscle anatomical model, a correspondence between "electrophysiological activity (neural drive, muscle activation) - mechanical movement (deformation, trajectory) - anatomical location (specific tongue muscle group)" can be established, avoiding the disconnect between electrophysiological signals and movement state analysis, and making functional assessment more spatially oriented.

[0072] The generation of the Tongue Region Function Map (TFM) is the core output of this step: TFM integrates regional activation intensity (reflecting the level of local electromyographic activation), regional deformation amplitude (reflecting the degree of mechanical deformation), regional movement path (reflecting the spatial movement law), and regional stress-deformation coupling curve (reflecting the muscle mechanical properties), comprehensively depicting the functional state of each tongue muscle region; while the comprehensive functional index formed by merging electromyographic activation intensity and regional deformation amplitude according to weights further realizes the quantitative integration of multi-dimensional functional information, and intuitively reflects the functional integrity of the region through a single index, which simplifies the complexity of subsequent lesion analysis and ensures the comprehensiveness of the assessment.

[0073] The core logic of step S5 is "from signal analysis to functional visualization". By decomposing and mining deep electrophysiological information through MUAP, and establishing the association with anatomical structures through multi-data joint mapping, the final generated TFM not only realizes the spatial and quantitative representation of tongue function, but also builds a bridge between the preceding signal processing, three-dimensional reconstruction and subsequent lesion identification, enabling lesion analysis to accurately locate specific tongue muscle areas, providing key technical support for achieving "regional lesion localization".

[0074] S6: Perform lesion feature extraction and diagnostic reasoning, construct lesion feature vectors based on multi-dimensional abnormal features, generate tongue lesion risk map TDM through machine learning model analysis, and output the potentially damaged tongue muscle areas, nerve branches and lesion types.

[0075] In step S6, the multidimensional abnormal features include MUAP abnormality, regional motion abnormality, deformation abnormality, EMG-motor temporal coupling abnormality, and anatomical mapping abnormality. Among them, MUAP abnormality includes neural and myogenic abnormalities, specifically including low activation, reduced conduction velocity, and abnormal synchronization; regional motion abnormality includes limited motion amplitude and deviation; deformation abnormality includes contralateral asymmetry and local collapse; EMG-motor temporal coupling abnormality includes increased delay and lack of correspondence; and anatomical mapping abnormality includes mismatch between activated area and expected area. The lesion feature vectors include EMG-motion coupling delay (EMD), left-right symmetry index (SI), and local deformation stiffness changes; the machine learning models include graph convolutional networks and Bayesian inference models; in the output results, the tongue muscle region includes the genioglossus and hyoidoglossus muscles, the nerve branches include the hypoglossal nerve or the lateral branches of the lingual nerve, and the lesion types include neurological, myogenic, and postoperative structural lesions; the method ultimately generates a three-dimensional dynamic model of the tongue, a functional map of the tongue region (TFM), and a risk map of tongue lesions (TDM), which are used for clinical diagnostic report output; If a decrease in left / right tongue movement and a reduction in MUAP conduction velocity are detected, it is diagnosed as possible damage to one branch of the hypoglossal nerve; if a significant reduction in local deformation is detected but MUAP activity is normal, it is diagnosed as a risk of structural or fibrotic lesions of the tongue; if MUAP activation is weakened and deformation is almost gone, it is judged as a myogenic lesion or postoperative scar traction.

[0076] Step S6, as the core diagnostic output link of the entire technical solution, aims to achieve accurate localization, type determination and visualization of tongue neuromuscular lesions through "multi-dimensional abnormal feature integration + quantitative modeling + intelligent reasoning", providing direct and objective quantitative evidence for clinical diagnosis and completing the final closed loop from data processing to diagnostic conclusion.

[0077] The construction of multi-dimensional abnormal features is the foundation for comprehensive diagnosis. Its design fully covers the entire chain of lesion manifestations across "electrophysiology-motor-morphology-anatomy": MUAP abnormalities distinguish between neurological and myogenic lesions; specific manifestations such as low activation, reduced conduction velocity, and abnormal synchronization directly reflect the functional state of the neuromuscular unit, which is crucial for determining the root cause of the lesion. Regional motor abnormalities (limited range of motion, deviation) and deformation abnormalities (contralateral asymmetry, local collapse) directly reflect damage to the tongue's mechanical motor function, reflecting the actual impact of the lesion on physiological function. EMG-motor temporal coupling abnormalities (increased delay, lack of correspondence) focus on the synergy between nerve signal transmission and muscle motor execution, capturing damage to the neuromuscular transmission pathway. Anatomical mapping abnormalities (mismatch between activated and expected areas) link function and anatomical structure, revealing the correspondence between functional abnormalities and specific tongue muscle groups, ensuring the accuracy of lesion localization. These five types of abnormal features complement each other, avoiding the one-sidedness of single-dimensional diagnosis and comprehensively covering the core manifestations of tongue neuromuscular lesions.

[0078] The construction of lesion feature vectors enables the quantitative integration of abnormal features: EMG-Motion Coupling Delay (EMD) transforms temporal coupling anomalies into quantifiable time indicators, accurately reflecting the degree of delay in neuromuscular transmission; the Left-Right Symmetry Index (SI) quantifies asymmetric anomalies by comparing the consistency of movement and deformation on both sides of the tongue; and changes in local deformation stiffness characterize alterations in the mechanical properties of the tongue tissue, providing a quantitative basis for judging structural lesions. Together, these three elements transform qualitative abnormal features into standardized quantitative vectors, laying the data foundation for accurate analysis by machine learning models.

[0079] The choice of machine learning models is well-suited to diagnostic needs: graph convolutional networks excel at handling the spatial topological relationships of anatomical regions of the tongue, effectively utilizing anatomical information to improve the accuracy of lesion localization; while Bayesian inference models possess powerful inference capabilities for uncertainties, outputting reliable lesion risk probabilities even when multimodal features are noisy or incomplete. This choice of two models ensures both the accuracy of spatial correlation analysis and the reliability of diagnostic results, efficiently processing quantified lesion feature vectors to generate a tongue lesion risk map (TDM).

[0080] The design of diagnostic rules and output results directly addresses clinical needs: It clearly defines the output range for tongue muscle regions (genioglossus, hyoidoglossus, etc.), nerve branches (lateral and branched branches of the hypoglossal nerve or lingual nerve), and lesion types (neurogenic, myogenic, postoperative structural lesions), making the diagnostic results more clinically relevant. Targeted diagnostic rules (such as unilateral motor impairment + decreased MUAP conduction velocity corresponding to hypoglossal nerve branch injury) are based on precise judgment logic summarized from numerous clinical scenarios, ensuring the practicality of the diagnostic results. The final generated three-dimensional dynamic model of the tongue, regional functional map (TFM), and lesion risk map (TDM) comprehensively present the diagnostic results from three dimensions: morphology, function, and lesion risk, providing rich and intuitive quantitative evidence for clinical diagnostic reports and assisting doctors in quickly developing treatment and rehabilitation plans.

[0081] The core logic of the entire step S6 is "from feature quantification to intelligent diagnosis". Through comprehensive coverage of multi-dimensional abnormal features, accurate construction of quantified vectors, and intelligent reasoning of the adaptive model, the precise localization, type identification and visualization output of tongue neuromuscular lesions are finally achieved. This fully reflects the core advantages of the invention of "multimodal fusion, anatomy-driven and quantitative diagnosis", and is a key link connecting the technology implementation and clinical application.

[0082] Second Embodiment This embodiment is based on a tongue multimodal fusion analysis system. Through a complete multimodal data acquisition, preprocessing, three-dimensional reconstruction, and lesion analysis process, it achieves lesion localization in patients suspected of hypoglossal nerve injury. The specific implementation process is as follows: I. Overall System Structure The tongue multimodal fusion analysis system used in this embodiment includes the following seven core software components, which work together to complete signal processing, reconstruction, and diagnosis: Signal acquisition and management module: responsible for receiving multimodal input signals from the tongue flexible patch, including surface electromyography (sEMG) signals, inertial measurement unit (IMU) signals, tongue surface geometric deformation signals, tongue and palate pressure contact event sequences and timestamp synchronization information; Signal preprocessing and alignment module: Implements signal filtering, feature processing, motion artifact removal, and multimodal time axis alignment; Tongue surface geometry reconstruction module: Utilizing the electrode array node position information, local geometry reconstruction of the tongue surface is achieved through thin-plate spline (TPS) or finite element method; Tongue 3D Motion - Morphological Fusion and Reconstruction Module: Combines tongue surface deformation, IMU posture, tongue muscle 3D anatomical prior model and structural constraints to generate a dynamic 3D motion model of the tongue. The muscle power source separation and motor unit (MU) feature extraction module: performs dealiasing on sEMG and extracts motor unit action potential (MUAP) features; Tongue muscle region functional mapping module: Corresponds local deformation and MUAP features to the three-dimensional model of tongue muscles to generate a tongue functional map (TFM). Neuromuscular lesion analysis module: Input motion trajectory, deformation field, MUAP abnormality and temporal coupling relationship into lesion identification model, output Tongue Dysfunction Map (TDM), and locate the tongue muscle area or nerve branch that may be damaged.

[0083] II. Multimodal Data Acquisition and Preprocessing Process (a) Multimodal data acquisition Subjects wore flexible tongue patches and performed standard tasks such as "extending the tongue, raising it, and swaying it from side to side." The following raw signals were collected: sEMG signal: 16-channel surface electromyographic potential sequence; IMU signal: 100Hz sampling rate per channel, including triaxial acceleration and triaxial angular velocity; Deformation signal: calculated from the distance between electrode nodes and the two-dimensional stretching matrix; Tongue-palatine contact event sequence: used to indicate the phase of motion; System timestamp: Used to align all modal signals.

[0084] (II) Artifact Removal and Filtering Bandpass filtering and artifact removal are performed on the sEMG signal; Low-pass filtering is applied to the IMU signal; Smooth the deformation signal; Threshold detection is performed on the tongue and palate pressure event signal.

[0085] (III) Multimodal time synchronization Achieve precise cross-modal alignment using the following three methods: Specific action events such as "rapid upward movement" are used as synchronization anchor points; The optimal alignment offset is calculated using the cross-correlation method; Introduce system clock skew compensation.

[0086] (iv) Feature extraction The following core features were extracted from the processed signal for subsequent motion reconstruction and lesion assessment: Electromyographic onset time (EMG-onset), motor onset time (MOV-onset); Peak value, duration of motion, integral of acceleration; Tongue surface deformation center, curvature characteristics, and local maximum deformation point.

[0087] III. Application of 3D Tongue Reconstruction Model Based on Multimodal Fusion A three-dimensional reconstruction model driven by a combination of tongue surface deformation, tongue posture information, and prior knowledge of tongue muscle anatomy is implemented as follows: Tongue surface geometric deformation reconstruction layer: Using electrode array node displacement data, the local tongue surface curvature change is reconstructed through the TPS model to construct the local dynamic surface of the tongue surface; Tongue pose estimation layer: Based on the acceleration and angular velocity information provided by the IMU, the overall posture change of the tongue is inferred by the extended Kalman filter; Anatomical prior constraint layer: Introduce a tongue muscle anatomical model including external muscles such as longitudinal muscles, transverse muscles, vertical muscles, genioglossus, hyoidoglossus, and styloglossus, and apply continuity constraints, impenetrable constraints, range of motion constraints for each anatomical region, and structural stress-deformation relationship constraints. Multimodal fusion optimization layer: A unified optimization objective function is established, which jointly considers deformation field matching error, pose prediction error, anatomical structure deviation, time continuity constraint and EMG activation intensity in some regions. Combined with the spatial topological relationship of electrode fixing points, the three-dimensional dynamic morphological sequence of the tongue is obtained through iterative solution.

[0088] IV. Application of Neuromuscular Disease Analysis Methods After obtaining the three-dimensional motion and deformation model of the tongue, the following steps are used to perform lesion analysis: MUAP decomposition and motor unit analysis: Convolutional sparse coding method is used to dealias and decompose sEMG, extracting features such as MU recruitment mode, MU firing frequency, conduction velocity, MU synchronization and neural drive strength estimation. Multimodal region functional mapping: The MUAP activated region is mapped to the tongue muscle anatomical model to construct a TFM that includes the region activation intensity, region deformation amplitude, region motion path and region stress-deformation coupling curve; Multimodal feature construction of lesions: The lesion feature vector is constructed from multiple dimensions, including MUAP abnormalities (low activation, reduced conduction velocity, abnormal synchronization, etc.), regional motion abnormalities (limited motion amplitude, deviation), deformation abnormalities (contralateral asymmetry, local collapse), EMG-motion temporal coupling abnormalities (increased delay, lack of correspondence), and anatomical mapping abnormalities (mismatch between activated area and expected area). The lesion feature vector specifically includes EMG-motion coupling delay (EMD), left-right symmetry index (SI), and local deformation stiffness change (Stiffness Shift). Lesion risk inference: The lesion feature vector is analyzed by graph convolutional network to generate a tongue lesion risk map (TDM).

[0089] V. Example Explanation Example 1: Three-dimensional reconstruction of tongue movement in normal subjects S1: Signal Acquisition: A subject with a flexible tongue patch was subjected to tasks such as "tongue extension—lifting—swinging left and right." 16-channel sEMG, 100Hz IMU signals per channel, electrode deformation signals in the form of a two-dimensional stretching matrix, and pressure signals from tongue-palatal contact events were acquired. S2: Preprocessing and Synchronization: Time axis calibration was performed using IMU peak events, followed by filtering and motion artifact removal. S3: Tongue Surface Deformation Reconstruction: Local tongue surface curvature changes were reconstructed using TPS based on electrode array node displacement. S4: Three-Dimensional Tongue Reconstruction: The deformed surface was mapped onto a tongue anatomical model, and the three-dimensional motion trajectory was obtained using constraint optimization. Results: The tongue tip extended 21mm forward, the tongue body lifting curvature increased by 32%, and the left-right offset symmetry was >95%, indicating a normal tongue movement pattern.

[0090] Example 2: Lesion localization in a patient suspected of hypoglossal nerve injury S1–S4 Same as Case 1; S5: MUAP decomposition: MUAP recruitment in the left genioglossus muscle region was reduced by 47%, and MUAP synchronicity was decreased; S6: Regional function mapping: The corresponding region of the left genioglossus muscle showed a lack of functional activity in TDM; S7: Lesion identification: TDM showed a lesion probability of 0.81 in the left genioglossus muscle region, with the characteristic combination of low MUAP activity + deviation of the motor path; Final inference: Damage to the left hypoglossal nerve or genioglossus muscle.

[0091] Example 3: Analysis of tongue morphology compensation in patients after tongue cancer surgery Reconstruction results showed that local tongue defects led to abnormal depressions on the tongue surface, restricted deformation of the transverse tongue muscle, increased EMG-motor coupling delay, and TDM showed structural defects and myogenic functional insufficiency. This method can be used for postoperative rehabilitation monitoring.

[0092] Third Embodiment like Figure 2 As shown, this embodiment provides a tongue movement morphology reconstruction and neuromuscular lesion identification system for performing the tongue movement morphology reconstruction and neuromuscular lesion identification method as described in the first embodiment, comprising: The multimodal signal acquisition module is used to acquire multimodal signals. It uses a flexible electrode array attached to the tongue surface to acquire multimodal raw signals and record system timestamps. The multimodal raw signals include surface electromyography (sEMG) signals, inertial measurement unit (IMU) signals, tongue surface geometric deformation signals, and tongue-palatal contact pressure event sequences. The preprocessing and feature extraction module is used to perform signal preprocessing and temporal feature extraction. It performs targeted filtering, artifact removal and feature extraction on the multimodal raw signal, and achieves cross-modal time axis alignment through multimodal time synchronization to obtain the electromyographic temporal coupling feature set of multimodal preprocessed signal set and tongue movement. The tongue surface deformation and posture estimation module is used to jointly estimate the tongue surface deformation and tongue body posture. Based on the processing results of step S2, the module reconstructs the local dynamic surface of the tongue surface using a preset model, and infers the overall posture sequence of the tongue body by combining IMU signals. The 3D tongue dynamic reconstruction module is used to perform 3D tongue dynamic fusion reconstruction. It introduces a tongue anatomical prior model containing multiple anatomical regions and constraints, and constructs a joint optimization objective function by combining tongue surface deformation data and pose information. The 3D dynamic morphological sequence of the tongue is obtained through iterative solution. The MUAP analysis TFM construction module is used to perform MUAP analysis and regional functional map construction. It performs motor unit action potential (MUAP) decomposition and feature extraction on surface electromyography (sEMG) signals, maps the extracted features and tongue movement-related data to the tongue anatomical model, and generates tongue regional functional maps (TFM). The lesion feature extraction and diagnosis module is used to extract lesion features and make diagnostic inferences. It constructs lesion feature vectors based on multi-dimensional abnormal features, and generates a tongue lesion risk map (TDM) through machine learning model analysis, outputting the potentially damaged tongue muscle areas, nerve branches, and lesion types.

[0093] A computer-readable storage medium stores computer code that, when executed, performs the methods described above. Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. This program can be stored in a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0094] The above description is merely a preferred embodiment of the present invention, and the scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principle of the present invention should also be considered within the scope of protection of the present invention.

[0095] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0096] It should be noted that the above embodiments can be freely combined as needed. The above description is only a preferred embodiment of the present invention. It should be pointed out that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method of tongue movement pattern reconstruction and neuromuscular pathology identification, characterized in that, The method comprises the following steps: S1: performing multi-modal signal acquisition, acquiring multi-modal original signals by using a flexible electrode array attached to the tongue surface, and recording system time stamps, wherein the multi-modal original signals include surface electromyography signals sEMG, inertial measurement unit signals IMU, tongue surface geometric deformation signals, and a tongue-palate contact pressure event sequence; S2: performing signal preprocessing and time sequence feature extraction, performing targeted filtering, artifact removal, and feature extraction processing on the multi-modal original signals, achieving cross-modal time axis alignment through multi-modal time synchronization, and obtaining a multi-modal preprocessed signal set and an electromyography time sequence coupling feature set of tongue movement; S3: performing tongue surface deformation and tongue posture joint estimation, based on the processing results of step S2, using a preset model to reconstruct the local dynamic surface of the tongue surface, and simultaneously combining the IMU signals to deduce the overall posture sequence of the tongue body; S4: performing three-dimensional tongue body dynamic fusion reconstruction, introducing a tongue body anatomical prior model containing multiple anatomical regions and constraint conditions, combining tongue surface deformation data and posture information to construct a joint optimization objective function, and obtaining a three-dimensional dynamic morphology sequence of the tongue body through iterative solving; S5: performing MUAP analysis and regional function map construction, decomposing and extracting features of the surface electromyography signals sEMG, mapping the extracted features and tongue movement related data to the tongue body anatomical model, and generating a tongue body regional function map TFM; S6: performing lesion feature extraction and diagnosis reasoning, constructing a lesion feature vector based on multi-dimensional abnormal features, generating a tongue lesion risk map TDM through machine learning model analysis, and outputting possible damaged tongue muscle regions, nerve branches, and lesion types.

2. The method of claim 1, wherein the tongue movement pattern reconstruction and neuromuscular disorder identification method is characterized by, In step S1, the inertial measurement unit signals IMU include three-axis acceleration and three-axis angular velocity; the tongue surface geometric deformation signals are obtained by an electrode node distance, a stretch matrix, or a stretch sensor array inside a flexible patch, an electrode distance measurement unit; the surface electromyography signals sEMG are a surface electromyography potential sequence of several channels; and the tongue-palate contact pressure event sequence is used to prompt a movement phase.

3. The method of claim 1, wherein the tongue movement pattern reconstruction and neuromuscular disorder identification method is characterized by, In step S2, the targeted processing specifically includes: performing band-pass filtering and artifact removal on the surface electromyography signals sEMG, performing low-pass filtering on the inertial measurement unit signals IMU, performing smoothing processing on the tongue surface geometric deformation signals, and performing threshold detection on the tongue-palate contact pressure event signals; The artifact removal adopts independent component analysis ICA or a time-frequency domain denoising method; The features of the time sequence feature extraction include electromyography onset time EMG-onset, movement onset time MOV-onset, peak time, movement duration, acceleration integral, tongue surface deformation center, curvature feature, and local maximum deformation point; The multi-modal time synchronization method includes using a specific action event as a synchronization anchor point, calculating an optimal alignment offset using a cross-correlation method, introducing system clock offset compensation, and further using a movement phase segmentation method using tongue-palate pressure events as start and end markers.

4. The method of claim 1, wherein the tongue movement pattern reconstruction and neuromuscular disorder identification method is characterized by, In step S3, the preset model is a thin plate spline (TPS) or a finite element model based on a triangular mesh or a tetrahedral mesh; and the tongue posture sequence is inferred by using acceleration and gyroscope signals of the IMU in combination with an extended Kalman filter or by using a deep inference network.

5. The method of claim 1, wherein the tongue movement pattern reconstruction and neuromuscular disorder identification method is characterized by, In step S4, the anatomical regions of the tongue anatomical prior model include longitudinal muscle, transverse muscle, vertical muscle, genioglossus muscle, hyoglossus muscle, and styloglossus muscle, and further include tongue fiber direction, muscle bundle orientation, and regional elasticity parameters; the constraint conditions include continuity constraint, non-penetrability constraint, motion range constraint of each anatomical region, and structure stress-deformation relationship constraint; the joint optimization objective function includes deformation field matching error, posture prediction error, anatomical structure deviation, time continuity constraint, and EMG activation intensity subterm of part of the region; and the spatial topological relationship of the electrode fixed point is combined in the iterative solving process.

6. The method of claim 1, wherein the tongue movement pattern reconstruction and neuromuscular disorder identification method is characterized by, In step S5, the MUAP decomposition uses sparse coding, blind source separation, convolution sparse coding, or a deep neural network to perform deconvolution processing on the sEMG, and the extracted features include motor unit recruitment pattern, discharge frequency, conduction velocity, unit synchrony, and neural drive strength estimation; the tongue movement related data includes local deformation amplitude and regional motion trajectory; and the tongue region function map (TFM) includes regional activation intensity, regional deformation amplitude, regional motion path, and regional stress-deformation coupling curve, and the muscle electrical activation intensity and the regional deformation amplitude are fused into a comprehensive function index according to the weight.

7. The method of claim 1, wherein the tongue movement pattern reconstruction and neuromuscular disorder identification method is characterized by, In step S6, the multi-dimensional abnormal features include MUAP abnormalities, regional motion abnormalities, deformation abnormalities, EMG-movement timing coupling abnormalities, and anatomical mapping abnormalities, wherein the MUAP abnormalities include neuromuscular abnormalities, specifically including low activation, reduced conduction velocity, and abnormal synchronization, the regional motion abnormalities include limited motion amplitude and deviation, the deformation abnormalities include contralateral asymmetry and local collapse, the EMG-movement timing coupling abnormalities include increased delay and lack of correspondence, and the anatomical mapping abnormalities include mismatch between activated regions and expected regions. The lesion feature vector includes EMG-movement coupling delay EMD, left-right symmetry index SI, and local deformation stiffness change; the machine learning model includes a graph convolution network and a Bayesian inference model; in the output result, the tongue muscle region includes genioglossus muscle and hyoglossus muscle, the nerve branch includes the side and branch of hypoglossal nerve or lingual nerve, and the lesion type includes neuromuscular, muscular, and postoperative structural lesions; and the method finally generates a tongue three-dimensional dynamic model, a tongue region function map (TFM), and a tongue lesion risk map (TDM) for clinical diagnosis report output. If the left / right side motion ability of the tongue is detected to decrease and the MUAP conduction velocity is reduced, it is diagnosed that the branch of the hypoglossal nerve on one side may be damaged; if the local deformation is significantly reduced but the MUAP activity is normal, it is diagnosed that the tongue has a risk of structural or fibrotic lesion; and if the MUAP activation is weakened and the deformation almost disappears, it is judged that the lesion is muscular or postoperative scar traction.

8. A tongue movement pattern reconstruction and neuromuscular disorder identification system for performing the tongue movement pattern reconstruction and neuromuscular disorder identification method according to any one of claims 1 to 7, characterized in that The method comprises the following steps: A multi-modal signal acquisition module is configured to acquire multi-modal signals, obtain multi-modal raw signals including surface electromyography (sEMG), inertial measurement unit (IMU) signals, tongue surface geometric deformation signals, and a tongue-palate contact pressure event sequence, and record system timestamps using a flexible electrode array attached to the tongue surface; A preprocessing and feature extraction module is configured to preprocess and extract time-series features, perform targeted filtering, de-noising, and feature extraction processing on the multi-modal raw signals, achieve cross-modal time axis alignment through multi-modal time synchronization, and obtain a multi-modal preprocessed signal set and an electromyography time-series coupling feature set of tongue movement; A tongue surface deformation posture estimation module is configured to jointly estimate tongue surface deformation and tongue posture, reconstruct a tongue surface local dynamic surface based on the processing results of step S2 using a preset model, and infer a tongue overall posture sequence in combination with IMU signals; A three-dimensional tongue dynamic reconstruction module is configured to perform three-dimensional tongue dynamic fusion reconstruction, introduce a tongue anatomical prior model containing multiple anatomical regions and constraint conditions, construct a joint optimization objective function in combination with tongue surface deformation data and posture information, and obtain a tongue three-dimensional dynamic morphology sequence through iterative solving; An MUAP analysis and TFM construction module is configured to perform MUAP analysis and regional functional map construction, decompose and extract features of a motor unit action potential (MUAP) from sEMG, map the extracted features and tongue movement related data to a tongue anatomical model, and generate a tongue regional functional map (TFM); A lesion feature extraction and diagnosis module is configured to perform lesion feature extraction and diagnosis reasoning, construct a lesion feature vector based on multi-dimensional abnormal features, analyze and generate a tongue lesion risk map (TDM) through a machine learning model, and output possible damaged tongue muscle regions, nerve branches, and lesion types.

9. A computer device, comprising: A computer readable storage medium stores computer code that, when executed, causes one or more processors to perform the method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer code that, when executed, causes the method of any one of claims 1 to 7 to be performed.

Images