Craniofacial dynamic reconstruction method and system based on multi-modal data fusion

Abstract

The invention relates to the technical field of medical image processing, and discloses a craniofacial dynamic reconstruction method and system based on multi-modal data fusion, and the method comprises the steps: arranging a multi-modal data collection device in a target craniofacial region, and obtaining a static CT image, a static MRI image, a dynamic expression video sequence and a surface electromyogram signal; preprocessing the static CT image, the static MRI image, the dynamic expression video sequence and the surface electromyogram signal; inputting the preprocessed data into a multi-scale finite element model, and simulating a coupling relationship between muscle contraction force and skin deformation by adopting a biomechanical driving strategy to generate a dynamic craniofacial model; and fusing the geometric error and the motion consistency score of the real data by adopting a linear regression method, and outputting a comprehensive reconstruction quality index. According to the method, the problems of low craniofacial dynamic modeling accuracy and poor robustness in the prior art can be solved.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing. Specifically, it relates to a craniofacial dynamic reconstruction method and system based on multi-modal data fusion. Background Art

[0002] The dynamic reconstruction of the craniofacial region has important application values in fields such as facial expression analysis, personalized surgical planning, medical image research, and virtual human modeling. Existing methods mainly rely on single-modal data, such as static CT images or MRI images, which can only provide anatomical information of craniofacial tissues and lack accurate modeling of facial movements. In addition, although video-based facial reconstruction methods can obtain facial dynamic deformation information, it is difficult to deeply analyze biomechanical mechanisms such as muscle contraction and bone support, resulting in models lacking physical authenticity and individual adaptability.

[0003] In recent years, multi-modal craniofacial modeling methods have faced many challenges, including: registration and fusion of multi-modal images: due to the different characteristics of CT and MRI images, how to accurately align and integrate bone and soft tissue information is still a difficult point. Representation of facial dynamic movements: how to efficiently extract the movement trajectories of facial key points and combine deep learning methods to enhance the expression ability of movement features is the key to improving the modeling accuracy. Modeling of muscle activation features: there is noise interference in electromyography signals. How to accurately extract muscle activation features and reasonably map them into a finite element model is the core problem for realizing biomechanical driving. Physical simulation and computational efficiency: traditional finite element methods have high computational costs. How to improve computational efficiency while ensuring simulation accuracy is a difficult point in dynamic reconstruction.

[0004] In view of the above problems, the present invention proposes a craniofacial dynamic reconstruction method based on multi-modal data fusion, which comprehensively utilizes static CT images, static MRI images, dynamic expression video sequences, and surface electromyography signals, and combines a biomechanical driving strategy to construct a high-precision craniofacial dynamic model to improve the accuracy and robustness of modeling.

[0005] Therefore, there is an urgent need for a method that uses static CT images, static MRI images, dynamic expression video sequences, and surface electromyography signals, combines a biomechanical driving strategy, and constructs a high-precision craniofacial dynamic model to improve the accuracy and robustness of modeling. Summary of the Invention

[0006] In view of this, the present invention proposes a craniofacial dynamic reconstruction method and system based on multi-modal data fusion, aiming to solve the problems of low accuracy and poor robustness in current craniofacial dynamic modeling.

[0007] A craniofacial dynamic reconstruction method based on multi-modal data fusion proposed by the present invention includes:

[0008] Deploy a multi-modal data acquisition device in the target craniofacial region to obtain static CT images, static MRI images, dynamic expression video sequences, and surface electromyography signals;

[0009] Perform rigid registration and non-rigid registration on the static CT images and static MRI images to generate a fused static three-dimensional anatomical model;

[0010] Perform optical flow analysis and key frame extraction on the dynamic expression video sequences to obtain facial motion trajectory data;

[0011] Perform filtering and time-frequency analysis on the surface electromyography signals to extract muscle activation features;

[0012] Input the fused static three-dimensional anatomical model, facial motion trajectory data, and muscle activation features into a multi-scale finite element model and adopt a biomechanical driving strategy to simulate the coupling relationship between muscle contraction force and skin deformation to generate a dynamic craniofacial model;

[0013] Adopt the linear regression method to fuse the geometric error and motion consistency score of the real data and output the comprehensive reconstruction quality index.

[0014] Further, the multi-modal data acquisition device includes a CT scanner, an MRI device, a high-speed camera, and a wireless surface electromyography sensor array.

[0015] Further, the specific content of performing rigid registration and non-rigid registration on the static CT images and static MRI images to generate a fused static three-dimensional anatomical model is: adopt a rigid registration algorithm based on mutual information for the static CT images and static MRI images to align the bone structures and obtain bone details;

[0016] Adopt a non-rigid registration algorithm based on B-spline for the static CT images and static MRI images to align the soft tissue contours and obtain soft tissue information;

[0017] Integrate the bone details and soft tissue information through a weighted fusion strategy to generate a high-precision static three-dimensional model.

[0018] Further, the specific content of performing optical flow analysis and key frame extraction on the dynamic expression video sequences to obtain facial motion trajectory data is: adopt an improved Lucas-Kanade optical flow method for the dynamic expression video sequences to extract the displacements of facial key points and generate the original motion trajectory data, then adopt a time series clustering algorithm to screen the key frames in the original motion trajectory data, construct a sparse representation of the facial motion trajectory, and finally extract the expression texture features in the dynamic expression video sequences through a convolutional neural network and fuse them with the sparse representation of the facial motion trajectory to obtain the facial motion trajectory data.

[0019] Further, finally, the expression texture features in the dynamic expression video sequence are extracted by a convolutional neural network and fused with the sparse representation of the facial motion trajectory. The specific content of the facial motion trajectory data is as follows: The optical flow timestamp is used to perform spatio-temporal alignment between the sparse representation of the facial motion trajectory and the expression texture features. Subsequently, the weights are dynamically adjusted by the cross-attention module in the convolutional neural network to enhance the modeling strength. Finally, the sparse representation of the facial motion trajectory and the expression texture features are concatenated into a joint feature vector to obtain the facial motion trajectory data.

[0020] Further, the specific content of filtering and time-frequency analysis of the surface electromyogram signal to extract muscle activation features is as follows: First, wavelet packet transform is used to remove power frequency interference and motion artifacts in the surface electromyogram signal; then, independent component analysis is used to separate the target muscle group signal in the surface electromyogram signal; finally, time-domain features and frequency-domain features are extracted to obtain muscle activation features.

[0021] Further, the specific content of inputting the fused static three-dimensional anatomical model, facial motion trajectory data, and muscle activation features into a multi-scale finite element model and using a biomechanical driving strategy to simulate the coupling relationship between muscle contraction force and skin deformation to generate a dynamic craniofacial model is as follows: The static three-dimensional anatomical model is discretized into nonlinear hyperelastic soft tissue elements to construct a stiffness matrix K. The muscle activation features are converted into muscle contraction forces through a muscle force model. The facial motion trajectory data is used as a boundary condition and embedded in the dynamic equation by the Lagrange multiplier method. The Newmark-β method is used to solve the dynamic equation. The node displacement u is obtained by balancing the calculation efficiency and stability. The obtained node displacement u is mapped to the skin surface, and a dynamic craniofacial model is generated by radial basis function interpolation.

[0022] Further, the specific content of using the linear regression method to fuse the geometric error and motion consistency score of the real data and output the comprehensive reconstruction quality index is as follows: The Hausdorff distance between the vertices of the static three-dimensional anatomical model and the real anatomical landmark points is calculated as the geometric error score. The dynamic time warping algorithm is used to quantify the consistency score of the facial motion trajectory data. The linear regression method is used to fuse the geometric error score and the motion consistency score to generate the reconstruction quality index.

[0023] Further, the linear regression method is used to fuse the geometric error score and the motion consistency score to generate the reconstruction quality index, which is expressed as:

[0024] Q = α·S geo + β·S motion ;

[0025] where Q is the reconstruction quality index, α is the first weight coefficient, β is the second weight coefficient, S geo is the geometric error score, Smotion For motion consistency scoring.

[0026] On the other hand, the present invention also proposes a craniofacial dynamic reconstruction system based on multi-modal data fusion, and the system includes:

[0027] A data acquisition module, configured to acquire static CT images, static MRI images, dynamic expression video sequences, and surface electromyography signals;

[0028] A data processing module, configured to perform rigid registration and non-rigid registration on the static CT images and static MRI images to generate a fused static three-dimensional anatomical model; perform optical flow analysis and key frame extraction on the dynamic expression video sequences to obtain facial motion trajectory data; perform filtering and time-frequency analysis on the surface electromyography signals to extract muscle activation features;

[0029] A dynamic modeling module, configured to couple the simulated muscle contraction force and skin deformation relationship of the fused static three-dimensional anatomical model, facial motion trajectory data, and muscle activation features to generate a dynamic craniofacial model;

[0030] An evaluation and output module, configured to fuse the geometric error and motion consistency score of the real data by using the linear regression method and output a comprehensive reconstruction quality index.

[0031] Compared with the prior art, the beneficial effects of the present invention are as follows: The craniofacial dynamic reconstruction method and system based on multi-modal data fusion of the present invention adopt rigid registration and non-rigid registration of static CT images and static MRI images, align the bone structures based on the mutual information algorithm, and align the soft tissues based on the B-spline algorithm, which can retain both bone details and soft tissue information and improve the accuracy of the static three-dimensional anatomical model; The present invention uses an improved Lucas-Kanade optical flow method to extract the displacements of facial key points, combines a time series clustering algorithm to extract key frames, and further uses a convolutional neural network to fuse expression texture features to accurately represent the facial movement trajectory; The present invention removes power frequency interference and motion artifacts in the electromyogram signal through wavelet packet transform, uses independent component analysis (ICA) to separate the signals of target muscle groups, and extracts time-domain and frequency-domain features to accurately obtain muscle activation features, providing high-quality input data for biomechanical driving; The present invention discretizes facial tissues using a non-linear hyperelastic finite element model to construct a stiffness matrix K, converts muscle activation features into muscle contraction forces through a muscle force model, combines facial movement trajectory data as boundary conditions, embeds them into the dynamic equation using the Lagrange multiplier method, and uses the Newmark-β method to solve, balancing computational efficiency and stability to generate a more physiologically characteristic dynamic craniofacial model; The present invention calculates the geometric error score between the vertices of the static three-dimensional anatomical model and the real anatomical landmark points through the Hausdorff distance, quantifies the consistency score of the facial movement trajectory using the dynamic time warping algorithm, and fuses these two indicators using linear regression to generate a comprehensive reconstruction quality index, thereby providing an objective model evaluation criterion; The present invention can be used in multiple application scenarios such as medical surgical planning, facial expression disorder assessment, personalized prosthesis design, virtual human construction, etc., providing an accurate craniofacial dynamic modeling solution for the fields of medicine and computer graphics. Description of the Drawings

[0032] By reading the following detailed description of the preferred embodiments, various other advantages and benefits will become clear to those of ordinary skill in the art. The drawings are only for the purpose of showing the preferred embodiments and are not considered to be a limitation of the present invention. Moreover, throughout the drawings, the same reference numerals are used to represent the same components. In the drawings:

[0033] Figure 1 It is a flowchart of a craniofacial dynamic reconstruction method based on multi-modal data fusion according to an embodiment of the present invention;

[0034] Figure 2 It is a structural block diagram of a craniofacial dynamic reconstruction system based on multi-modal data fusion according to an embodiment of the present invention. Detailed Embodiments

[0035] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided so that the present disclosure can be more thoroughly understood and the scope of the present disclosure can be fully conveyed to those skilled in the art. It should be noted that, without conflict, the embodiments in the present invention and the features in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the drawings and in combination with the embodiments.

[0036] Referring to Figure 1 as shown, an embodiment of the present invention provides a craniofacial dynamic reconstruction method based on multi-modal data fusion, including:

[0037] S1: Deploy a multi-modal data acquisition device in the target craniofacial area to obtain static CT images, static MRI images, dynamic expression video sequences, and surface electromyography signals;

[0038] S2: Perform rigid registration and non-rigid registration on the static CT images and static MRI images to generate a fused static three-dimensional anatomical model;

[0039] S3: Perform optical flow analysis and key frame extraction on the dynamic expression video sequence to obtain facial movement trajectory data;

[0040] S4: Filter and perform time-frequency analysis on the surface electromyography signals to extract muscle activation features;

[0041] S5: Input the fused static three-dimensional anatomical model, facial movement trajectory data, and muscle activation features into a multi-scale finite element model and adopt a biomechanical driving strategy to simulate the coupling relationship between muscle contraction force and skin deformation to generate a dynamic craniofacial model;

[0042] S6: Use the linear regression method to fuse the geometric error and motion consistency score of the real data and output a comprehensive reconstruction quality index.

[0043] Furthermore, the multi-modal data acquisition device includes a CT scanner, an MRI device, a high-speed camera, and a wireless surface electromyography sensor array.

[0044] Furthermore, the specific content of performing rigid registration and non-rigid registration on the static CT images and static MRI images to generate a fused static three-dimensional anatomical model is as follows: Use a rigid registration algorithm based on mutual information to align the bone structures of the static CT images and static MRI images to obtain bone details;

[0045] Use a non-rigid registration algorithm based on B-spline to align the soft tissue contours of the static CT images and static MRI images to obtain soft tissue information;

[0046] Integrate bone details and soft tissue information through a weighted fusion strategy to generate a high-precision static three-dimensional model.

[0047] Specifically, first, preprocess the static CT images and static MRI images, including gray normalization, histogram matching, and noise suppression, to reduce the contrast differences between different imaging modalities and improve the robustness and accuracy of subsequent registration.

[0048] Next, adopt a rigid registration algorithm based on mutual information (MI) to align the bone structures in the CT images and MRI images. This method is based on the principle of information theory and realizes image alignment by maximizing the mutual information between the CT and MRI images, thus overcoming the limitations of traditional gray-difference-based registration methods on multi-modal images. During the rigid registration process, use the gradient descent optimization strategy to iteratively adjust the rotation and translation parameters to ensure the alignment of the bone details in the CT images and MRI images, thereby improving the registration accuracy and obtaining clear bone structure information.

[0049] Then, for soft tissue information, adopt a non-rigid registration algorithm based on B-spline. Since CT images mainly highlight bone information and MRI images have higher resolution for soft tissues, on the basis of bone structure alignment, non-rigid registration is required to accurately match the soft tissue morphology. The B-spline deformation model performs free deformation on the images by constructing a low-dimensional control point grid to align the soft tissue contours and compensate for the differences in individual facial anatomical features. This method uses a multi-scale optimization strategy, first performing large-scale global alignment and then gradually optimizing local details to improve the fineness of soft tissue registration and ensure the integrity of soft tissue contour information.

[0050] Finally, adopt a weighted fusion strategy to integrate bone details and soft tissue information to generate a high-precision static three-dimensional anatomical model. Mainly use a weighted method based on the Gaussian kernel function, giving higher weights to the CT images in the bone region and higher weights to the MRI images in the soft tissue region to ensure that the fused model maintains clear geometric details in the bone structure and has a more realistic anatomical morphology in the soft tissue region.

[0051] It should be noted that rigid registration ensures the consistency of the bone structure, non-rigid registration makes up for individual differences in soft tissues, and weighted fusion optimizes information integration, making the overall registration result more accurate; combining the high-resolution bone information of CT and the soft tissue details of MRI enables the fused model to accurately represent the structure of hard tissues (such as the skull) and clearly present the morphology of soft tissues (such as muscles and fat).

[0052] Furthermore, the specific content of performing optical flow analysis and key frame extraction on the dynamic expression video sequence to obtain the facial motion trajectory data is as follows: For the dynamic expression video sequence, the improved Lucas-Kanade optical flow method is used to extract the displacements of facial key points to generate the original motion trajectory data. Then, the time series clustering algorithm is used to screen the key frames in the original motion trajectory data to construct the sparse representation of the facial motion trajectory. Finally, the convolutional neural network is used to extract the expression texture features in the dynamic expression video sequence and fuse them with the sparse representation of the facial motion trajectory to obtain the facial motion trajectory data.

[0053] Specifically, first, preprocess the dynamic expression video sequence, including denoising, gray-scale normalization, and face region alignment, to reduce the influence of illumination changes and camera angles on optical flow calculation and improve the robustness of key point detection.

[0054] Next, use the improved Lucas-Kanade optical flow method to extract the displacement information of facial key points and generate the original motion trajectory data. The Lucas-Kanade optical flow method is based on the gray-scale consistency assumption of the local window and can effectively track the motion trajectories of facial feature points. However, traditional methods are prone to failure in the case of large displacements or fast movements. Therefore, a multi-scale pyramid structure is introduced to enhance the stability of optical flow estimation. At the same time, regularization constraints and edge-preserving strategies are combined to avoid the drift problem of facial key points caused by occlusion or noise and improve the continuity and accuracy of the motion trajectory.

[0055] Then, perform time series analysis on the original motion trajectory data and use the key frame screening method based on time series clustering. Since there are a large number of redundant frames in the process of facial expression changes, directly processing the complete motion trajectory will increase the computational cost and may introduce noise. Therefore, first calculate the motion amplitude and acceleration changes of key points in each frame, cluster similar trajectories based on the dynamic time warping (DTW) method, and use the K-means clustering algorithm to select representative key frames, thereby constructing the sparse representation of the facial motion trajectory. This process can retain the main features of expression changes, reduce the data dimension at the same time, and improve the computational efficiency of subsequent analysis.

[0056] Subsequently, expression texture features in the dynamic expression video sequence are extracted through a Convolutional Neural Network (CNN). Facial expressions not only involve geometric deformations (such as key point displacements), but also contain complex texture information (such as wrinkles, shadow changes, etc.). Therefore, deep convolutional networks such as ResNet or EfficientNet are used to perform multi-scale feature extraction on video frames to capture expression details at different levels. In addition, combined with the Self-Attention mechanism, the weights of key regions (such as eyebrows, corners of the mouth, and around the eyes) are enhanced during the feature fusion process, thereby improving the expression ability of expression texture information.

[0057] Finally, the sparse representation of the facial motion trajectory is fused with the expression texture features to obtain the final facial motion trajectory data. Specifically, the Cross-Attention mechanism is used to dynamically weight the data of the two modalities to ensure the synergistic effect of geometric motion features and expression texture features. Finally, using the Feature Concatenation strategy, the motion trajectory and texture information are combined into a unified feature vector, providing high-quality input data for subsequent craniofacial dynamic modeling.

[0058] Furthermore, the specific content of obtaining the facial motion trajectory data by finally extracting the expression texture features in the dynamic expression video sequence through a convolutional neural network and fusing them with the sparse representation of the facial motion trajectory is as follows: The optical flow timestamp is used to align the sparse representation of the facial motion trajectory and the expression texture features in space-time. Subsequently, the weights are dynamically adjusted through the cross-attention module in the convolutional neural network to enhance the modeling strength. Finally, the sparse representation of the facial motion trajectory and the expression texture features are concatenated into a joint feature vector to obtain the facial motion trajectory data.

[0059] Furthermore, the specific content of filtering and time-frequency analysis of the surface electromyogram signal to extract muscle activation features is as follows: First, wavelet packet transform is used to remove power frequency interference and motion artifacts in the surface electromyogram signal; then, Independent Component Analysis (ICA) is used to separate the target muscle group signals in the surface electromyogram signal; finally, time-domain features and frequency-domain features are extracted to obtain muscle activation features.

[0060] Further, the specific content of inputting the fused static three-dimensional anatomical model, facial movement trajectory data, and muscle activation characteristics into a multi-scale finite element model and simulating the coupling relationship between muscle contraction force and skin deformation using a biomechanical driving strategy to generate a dynamic craniofacial model is as follows: discretize the static three-dimensional anatomical model into non-linear hyperelastic soft tissue elements, construct the stiffness matrix K, convert the muscle activation characteristics into muscle contraction forces through a muscle force model, use the facial movement trajectory data as boundary conditions, embed them into the dynamic equation using the Lagrange multiplier method, solve the dynamic equation using the Newmark-β method, balance the computational efficiency and stability to obtain the nodal displacement u, map the obtained nodal displacement u to the skin surface, and generate a dynamic craniofacial model through radial basis function interpolation.

[0061] Specifically, first, perform finite element discretization on the static three-dimensional anatomical model. Adopt an adaptive mesh refinement strategy to divide the craniofacial region into finite element meshes of different scales. Larger elements are used in the bone region to improve computational efficiency, while smaller tetrahedral or hexahedral elements are used in the soft tissue region to ensure fine simulation of complex geometric structures. For the soft tissue part, introduce a non-linear hyperelastic constitutive model (such as the Mooney-Rivlin or Neo-Hookean model) to describe the stress-strain relationship of the skin and muscle tissues during deformation, and construct the stiffness matrix K of the overall system.

[0062] Then, construct a biomechanical muscle force model to convert the muscle activation characteristics extracted from surface electromyogram signals into the contraction forces of the corresponding muscles. Specifically, based on the Hill muscle model or the finite element-based active muscle force model, calculate the muscle contraction force Fm and apply it to the corresponding finite element elements. This model can consider the non-linear stress-strain characteristics of muscles, as well as the dynamic relationship between muscle length, activation level, and force, thereby improving the realism of muscle driving. In addition, to ensure the transmission effect of muscle contraction on skin deformation, a fiber-guided interpolation method is used to construct the muscle stress field, enabling the muscle force to affect the surrounding soft tissues in an anisotropic manner.

[0063] Next, introduce the facial movement trajectory data as boundary conditions into the finite element dynamic equation, and combine the bone constraint conditions to ensure that the facial muscle movement conforms to the real physical constraint relationship. Use the Lagrange multiplier method to enforce the boundary constraint conditions on the dynamic equation, avoiding the numerical instability that may be introduced by the traditional penalty function method, and ensuring the coordination between the facial movement trajectory and the muscle force.

[0064] Subsequently, the Newmark-β method is used to solve the dynamic equation, balancing the computational efficiency and stability to obtain the displacements of each finite element node. The Newmark-β method is an implicit integration method that can effectively handle large deformation problems and suppress high-frequency numerical oscillations, making the facial deformation process smoother and more physiologically realistic. During the solution process, to further improve the computational stability, a nonlinear iterative method (such as Newton-Raphson iteration) is adopted to optimize the displacement convergence effect, and a multi-scale solution strategy is combined. Pre-computation is carried out on a coarse grid to accelerate convergence, and local accuracy is improved on a fine grid.

[0065] Finally, the obtained nodal displacements u are mapped to the skin surface, and a dynamic craniofacial model is generated by interpolation using radial basis functions (RBFs). Radial basis function interpolation can gently propagate the displacement information of the finite element mesh while maintaining the geometric continuity of the face, thus generating a craniofacial model with high-precision dynamic deformation effects. Further, texture mapping technology can be combined to finely model the texture changes such as skin wrinkles and stretching during facial expression changes, making the final dynamic craniofacial model more realistic in both visual and physical performances.

[0066] It should be noted that based on the nonlinear hyperelastic soft tissue model, the true physical properties of facial tissues can be accurately described, improving the simulation effect of the influence of muscle contraction on skin deformation. By adopting a muscle force model based on electromyogram signals, the muscle contraction process can truly reflect facial expression movements of different intensities, improving the physiological rationality of the dynamic craniofacial model. Using the Newmark-β method combined with Newton-Raphson iteration ensures the numerical stability of the solution process of the facial dynamic equation, and the multi-scale solution method is used to accelerate the calculation and improve the simulation efficiency.

[0067] Furthermore, the linear regression method is used to fuse the geometric error and motion consistency scores of real data, and the specific content of the output comprehensive reconstruction quality index is as follows: The Hausdorff distance between the vertices of the static three-dimensional anatomical model and the real anatomical landmark points is calculated as the geometric error score, and the dynamic time warping algorithm is used to quantify the consistency score of the facial motion trajectory data. The linear regression method is used to fuse the geometric error score and the motion consistency score to generate the reconstruction quality index.

[0068] Furthermore, the linear regression method is used to fuse the geometric error score and the motion consistency score to generate the reconstruction quality index, expressed as:

[0069] Q = α·S geo + β·S motion ;

[0070] where Q is the reconstruction quality index, α is the first weight coefficient, β is the second weight coefficient, and Sgeo is the geometric error score, S motion is the motion consistency score.

[0071] Refer to Figure 2 As shown, an embodiment of the present invention further provides a craniofacial dynamic reconstruction system based on multi-modal data fusion. The system includes:

[0072] A data acquisition module configured to acquire static CT images, static MRI images, dynamic expression video sequences, and surface electromyography signals;

[0073] A data processing module configured to perform rigid registration and non-rigid registration on the static CT images and static MRI images to generate a fused static three-dimensional anatomical model; perform optical flow analysis and key frame extraction on the dynamic expression video sequences to obtain facial motion trajectory data; perform filtering and time-frequency analysis on the surface electromyography signals to extract muscle activation features;

[0074] A dynamic modeling module configured to couple the simulated muscle contraction force and skin deformation relationship of the fused static three-dimensional anatomical model, facial motion trajectory data, and muscle activation features to generate a dynamic craniofacial model;

[0075] An evaluation output module configured to fuse the geometric error and motion consistency scores of the real data using the linear regression method and output a comprehensive reconstruction quality index.

[0076] Therefore, the craniofacial dynamic reconstruction method and system based on multi-modal data fusion in the present invention adopt rigid registration and non-rigid registration of static CT images and static MRI images, align the bone structures based on the mutual information algorithm, and align the soft tissues based on the B-spline algorithm, which can retain both bone details and soft tissue information simultaneously and improve the accuracy of the static three-dimensional anatomical model; the present invention uses an improved Lucas-Kanade optical flow method to extract the displacements of facial key points, combines a time series clustering algorithm to extract key frames, and further uses a convolutional neural network to fuse expression texture features to achieve accurate characterization of facial movement trajectories; the present invention removes power frequency interference and motion artifacts in the electromyogram signals through wavelet packet transform, uses independent component analysis (ICA) to separate the signals of target muscle groups, and extracts time-domain and frequency-domain features to accurately obtain muscle activation features, providing high-quality input data for biomechanical driving; the present invention uses a non-linear hyperelastic finite element model to discretize facial tissues, constructs a stiffness matrix K, converts the muscle activation features into muscle contraction forces through a muscle force model, combines the facial movement trajectory data as boundary conditions, embeds them into the dynamic equation using the Lagrange multiplier method, and uses the Newmark-β method to solve, balancing computational efficiency and stability to generate a more physiologically characteristic dynamic craniofacial model; the present invention calculates the geometric error score between the vertices of the static three-dimensional anatomical model and the real anatomical landmark points through the Hausdorff distance, quantifies the consistency score of the facial movement trajectories using the dynamic time warping algorithm, and uses linear regression to fuse these two indicators to generate a comprehensive reconstruction quality index, thereby providing an objective model evaluation criterion; the present invention can be used in multiple application scenarios such as medical surgical planning, facial expression disorder assessment, personalized prosthesis design, and virtual human construction, providing an accurate craniofacial dynamic modeling solution for the fields of medicine and computer graphics.

[0077] The present invention can be used in multiple application scenarios such as medical surgical planning, facial expression disorder assessment, personalized prosthesis design, and virtual human construction. For example, by capturing the facial muscle movements of users (such as subtle changes in eyebrows and corners of the mouth) through a camera or sensor, combining deep learning algorithms to recognize emotional states (such as anger, sadness, surprise), and at the same time cooperating with a generative adversarial network (GAN) or 3D modeling technology to generate micro-expressions (such as raising eyebrows and pursing lips during a conversation) that match the semantic content for virtual characters.

[0078] The present invention can be applied to virtual humans, such as virtual assistants, virtual idols, and corporate customer service images. These virtual humans can recognize the emotions of users through micro-expressions and dynamically adjust the conversation strategy (such as shortening the answer when it is found that the user is impatient).

[0079] It can also be applied to mental health and education, such as being applied to a psychological counseling robot to recognize the hidden emotions of users (such as unnatural twitching of the corners of the mouth when forcing a smile) through micro-expressions and generate empathic conversations.

[0080] When applied to language learning assistants, when correcting pronunciation, the virtual teacher shows encouraging / affirmative micro-expressions to enhance learning motivation;

[0081] It can also be applied to social interaction and entertainment. For example, users can upload selfies to generate "animated virtual avatars with micro-expressions", send voice messages with micro-expressions on social platforms, or game NPCs trigger different plot branches according to the player's emotions (such as a fear expression triggering a soothing plot).

[0082] The present invention can also be applied to dynamic micro-expression monitoring technology. By capturing and analyzing human's short-term and unconscious micro-expressions (usually lasting from 0.04 seconds to 0.5 seconds), it can reveal hidden emotions and true mental states. With the progress of artificial intelligence, computer vision, and neuroscience. The following are the possible fields of in-depth application and specific scenario analysis:

[0083] 1. Mental health and clinical diagnosis and treatment

[0084] Early screening for depression / anxiety disorders: Quantify emotional fluctuations through micro-expressions (such as a brief downward droop of the corners of the mouth and a slight tightening of the eyebrows) during a patient's conversation, and combine with an AI model to predict the risk of mental illness and assist doctors in formulating intervention plans.

[0085] Autism intervention: Real-time monitor the micro-reactions of autistic children to social stimuli (such as slight muscle tremors in the face when avoiding eye contact), and adjust the content of rehabilitation training.

[0086] Post-traumatic stress disorder (PTSD) assessment: Monitor the fear micro-expressions (such as flaring nostrils and rapid eye movement) of patients in exposure therapy to trigger factors, and dynamically adjust the treatment intensity.

[0087] 2. Medical diagnosis assistance

[0088] Early diagnosis of amyotrophic lateral sclerosis (ALS): Assist in screening for neurodegenerative diseases through minor movement disorders of facial muscles (such as a delay in one side of the mouth when attempting to smile).

[0089] Pain management: Quantify the hidden pain expressions of postoperative patients (such as a brief raising of the eyebrows) to optimize the dosage of analgesics.

[0090] Drug side effect monitoring: Capture the disgust micro-expressions (such as a brief nose wrinkle) of subjects after taking medicine in clinical trials to quickly discover potential problems.

[0091] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the above embodiments, those of ordinary skill in the art should understand that: it is still possible to modify the specific implementation manners of the present invention or make equivalent replacements, and any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the protection scope of the claims of the present invention.

Claims

1. A craniofacial dynamic reconstruction method based on multimodal data fusion, characterized in that: include: Multimodal data acquisition devices are deployed in the target craniofacial area to obtain static CT images, static MRI images, dynamic expression video sequences, and surface electromyographic signals; Perform rigid registration and non-rigid registration on static CT images and static MRI images to generate a fused static 3D anatomical model; Perform optical flow analysis and key frame extraction on dynamic expression video sequences to obtain facial motion trajectory data; Filter and perform time-frequency analysis on surface electromyographic signals to extract muscle activation features; The fused static 3D anatomical model, facial motion trajectory data and muscle activation features are input into the multi-scale finite element model and a biomechanical driving strategy is used to simulate the coupling relationship between muscle contraction force and skin deformation to generate a dynamic craniofacial model. The linear regression method is used to fuse the geometric error and motion consistency score of the real data to output a comprehensive reconstruction quality index.

2. The method for dynamic craniofacial reconstruction based on multimodal data fusion according to claim 1, characterized in that: The multimodal data acquisition device includes a CT scanner, an MRI device, a high-speed camera and a wireless surface electromyography sensor array.

3. The method for dynamic craniofacial reconstruction based on multimodal data fusion according to claim 2, characterized in that: The specific contents of performing rigid and non-rigid registration on the static CT image and the static MRI image to generate a fused static three-dimensional anatomical model are as follows: aligning the bone structure of the static CT image and the static MRI image using a rigid registration algorithm based on mutual information to obtain bone details; The soft tissue contours of static CT images and static MRI images were aligned using a non-rigid registration algorithm based on B-spline to obtain soft tissue information. The bone details and soft tissue information are integrated through a weighted fusion strategy to generate a high-precision static three-dimensional model.

4. The method for dynamic craniofacial reconstruction based on multimodal data fusion according to claim 3, characterized in that: The specific content of performing optical flow analysis and key frame extraction on the dynamic expression video sequence to obtain facial motion trajectory data is as follows: using the improved Lucas-Kanade optical flow method to extract the displacement of facial key points in the dynamic expression video sequence to generate original motion trajectory data, then using a time series clustering algorithm to screen the key frames in the original motion trajectory data to construct a sparse representation of the facial motion trajectory, and finally extracting the expression texture features in the dynamic expression video sequence through a convolutional neural network, and fusing them with the sparse representation of the facial motion trajectory to obtain the facial motion trajectory data.

5. The method for dynamic craniofacial reconstruction based on multimodal data fusion according to claim 4, characterized in that: Finally, the expression texture features in the dynamic expression video sequence are extracted through the convolutional neural network and fused with the sparse representation of the facial motion trajectory. The specific content of the facial motion trajectory data is obtained as follows: Optical flow timestamps are used to align the sparse representation of facial motion trajectory with the expression texture features in time and space. Then, the weights are dynamically adjusted through the cross-attention module in the convolutional neural network to enhance the modeling strength. Finally, the sparse representation of facial motion trajectory and the expression texture features are spliced ​​into a joint feature vector to obtain the facial motion trajectory data.

6. The method for dynamic craniofacial reconstruction based on multimodal data fusion according to claim 5, characterized in that: The specific contents of filtering and time-frequency analysis of the surface electromyographic signal to extract the muscle activation feature are as follows: firstly, wavelet packet transform is used to remove the power frequency interference and motion artifacts in the surface electromyographic signal; then, independent component analysis is used to separate the target muscle group signal in the surface electromyographic signal; finally, time domain features and frequency domain features are extracted to obtain the muscle activation feature.

7. The method for dynamic craniofacial reconstruction based on multimodal data fusion according to claim 6, characterized in that: The method comprises inputting the fused static three-dimensional anatomical model, facial motion trajectory data and muscle activation features into a multi-scale finite element model and adopting a biomechanical driving strategy to simulate the coupling relationship between muscle contraction force and skin deformation to generate a dynamic craniofacial model. The specific contents are as follows: discretizing the static three-dimensional anatomical model into nonlinear hyperelastic soft tissue units, constructing a stiffness matrix K, converting muscle activation features into muscle contraction force through a muscle force model, using facial motion trajectory data as boundary conditions, embedding the Lagrange multiplier method into the dynamic equation, solving the dynamic equation using the Newmark-β method, balancing computational efficiency and stability to obtain the node displacement u, mapping the solved node displacement u to the skin surface, and generating a dynamic craniofacial model through radial basis function interpolation.

8. The method for dynamic craniofacial reconstruction based on multimodal data fusion according to claim 7, characterized in that: The specific content of using the linear regression method to fuse the geometric error and motion consistency score of the real data and outputting the comprehensive reconstruction quality index is as follows: calculating the Hausdorff distance between the vertices of the static three-dimensional anatomical model and the real anatomical landmarks as the geometric error score, quantifying the consistency score of the facial motion trajectory data through the dynamic time warping algorithm, and using the linear regression method to fuse the geometric error score and the motion consistency score to generate the reconstruction quality index.

9. The method for dynamic craniofacial reconstruction based on multimodal data fusion according to claim 8, characterized in that: The linear regression method is used to fuse the geometric error score and the motion consistency score to generate a reconstruction quality index, which is expressed as: Q=α·S geo +β·S motion ; Where Q is the reconstruction quality index, α is the first weight coefficient, β is the second weight coefficient, S geo S is the geometric error score, motion Score movement consistency.

10. A craniofacial dynamic reconstruction system based on multimodal data fusion, implemented in a craniofacial dynamic reconstruction method based on multimodal data fusion according to any one of claims 1 to 9, characterized in that: The system comprises: A data acquisition module, configured to acquire static CT images, static MRI images, dynamic expression video sequences, and surface electromyography signals; The data processing module is configured to perform rigid and non-rigid registration on static CT images and static MRI images to generate a fused static 3D anatomical model; perform optical flow analysis and key frame extraction on dynamic expression video sequences to obtain facial motion trajectory data; perform filtering and time-frequency analysis on surface electromyography signals to extract muscle activation features; A dynamic modeling module is configured to simulate the coupling relationship between muscle contraction force and skin deformation by fusion of static three-dimensional anatomical model, facial motion trajectory data and muscle activation features to generate a dynamic craniofacial model; The evaluation output module is configured to fuse the geometric error and motion consistency scores of the real data using a linear regression method and output a comprehensive reconstruction quality index.

Images