Facelift surgery planning method and system based on multi-modal dynamic expression analysis

Abstract

The application discloses a face plastic surgery planning method and system based on multi-modal dynamic expression analysis, relates to the field of face plastic surgery, and solves the problem of multi-expression incoordination caused by the traditional plastic surgery method using single-expression scanning data for surgery. The technical scheme is as follows: a multi-modal dynamic expression dataset based on a standardized expression sequence of multi-expression is used, and the multi-modal dynamic expression dataset is subjected to space-time alignment, the coordinated motion relationship of bones-muscles-soft tissues in the multi-expression process is comprehensively captured, a basic three-dimensional anatomical model and a biomechanical coordination model of bone-muscle-soft tissue linkage are constructed, preoperative surgery optimization is performed according to the simulation result of the biomechanical coordination model, and double feedback of doctors and patients is performed after surgery, forming a scientific surgery method of data source-head modeling mechanism-planning target-postoperative feedback, and achieving the technical effects of improving the natural degree and coordination degree of postoperative facial expression and improving the patient satisfaction.

Description

Technical Field

[0001] This invention relates to the field of facial plastic surgery, and more specifically, to a method and system for planning facial plastic surgery based on multimodal dynamic expression analysis. Background Technology

[0002] Plastic surgery is a branch of surgery, also known as reconstructive surgery or plastic surgery. It mainly repairs and reshapes the skin, muscles, bones and other tissues or organs to repair injuries, diseases or congenital malformations through surgery, and covers two major categories: functional reconstruction and aesthetic correction.

[0003] Current methods for planning facial plastic surgery primarily rely on medical imaging data from single static expressions, such as CT / MRI scans of neutral expressions. These methods have inherent limitations: first, data limitations, as they only collect facial data from a single expression state (e.g., a neutral expression), failing to capture the coordinated movement of bones, soft tissues, and muscles during dynamic expressions; second, postoperative coordination disorders, as patients may experience difficulties when performing multi-expression actions (e.g., smiling, frowning) after the surgical plan based on static data is executed.

[0004] Skeletal displacement and muscle contraction out of sync lead to limited joint movement; abnormal soft tissue deformation paths cause local stress concentration; and decreased muscle linkage efficiency leads to an increase in facial asymmetry.

[0005] Current methods for planning facial plastic surgery primarily rely on medical imaging data from a single static facial expression, leading to three major clinical problems:

[0006] (1) It is impossible to predict the abnormal traction of the masticatory muscles on the bone after osteotomy;

[0007] (2) The implant placement did not take into account the dynamic compression caused by facial muscle contraction;

[0008] (3) Muscle tightening surgery ignores the differences in strain distribution of the fascia layer under multiple facial expressions;

[0009] Therefore, there is an urgent need for a multimodal dynamic expression-driven surgical planning method to solve the above-mentioned technical problems. Summary of the Invention

[0010] To address the shortcomings of existing technologies, the present invention aims to provide a method and system for planning facial plastic surgery based on multimodal dynamic expression analysis. By employing a multimodal dynamic expression dataset based on standardized expression sequences and performing spatiotemporal alignment on this dataset, the invention comprehensively captures the coordinated movement relationships of bones, muscles, and soft tissues during multiple expressions. It constructs a basic three-dimensional anatomical model and a biomechanical coordination model linking bones, muscles, and soft tissues. Preoperative surgical optimization is performed based on the simulation results of this biomechanical coordination model, and postoperative feedback is provided to both doctors and patients. This forms a coordinated and scientific management method encompassing data source, modeling mechanism, planning objectives, and postoperative feedback. This method solves the problem of incoordination in multi-expression surgery caused by traditional plastic surgery methods that use data from a single expression scan, thereby improving the naturalness and coordination of postoperative facial expressions and increasing patient satisfaction.

[0011] The above-mentioned technical objective of the present invention is achieved through the following technical solution:

[0012] Firstly, a method for planning facial plastic surgery based on multimodal dynamic expression analysis is provided, including the following steps:

[0013] S1. Collect a multimodal dynamic expression dataset of the patient's face under a preset standardized expression sequence. The multimodal dynamic expression dataset includes optical geometric data, ultrasound tissue motion data, four-dimensional imaging data, and electromyographic signal data.

[0014] S2. Perform spatiotemporal registration on the multimodal dynamic expression dataset using spatial reference markers and timestamps to ensure that the data of each modality are aligned in time and space, thus obtaining a spatiotemporally aligned multimodal dynamic expression dataset;

[0015] S3. Based on the spatiotemporally aligned multimodal dynamic facial expression dataset, perform anatomical structure segmentation and topology generation to construct a basic three-dimensional anatomical model of the patient's face.

[0016] S4. Based on the spatiotemporally aligned multimodal dynamic facial expression dataset and the basic three-dimensional anatomical model, construct a biomechanical coordination model;

[0017] S5. Input the proposed surgical plan parameters into the biomechanical coordination model. The biomechanical coordination model performs biomechanical simulation in the background and outputs visualized simulation results.

[0018] S6. Iteratively optimize the surgical plan parameters based on the simulation results until the postoperative dynamic coordination index meets expectations, and generate the final surgical plan.

[0019] S7. Generate intraoperative navigation instructions based on the final surgical plan to guide intraoperative osteotomy positioning and soft tissue fixation operations.

[0020] Furthermore, step S2 includes the following steps:

[0021] Spatial alignment and temporal synchronization are performed, followed by data fusion, to output a spatiotemporally aligned multimodal dynamic facial expression dataset;

[0022] The spatial alignment includes detecting the coordinates of a reference marker point in the optical geometric data and setting it as the origin of the spatial reference system; performing rigid body transformations on the ultrasound tissue motion data and the four-dimensional imaging data using rotation matrices and translation vectors, so that the ultrasound tissue motion data and the four-dimensional imaging data are initially aligned to the origin of the spatial reference system.

[0023] And by using a non-rigid registration algorithm, the local coordinate offset between the ultrasound tissue motion data and the four-dimensional imaging data caused by soft tissue deformation is compensated;

[0024] The time synchronization includes using the high frame rate time axis of the optical geometric data as the main timeline;

[0025] The low-frame-rate ultrasound tissue motion data is interpolated by timestamps using spline interpolation to generate an equivalent high-frame-rate sequence;

[0026] The electromyographic signal data is directly bound to the corresponding optical frame via a timestamp;

[0027] The spatiotemporally aligned multimodal data are integrated into a structured spatiotemporal array, and the spatiotemporally aligned multimodal dynamic expression dataset is output.

[0028] Furthermore, in step S3, the spatiotemporally aligned multimodal dynamic facial expression dataset is input, and anatomical structure segmentation is performed to generate a structural identification map containing the initial contour, including the following steps;

[0029] The optical geometry data is reconstructed using surface skinning to generate a skin surface;

[0030] Based on the strain characteristic analysis of the ultrasound tissue motion data of the subcutaneous tissue, the anatomical boundary of the muscle layer or the anatomical boundary of the fat layer can be deduced.

[0031] The skeletal contour is extracted from the four-dimensional imaging data;

[0032] Furthermore, in step S3, the topology is generated based on the structural identification diagram, including the following steps:

[0033] The bone contour is fitted with NURBS surface to obtain a parametric bone model;

[0034] The anatomical boundaries of the muscle layer are constructed as dynamic cable units with start and end points to obtain the facial muscle model;

[0035] A soft tissue model is generated by using a viscoelastic mesh modeling method based on the skin surface and the anatomical boundary of the muscle layer.

[0036] Output a basic three-dimensional anatomical model, which includes the skeletal model, the facial muscle model, and the soft tissue model.

[0037] Furthermore, in step S4, the biomechanical coordination model is constructed, including the following steps:

[0038] The spatiotemporally aligned multimodal datasets are synchronously integrated to determine dynamic behavior rules;

[0039] A physics-based biomechanical framework model is established based on the dynamic behavior rules and the basic three-dimensional anatomical model.

[0040] The biomechanical framework model is calibrated using reverse engineering or optimization algorithms to obtain the biomechanical coordination model.

[0041] Furthermore, the synchronous integration of the spatiotemporally aligned multimodal dataset to determine dynamic behavior rules includes the following steps:

[0042] The surface point motion trajectory of the optical geometry data is used as the driving target.

[0043] The muscle activation pattern of the electromyographic signal data is used as input stimulus.

[0044] The dynamic behavior rules are obtained by using the ultrasound tissue motion data, the key tissue layer displacement vectors and strain distributions obtained from the four-dimensional imaging data as motion constraints and target verification points.

[0045] Furthermore, the step of constructing a physics-based biomechanical framework model based on the dynamic behavior rules and the basic three-dimensional anatomical model includes the following steps:

[0046] The facial muscle model is simulated as a contractile unit with attachment points. The muscle contraction force is driven by the intensity value of the electromyographic signal data, and the attachment point coordinates are bound to the corresponding anatomical sites of the NURBS curved skeleton model.

[0047] In the soft tissue model, fat, ligaments, and fascia are simulated as deformable materials with viscoelastic properties, connecting muscles to bones or skin. The constitutive relationship of the materials is described by the hyperelastic strain energy function, and the material parameters are calibrated by the strain distribution data of the ultrasonic displacement field.

[0048] Using the skeletal model as a relatively rigid constraint basis and the NURBS surface boundary as an impenetrable constraint for soft tissue deformation, the biomechanical basic framework model is generated.

[0049] Furthermore, step S5 includes the following steps:

[0050] The calculations cover the position, deformation, stress state, and contact relationships of bones, muscles, soft tissues, and skin when a standardized facial expression sequence is executed after a surgical plan has been applied to the patient.

[0051] Visualized simulation results include dynamic surface change cloud maps, organizational displacement animations, or stress hotspot maps, which identify potential unnatural areas or conflict points.

[0052] Furthermore, step S6 includes the following steps:

[0053] The biomechanical coordination model automatically calculates and compares the differences in the movement trajectory, key point displacement, and simulated muscle activation patterns of key facial expressions before and after surgery, and identifies postoperative incoordination risk points predicted by the model to obtain prediction results.

[0054] The surgeon analyzes the predicted results and adjusts the surgical parameters, resulting in the adjusted treatment plan.

[0055] The biomechanical coordination model then simulates the adjusted scheme again;

[0056] Multiple rounds of iterative optimization were carried out until the model's predicted postoperative facial expressions met the target and reached the expected level in terms of biomechanical coordination indicators;

[0057] The optimized surgical plan was finally determined.

[0058] Secondly, it provides a facial plastic surgery planning system based on multimodal dynamic expression analysis, including:

[0059] The multimodal data acquisition module is used to acquire a multimodal dynamic expression dataset of the patient's face under a preset standardized expression sequence. The multimodal dynamic expression dataset includes optical surface scanning data, subcutaneous tissue motion data, deep structure data, and electromyographic signal data.

[0060] The spatiotemporal registration module is used to perform spatiotemporal registration on the multimodal dynamic expression dataset using spatial reference markers and timestamps to ensure that the data of each modality are aligned in time and space, thus obtaining a spatiotemporally aligned multimodal dynamic expression dataset.

[0061] The basic 3D anatomical modeling module is used to perform anatomical structure segmentation and topology generation based on the spatiotemporally aligned multimodal dynamic facial expression dataset, and to construct a basic 3D anatomical model of the patient's face;

[0062] The biomechanical modeling module is used to construct a biomechanical coordination model based on the spatiotemporally aligned multimodal dynamic facial expression dataset and the basic three-dimensional anatomical model.

[0063] The surgical simulation optimization module is used to input the proposed surgical plan parameters into the biomechanical coordination model. The biomechanical coordination model performs biomechanical simulation in the background and outputs visualized simulation results. The surgical plan parameters are iteratively optimized based on the simulation results until the postoperative dynamic coordination index meets the expectations, and the final surgical plan is generated.

[0064] The intraoperative navigation output module is used to generate intraoperative navigation instructions based on the final surgical plan to guide intraoperative osteotomy positioning and soft tissue fixation operations.

[0065] Compared with the prior art, the present invention has the following beneficial effects:

[0066] 1. This invention employs a multimodal dynamic facial expression dataset based on standardized facial expression sequences and performs spatiotemporal alignment on the dataset to comprehensively capture the coordinated movement relationship between bones, muscles, and soft tissues during multiple facial expressions. It constructs a basic three-dimensional anatomical model and a biomechanical coordination model linking bones, muscles, and soft tissues. Based on the simulation results of the biomechanical coordination model, preoperative surgical optimization is performed, and postoperative feedback is provided to both doctors and patients. This forms a coordinated scientific management method encompassing data source, modeling mechanism, planning goals, and postoperative feedback, thereby improving the naturalness and coordination of postoperative facial expressions and patient satisfaction.

[0067] 2. This invention uses a biomechanical coordination model to simulate and pre-plan the mechanical response of procedures such as osteotomy, prosthesis implantation, and muscle tightening under various facial expressions, thereby avoiding problems such as the risk of exposed bone edges, excessive soft tissue traction, and compensatory muscle contraction in advance, thus improving surgical safety.

[0068] 3. This invention generates a quantitative coordination assessment report by automatically comparing postoperative dynamic scanning with the preoperative planning model. For abnormalities in the assessment report, it provides doctors with data-driven experience-based iterative correction of future surgical parameters and provides patients with targeted rehabilitation training. The comparison between early postoperative dynamic assessment and preoperative simulation prediction forms a technical closed loop, which can not only verify the effectiveness but also provide data for subsequent improvements.

[0069] 4. This invention separates and identifies basic structures such as bones, soft tissues, and muscles from raw data through anatomical structure segmentation. Through topological structure generation, the segmented contours are transformed into a computable mathematical model. The generated basic three-dimensional anatomical model is a digital static structural framework that integrates geometric shape and physical properties. It is not only a visualized 3D model, but also a computable carrier for subsequent biomechanical coordination models.

[0070] 5. The basic three-dimensional anatomical model constructed by this invention includes a facial muscle model, a soft tissue model, and a skeletal model. The skeletal model accurately describes the geometric shape of the bones, the facial muscles simulate the muscle contraction path and mechanical transmission, and the soft tissue model is used to calculate the stress distribution when the soft tissue deforms, providing spatial constraints for biomechanical coupling modeling. Attached Figure Description

[0071] The accompanying drawings, which are included to provide a further understanding of embodiments of the invention and form part of this application, do not constitute a limitation thereof. In the drawings:

[0072] Figure 1 This is a flowchart of Embodiment 1 of the present invention;

[0073] Figure 2 This is a block diagram of the modules in Embodiment 2 of the present invention. Detailed Implementation

[0074] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the embodiments and accompanying drawings. The illustrative embodiments and descriptions of this invention are only for explaining this invention and are not intended to limit this invention.

[0075] Example 1: A method for planning facial plastic surgery based on multimodal dynamic expression analysis is provided, including the following steps:

[0076] S1. Collect a multimodal dynamic expression dataset of the patient's face under a preset standardized expression sequence. The multimodal dynamic expression dataset includes optical geometric data, ultrasound tissue motion data, four-dimensional imaging data and electromyographic signal data.

[0077] S2. Spatiotemporal registration of the multimodal dynamic expression dataset is performed using spatial reference markers and timestamps to ensure that the data of each modality are aligned in time and space, resulting in a spatiotemporally aligned multimodal dynamic expression dataset.

[0078] S3. Based on a spatiotemporally aligned multimodal dynamic facial expression dataset, perform anatomical structure segmentation and topology generation to construct a basic three-dimensional anatomical model of the patient's face.

[0079] S4. Construct a biomechanical coordination model based on a spatiotemporally aligned multimodal dynamic facial expression dataset and a basic 3D anatomical model;

[0080] S5. Input the proposed surgical plan parameters into the biomechanical coordination model. The biomechanical coordination model performs biomechanical simulation in the background and outputs visualized simulation results.

[0081] S6. Iteratively optimize the surgical plan parameters based on the simulation results until the postoperative dynamic coordination indicators meet expectations, and generate the final surgical plan.

[0082] S7. Generate intraoperative navigation instructions based on the final surgical plan to guide the surgical procedure.

[0083] In step S1, the preset standardized expression sequence includes key expressions and transition states such as neutral relaxation, smiling (slight / maximum), frowning, surprise, pouting, closing eyes (forced / natural), puffing out cheeks, and opening mouth (different degrees).

[0084] Optical geometry data is obtained by using multiple high frame rate (≥120fps) high resolution cameras to surround the patient's face, requiring the patient to complete a standardized sequence of facial expressions under guidance, and then capturing dynamic information such as three-dimensional deformation of the facial skin surface, key point displacement trajectory, and wrinkle distribution in real time.

[0085] Ultrasound tissue motion data is obtained by integrating a high-frequency linear ultrasound probe and using an optical system to scan the position. When the patient makes standardized facial expressions, it non-invasively and in real time tracks the thickness changes and displacement vectors of the subcutaneous fat layer, fascia layer, and some facial muscles (such as the risorius, zygomaticus major, depressor supercilii, orbicularis oculi, orbicularis oris, etc.) in key areas, including the cheek, cheekbone, submandibular region, and forehead.

[0086] Four-dimensional imaging data is a brief scan performed on the skeletal structure and deeper soft tissue connections when the patient attempts to simulate key facial expressions (such as opening the mouth to smile). This scan obtains four-dimensional information on the bone position, range of motion of the joint (temporomandibular joint), and changes in deep muscle attachment points under the corresponding facial expressions.

[0087] Electromyographic signal data is obtained by attaching surface electrodes to the skin projection locations of key facial expression muscles in the patient, such as the frontalis, corrugator supercilii, orbicularis oculi, zygomaticus major, orbicularis oris, and depressor anguli oris, and simultaneously recording the activation timing and intensity patterns of the corresponding muscles when the patient performs a standardized facial expression sequence.

[0088] Step S2 includes the following steps:

[0089] Spatial alignment and temporal synchronization are performed, followed by data fusion, to output a spatiotemporally aligned multimodal dynamic facial expression dataset;

[0090] Spatial alignment includes detecting the coordinates of reference marker points in optical geometric data and setting them as the origin of the spatial reference system; performing rigid body transformations on ultrasound tissue motion data and four-dimensional imaging data using rotation matrices and translation vectors to initially align the ultrasound tissue motion data and four-dimensional imaging data to the origin of the spatial reference system.

[0091] Furthermore, a non-rigid registration algorithm is used to compensate for the local coordinate shift between ultrasound tissue motion data and four-dimensional imaging data caused by soft tissue deformation.

[0092] Time synchronization includes using a high frame rate timeline based on optical geometric data as the main timeline;

[0093] Low frame rate ultrasound tissue motion data is interpolated by timestamps using spline interpolation to generate an equivalent high frame rate sequence;

[0094] Electromyographic signal data is directly linked to the corresponding optical frame via timestamps;

[0095] The spatiotemporally aligned multimodal data are integrated into a structured spatiotemporal array, outputting a spatiotemporally aligned multimodal dynamic expression dataset.

[0096] Spatial reference markers are physical landmarks attached to the patient's face, such as reflective spheres or anatomical landmarks; timestamps are microsecond-level synchronization time tags generated by the master clock, which are shared by all devices.

[0097] Among them, electromyographic signal data does not have three-dimensional spatial coordinate attributes and does not require spatial alignment.

[0098] By strictly synchronizing and registering different modal data such as optical geometric data, ultrasound tissue motion data, four-dimensional imaging data and electromyographic signal data in time and space, it is possible to comprehensively capture the coordinated movement relationship of bones, muscles and soft tissues during the patient's facial expressions and establish multimodal data association.

[0099] In step S3, a spatiotemporally aligned multimodal dynamic facial expression dataset is input, anatomical structure segmentation is performed, and a structural identification map containing the initial contour is generated, including the following steps;

[0100] Optical geometry data is reconstructed using surface skinning to generate a skin surface;

[0101] Based on the strain characteristics analysis of ultrasound tissue motion data of subcutaneous tissue, the anatomical boundaries of the muscle layer or fat layer can be deduced.

[0102] Extracting skeletal contours from 4D imaging data;

[0103] In step S3, the topology is generated based on the structural identification diagram, including the following steps:

[0104] The bone contour is fitted with NURBS surface to obtain a parametric bone model;

[0105] The anatomical boundaries of the muscle layer are constructed as dynamic cable units with start and end points to obtain the facial muscle model;

[0106] Viscoelastic mesh modeling was performed using the skin surface and the anatomical boundary of the muscle layer to generate a soft tissue model;

[0107] Output a basic 3D anatomical model, which includes a skeletal model, a facial muscle model, and a soft tissue model.

[0108] Anatomical segmentation allows for the separation and identification of basic structures such as bones, soft tissues, and muscles from raw data. Topological generation transforms the segmented contours into a computable mathematical model, which is then output as a parametric model. These two processes form the basis for constructing a fundamental 3D anatomical model. Essentially, a fundamental 3D anatomical model is a digital static structural framework that integrates geometric form and physical properties. It is not only a visualized 3D model but also a computable carrier for subsequent biomechanical coordination models.

[0109] In step S4, a biomechanical coordination model is constructed, including the following steps: synchronously integrating the spatiotemporally aligned multimodal dataset, using the surface point motion trajectory of optical geometric data as the driving target, using the muscle activation pattern of electromyographic signal data as the input stimulus, and using the displacement vector and strain distribution of key tissue layers obtained from ultrasound tissue motion data and four-dimensional imaging data as motion constraints and target verification points to obtain dynamic behavior rules;

[0110] Based on dynamic behavior rules and a basic three-dimensional anatomical model, a physics-based biomechanical framework model is established, including the following steps: Facial muscle models are simulated as contractile units with attachment points, where muscle contraction force is driven by the intensity value of the sEMG activation state sequence, and the attachment point coordinates are bound to the corresponding anatomical sites of the NURBS surface skeleton model; fat, ligaments, and fascia in the soft tissue model are simulated as deformable materials with viscoelastic properties, connecting muscles to bones or skin, with the material constitutive relationship described by a hyperelastic strain energy function, and material parameters calibrated using strain distribution data from an ultrasonic displacement field; the skeleton model is used as a relatively rigid constraint foundation, and the NURBS surface boundary is used as an impenetrable constraint for soft tissue deformation, generating the biomechanical framework model.

[0111] The biomechanical framework model is calibrated using reverse engineering or optimization algorithms to obtain a biomechanical coordination model.

[0112] The biomechanical framework model is calibrated using reverse engineering or optimization algorithms, that is, the material parameters and muscle force-contraction relationship in the model are adjusted. Material parameters, such as the elastic modulus and damping coefficient of different tissue regions, are adjusted so that the simulated motion results of the model (surface deformation, deep tissue displacement) match the multimodal dynamic data of the patient under standardized facial expression sequences collected in reality to the greatest extent.

[0113] Through the above steps, an individualized multi-scale biomechanical model is obtained that can simulate the complete biomechanical chain from muscle activation to skin surface movement when the patient's face makes a specified expression. It clearly describes the interaction and dependence of bones, muscles, fat, fascia, and skin during movement.

[0114] Step S5 includes the following steps:

[0115] The simulation calculates the new positions, deformations, stress states, and contact relationships of bones, muscles, soft tissues, and skin when a standardized facial expression sequence is executed after a surgical plan is applied to the patient. The visualized simulation results include dynamic facial change contour maps, tissue displacement animations, or stress hotspot maps, which identify potential unnatural areas or conflict points.

[0116] Step S6 includes the following steps:

[0117] The biomechanical coordination model automatically calculates and compares the differences in the movement trajectory, key point displacement, and simulated muscle activation patterns of key facial expressions before and after surgery, and identifies the postoperative incoordination risk points predicted by the model to obtain the prediction results.

[0118] The surgeon analyzes the predicted results and adjusts the surgical parameters to obtain the adjusted plan;

[0119] The biomechanical coordination model was used to re-simulate the adjusted scheme.

[0120] Multiple rounds of iterative optimization were conducted until the model's predicted postoperative facial expressions met the target and reached the expected level in terms of biomechanical coordination indicators; finally, the optimized surgical plan was determined.

[0121] In step S7, the optimized final surgical plan is loaded into the surgical navigation system, such as optical or electromagnetic navigation. The navigation system displays the target position of bone osteotomy or movement aligned with the preoperative plan in real time during the operation, overlays the preoperative planned path, and provides visual or spatial guidance during key steps (such as prosthesis placement and tissue tightening fixation points) to ensure that the actual surgical operation is consistent with the planned biomechanical optimization results as much as possible.

[0122] In some embodiments, early postoperative dynamic assessment is further performed 1-3 months after surgery, ensuring that swelling has largely subsided, scars have not solidified, and the patient reproduces the standardized preoperative expression. A scanner records and captures 53 key facial points dynamically at a frame rate of ≥100fps, generating a three-dimensional motion trajectory dataset for comparative analysis. The root mean square error (RMSE) of the Euclidean distance between the measured trajectory and the planned prediction is calculated using an algorithm. Based on evaluation dimensions such as asymmetry, insufficient amplitude, and poor fluency, an evaluation report is generated. The formula for the root mean square error (RMSE) of the Euclidean distance is as follows:

[0123] ;

[0124] Where RMSE is the root mean square error; N is the total number of data points; ||...|| is the Euclidean distance; is the actual measured value at the i-th position; is the model predicted value at the i-th position.

[0125] Threshold standards:

[0126] Low risk: RMSE < 1.0 mm, indicating good postoperative coordination;

[0127] Moderate risk: 1.0mm ≤ RMSE ≤ 2.0mm, specific facial expressions require attention;

[0128] High risk: RMSE > 2.0 mm, significant inconsistency, intervention required.

[0129] Doctors' feedback is based on the evaluation report to verify the effectiveness of the original surgical plan. If RMSE < 1.0mm, it proves that the method is reliable. At the same time, data on the coordination of different surgical procedures are accumulated, such as the difference in effect between zygomatic osteotomy and implantation.

[0130] Based on the assessment report, the patient is guided to carry out rehabilitation training. For example, if the report shows that the right corner of the mouth can only be lifted to 65% of the predicted value, then the following rehabilitation plan is formulated:

[0131] Training exercise: unilateral grin (strengthens the right zygomaticus major muscle);

[0132] Frequency: 3 sets x 15 repetitions per day. Muscle strength can be monitored in real time during the training process using force-sensitive resistors.

[0133] Target: Achieve a displacement ratio of 0.9-1.1 on both sides within 6 weeks.

[0134] Example 2 provides a facial plastic surgery planning system based on multimodal dynamic expression analysis, including:

[0135] The multimodal data acquisition module is used to acquire a multimodal dynamic expression dataset of the patient's face under a preset standardized expression sequence. The multimodal dynamic expression dataset includes optical surface scanning data, subcutaneous tissue motion data, deep structure data, and electromyographic signal data.

[0136] The spatiotemporal registration module is used to perform spatiotemporal registration on the multimodal dynamic expression dataset using spatial reference markers and timestamps to ensure that the data of each modality are aligned in time and space, thus obtaining a spatiotemporally aligned multimodal dynamic expression dataset.

[0137] The basic 3D anatomical modeling module is used to perform anatomical structure segmentation and topology generation based on the spatiotemporally aligned multimodal dynamic facial expression dataset, and to construct a basic 3D anatomical model of the patient's face;

[0138] The biomechanical modeling module is used to construct a biomechanical coordination model based on the spatiotemporally aligned multimodal dynamic facial expression dataset and the basic three-dimensional anatomical model.

[0139] The surgical simulation optimization module is used to input the proposed surgical plan parameters into the biomechanical coordination model. The biomechanical coordination model performs biomechanical simulation in the background and outputs visualized simulation results. The surgical plan parameters are iteratively optimized based on the simulation results until the postoperative dynamic coordination index meets the expectations, and the final surgical plan is generated.

[0140] The intraoperative navigation output module is used to generate intraoperative navigation instructions based on the final surgical plan to guide intraoperative osteotomy positioning and soft tissue fixation operations.

[0141] Working principle: A multimodal dynamic expression dataset of the patient's face under a preset standardized expression sequence is collected and spatiotemporally registered to obtain a spatiotemporally aligned multimodal dynamic expression dataset. Based on this dataset, a basic 3D anatomical model is created, and a biomechanical coordination model is constructed. The proposed surgical plan parameters are input into this model, which then performs biomechanical simulations in the background and outputs visualized simulation results. The surgical plan parameters are iteratively optimized based on the simulation results until the postoperative dynamic coordination indicators meet expectations, generating the final surgical plan. Intraoperative navigation instructions are generated based on the final surgical plan to guide intraoperative osteotomy positioning and soft tissue fixation. The system comprehensively captures the coordinated movement relationship of bones, muscles, and soft tissues during multi-expression processes, constructing a biomechanical coordination model that links the basic 3D anatomical model with the bone-muscle-soft tissue linkage. Preoperative surgical optimization is performed based on the simulation results of the biomechanical coordination model, and postoperative feedback is provided to both the doctor and the patient.

[0142] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0143] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1A device that provides the functions specified in one or more boxes.

[0144] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0145] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0146] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A facial plastic surgery planning system based on multimodal dynamic expression analysis, characterized in that, include: The multimodal data acquisition module is used to acquire a multimodal dynamic expression dataset of the patient's face under a preset standardized expression sequence. The multimodal dynamic expression dataset includes optical surface scanning data, subcutaneous tissue motion data, deep structure data, and electromyographic signal data. The optical surface scanning data is optical geometric data, the subcutaneous tissue motion data is ultrasound tissue motion data, and the deep structure data is four-dimensional imaging data. The spatiotemporal registration module is used to perform spatiotemporal registration on the multimodal dynamic expression dataset using spatial reference markers and timestamps to obtain a spatiotemporally aligned multimodal dynamic expression dataset; the spatiotemporal registration module is specifically configured as follows: Spatial alignment and temporal synchronization are performed, followed by data fusion, to output a spatiotemporally aligned multimodal dynamic facial expression dataset; The spatial alignment includes detecting the coordinates of a reference marker point in the optical geometric data and setting it as the origin of the spatial reference system. Rigid body transformations of rotation matrix and translation vector are performed on the ultrasound tissue motion data and the four-dimensional imaging data to initially align the ultrasound tissue motion data and the four-dimensional imaging data to the origin of the spatial reference frame. And by using a non-rigid registration algorithm, the local coordinate offset between the ultrasound tissue motion data and the four-dimensional imaging data caused by soft tissue deformation is compensated; The time synchronization includes using the high frame rate time axis of the optical geometric data as the main timeline; The low-frame-rate ultrasound tissue motion data is interpolated by timestamps using spline interpolation to generate an equivalent high-frame-rate sequence; The electromyographic signal data is directly bound to the corresponding optical frame via a timestamp; The spatiotemporally aligned multimodal data is integrated into a structured spatiotemporal array, and the spatiotemporally aligned multimodal dynamic expression dataset is output. The basic 3D anatomical modeling module is used to perform anatomical structure segmentation and topology generation based on the spatiotemporally aligned multimodal dynamic facial expression dataset, and to construct a basic 3D anatomical model of the patient's face. The basic three-dimensional anatomical modeling module is specifically configured as follows: The optical geometry data is reconstructed using surface skinning to generate a skin surface; Based on the strain characteristic analysis of the ultrasound tissue motion data of the subcutaneous tissue, the anatomical boundary of the muscle layer or the anatomical boundary of the fat layer can be deduced. Extract the skeletal contour from the four-dimensional imaging data; The topology generation includes: The bone contour is fitted with NURBS surface to obtain a parametric bone model; The anatomical boundaries of the muscle layer are constructed as dynamic cable units with start and end points to obtain the facial muscle model; A soft tissue model is generated by using a viscoelastic mesh modeling method based on the skin surface and the anatomical boundary of the muscle layer. Output a basic three-dimensional anatomical model, which includes the skeletal model, the facial muscle model, and the soft tissue model; The biomechanical modeling module is used to construct a biomechanical coordination model based on the spatiotemporally aligned multimodal dynamic facial expression dataset and the basic three-dimensional anatomical model. The surgical simulation optimization module is used to input the proposed surgical plan parameters into the biomechanical coordination model. The biomechanical coordination model performs biomechanical simulation in the background and outputs visualized simulation results. The surgical plan parameters are iteratively optimized based on the simulation results until the postoperative dynamic coordination index meets the expectations, and the final surgical plan is generated. The intraoperative navigation output module is used to generate intraoperative navigation instructions based on the final surgical plan to guide intraoperative osteotomy positioning and soft tissue fixation operations.

2. The facial plastic surgery planning system based on multimodal dynamic expression analysis according to claim 1, characterized in that, Constructing the biomechanical coordination model includes: The spatiotemporally aligned multimodal datasets are synchronously integrated to determine dynamic behavior rules; A physics-based biomechanical framework model is established based on the dynamic behavior rules and the basic three-dimensional anatomical model. The biomechanical framework model is calibrated using reverse engineering or optimization algorithms to obtain the biomechanical coordination model.

3. The facial plastic surgery planning system based on multimodal dynamic expression analysis according to claim 2, characterized in that, The process of synchronously integrating the spatiotemporally aligned multimodal dataset to determine dynamic behavior rules includes: The surface point motion trajectory of the optical geometry data is used as the driving target; The muscle activation pattern of the electromyographic signal data is used as the input stimulus; The dynamic behavior rules are obtained by using the ultrasound tissue motion data, the key tissue layer displacement vectors and strain distributions obtained from the four-dimensional imaging data as motion constraints and target verification points.

4. The facial plastic surgery planning system based on multimodal dynamic expression analysis according to claim 2, characterized in that, The construction of a physics-based biomechanical framework model based on the dynamic behavior rules and the basic three-dimensional anatomical model includes: The facial muscle model is simulated as a contractile unit with attachment points. The muscle contraction force is driven by the intensity value of the sEMG activation state sequence, and the attachment point coordinates are bound to the corresponding anatomical sites of the NURBS surface skeleton model. In the soft tissue model, fat, ligaments, and fascia are simulated as deformable materials with viscoelastic properties, connecting muscles to bones or skin. The constitutive relationship of the materials is described by the hyperelastic strain energy function, and the material parameters are calibrated by the strain distribution data of the ultrasonic displacement field. Using the skeletal model as a relatively rigid constraint basis and the NURBS surface boundary as an impenetrable constraint for soft tissue deformation, the biomechanical basic framework model is generated.

5. The facial plastic surgery planning system based on multimodal dynamic expression analysis according to claim 1, characterized in that, The surgical simulation optimization module is specifically configured as follows: The calculations determine the new positions, deformations, stress states, and contact relationships of bones, muscles, soft tissues, and skin when a standardized facial expression sequence is executed after a surgical plan has been applied to the patient. Visualized simulation results include dynamic surface change cloud maps, organizational displacement animations, or stress hotspot maps, which identify potential unnatural areas or conflict points.

6. The facial plastic surgery planning system based on multimodal dynamic expression analysis according to claim 1, characterized in that, The intraoperative navigation output module is specifically configured as follows: The biomechanical coordination model automatically calculates and compares the differences in the movement trajectory, key point displacement, and simulated muscle activation patterns of key facial expressions before and after surgery, and identifies postoperative incoordination risk points predicted by the model to obtain prediction results. The surgeon analyzes the predicted results and adjusts the surgical parameters; The revised plan was obtained; The biomechanical coordination model then simulates the adjusted scheme again; Multiple rounds of iterative optimization were carried out until the model's predicted postoperative facial expressions met the target and reached the expected level in terms of biomechanical coordination indicators; The optimized surgical plan was finally determined.

Images