A surgical training method and system based on patient digital twinning

Abstract

The application relates to the technical field of computer-aided surgery, and discloses a surgical training method and system based on patient digital twinning, which comprises the following steps: acquiring multi-modal medical data, constructing a specific surgical digital twin in real-time alignment with a patient through a non-invasive registration process based on an external anatomical reference area; establishing a surgical world model to simulate the dynamic transfer of tissue states, generating an interactive surgical operating environment, and recording an operator behavior sequence; maintaining the spatial correspondence between the digital twin and the patient and generating navigation information, and performing whole-process quantitative evaluation in combination with the behavior sequence and the state transfer result; predicting postoperative results through a result prediction model; and generating personalized feedback or outputting a collaborative control signal according to the evaluation and prediction results; the application realizes non-invasive high-precision real-time registration, objective quantitative capability evaluation, individualized postoperative prediction and training, navigation, and integration of execution, and can significantly improve the precision and safety of complex surgical operations.

Description

Technical Field

[0001] This invention relates to the field of computer-assisted surgery, and more particularly to a surgical training method and system based on a patient's digital twin. Background Technology

[0002] Complex surgical fields such as orthognathic surgery, craniofacial tumor surgery, implant surgery, and skull base surgery place extremely high demands on surgeons' three-dimensional anatomical understanding, precise preoperative planning, real-time intraoperative navigation, and postoperative outcome prediction. These surgeries involve complex spatial relationships between bones, dentition, nerves, blood vessels, soft tissues, and organs, and the surgical outcome highly depends on the surgeon's accurate understanding of the patient's individual anatomical structure and their ability to make real-time intraoperative decisions.

[0003] Traditional surgical training relies primarily on apprenticeship, case observation, cadaver dissection, and the accumulation of clinical experience. These methods have significant limitations in patient-specific simulation, objective competency assessment, repeatable training, and postoperative outcome prediction. Specifically, cadaver dissection cannot reflect the biomechanical properties of living tissue and individual anatomical variations; case observation is limited by the number and diversity of cases, making it difficult to cover rare or complex situations; and the accumulation of experience lacks a standardized quantitative evaluation system, making it difficult to objectively measure training effectiveness.

[0004] Current computer-aided surgical systems primarily serve preoperative planning of clinical treatment protocols, rather than systematic training and competency assessment. Many virtual surgical simulation systems use generic anatomical models, failing to accurately reflect the individualized characteristics of real patients, such as tooth root location, bone thickness, variations in neurovascular course, soft tissue thickness, and facial asymmetry. Furthermore, traditional surgical navigation systems typically rely on implanted bone screws, bony markers, or intraoral fixation devices to achieve patient-image registration. These invasive registration methods not only increase patient trauma but also affect registration accuracy and intraoperative repeatability due to soft tissue movement and marker loosening.

[0005] In recent years, imaging technologies such as cone-beam computed tomography (CBCT), magnetic resonance imaging (MRI), 3D facial scanning, and intraoral scanning, as well as visualization technologies such as artificial intelligence, augmented reality (AR), and mixed reality (MR), have developed rapidly, providing a technological foundation for constructing patient-specific digital models and intelligent interactive environments. However, how to organically integrate these technologies to form a complete closed loop from preoperative planning, simulation training, intraoperative real-time navigation to postoperative outcome prediction, and achieve non-invasive, high-precision patient registration and dynamic spatial correspondence, remains a pressing technical problem to be solved in this field. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of existing surgical training and surgical navigation technologies by providing a surgical training method and system based on patient digital twins. By integrating multimodal medical imaging with non-invasive real-time registration technology, an individualized digital twin model dynamically aligned with the patient is constructed. Based on this model, a surgical world model and an interactive operating environment are established, enabling intelligent assessment, outcome prediction, and real-time navigation guidance for the operator throughout the entire process from simulation training to real surgery. This provides an integrated and precise operation support platform for complex surgical fields.

[0007] On the one hand, the present invention provides a surgical training method based on a patient's digital twin, comprising the following steps: S1: Acquire multimodal medical data of the target patient, perform anatomical segmentation on the multimodal medical data, and perform multimodal registration to construct a patient-specific surgical digital twin that is aligned with the patient in real time. The multimodal registration adopts a non-invasive real-time registration process, uses an external reference tracking device to establish a patient coordinate system, and dynamically registers the multimodal medical images to the patient's physical space. S2: Establish a surgical world model based on the patient-specific surgical digital twin to generate an interactive surgical simulation environment; record the operator's behavior sequence during simulation training, and use the surgical world model to simulate the dynamic state transition of tissue morphology and anatomical risk under surgical operation. S3: During the operation, the real-time spatial correspondence between the patient-specific surgical digital twin and the patient is continuously maintained. Navigation information is generated based on this correspondence. The behavior sequence, the real-time status of the patient-specific surgical digital twin, and the state transition results of the surgical world model are input into the evaluation engine to generate a full-process quantitative evaluation result of the operator's ability. S4: Based on the patient-specific surgical digital twin, the executed virtual surgical plan, the behavioral sequence, and the state transition results of the surgical world model, the postoperative outcome is predicted using the outcome prediction model. S5: Based on the quantitative assessment results of the entire process and the predicted postoperative results, generate personalized feedback or output collaborative control signals for controlling the physical interaction system.

[0008] Further, in step S1, the multimodal medical data includes several types of cone-beam computed tomography (CBCT) images, computed tomography (CT) images, magnetic resonance imaging (MRI) images, three-dimensional facial scan data, intraoral scan data, ultrasound images, and electronic medical record data. The dynamic alignment of the image space with the patient's physical space includes: Rigid registration and / or non-rigid registration are performed on images of different modalities to obtain multimodal fusion data in a unified image space; The external reference tracking device acquires the position and posture of the patient's head in physical space and calculates the transformation matrix from physical space to image space to complete real-time registration.

[0009] Further, in step S1, the non-intrusive real-time registration process includes: Based on the stable external anatomical structure of the patient's target area at various locations, a non-invasive reference area is set, and the three-dimensional coordinate system of the patient's target area is established using the external reference tracking device; Images of different modalities are registered to the three-dimensional coordinate system of the target area of ​​the patient, so as to achieve dynamic alignment between the image space and the physical space of the patient; The segmented three-dimensional anatomical structures are superimposed and displayed in the field of view of virtual reality or augmented reality devices in the form of solid rendering, so that the operator can intuitively observe the spatial relationship between the anatomical structures in an immersive environment. The non-invasive real-time registration process does not rely on bone screws, bone markers, or intraoral fixation devices. The external reference tracking device includes at least one of an optical tracking device, an electromagnetic tracking device, or an inertial measurement unit; The three-dimensional coordinate system of the patient's target site can be repeatedly established by repeatedly identifying the surface geometric features of the non-invasive reference area or non-invasive markers fixed to the area.

[0010] Further, in step S1, the construction of the patient-specific surgical digital twin includes: By integrating the segmented and registered data, a multi-layered digital model is generated, which includes at least the bone layer, dental arch layer, neurovascular layer, soft tissue layer, and organ function layer. Each tissue is assigned specific elastic modulus, Poisson's ratio, and density parameters, and a biomechanical model is established using finite element analysis and / or deep learning prediction models to simulate tissue traction, compression, resection, reconstruction, bone segment movement, and soft tissue response.

[0011] Furthermore, in step S2, the generation of the interactive surgical simulation environment also includes: A surgical knowledge graph is established, which is represented as a graph structure, where nodes represent medical entities and edges represent relationships between entities. The medical entities include a variety of entities such as bones, tooth roots, nerves, blood vessels, soft tissues, instruments, training tasks, risk events, and postoperative indicators; the relationships include a variety of entities such as spatial proximity, connectivity, blood supply, nerve innervation, support, risk association, and outcome influence. The surgical knowledge graph is used to identify risky procedures, explain error events, and generate instructional feedback within the interactive surgical environment.

[0012] Further, in step S2, the behavioral sequence includes at least one of instrument trajectory, line-of-sight trajectory, tissue contact event, operation sequence, and decision path, wherein: The instrument trajectory is represented as a function of time and is used to describe the change of the instrument's position in space; The gaze trajectory is represented as a function of time and is used to describe the change in the operator's gaze point; The tissue contact event is represented as a function of time and is used to describe the contact state between the device and the virtual tissue; The operation sequence is represented as a function of time and is used to record the order of operation steps; The decision path is represented as a function of time, used to describe the choices made by the operator at different points in time; The above behavioral sequences together constitute the operator's behavioral time series, which can be used to assess their spatial understanding, risk identification, operational stability, efficiency, and decision-making logic.

[0013] Furthermore, in step S2, the generation of the interactive surgical simulation environment also includes: In the patient-specific surgical digital twin, a risk weight is assigned to each key anatomical structure, and the total risk is calculated based on the distance field between the virtual instrument, operation path, or bone segment movement and each key anatomical structure. The total risk is expressed as a weighted sum of the risk weight of each structure and its damage prediction probability. During interactive surgical simulations, key anatomical structures can be selectively displayed fully, semi-transparently, with only risk warnings, or hidden, depending on the training mode.

[0014] Further, in step S3, the continuous maintenance of the real-time spatial correspondence between the patient-specific surgical digital twin and the patient includes: During the surgery, the positional changes of the external reference tracking device are continuously tracked, and the patient coordinate system is dynamically updated. The key anatomical landmarks exposed are verified using structured light, laser scanning, or ultrasound surface information obtained during the operation. Calculate the deviation between the position of the marker point in the current patient coordinate system and the corresponding point in the digital twin. When the deviation exceeds a preset threshold, trigger local or global registration correction.

[0015] Further, in step S3, the generation of navigation information includes: Using augmented reality or mixed reality devices, the anatomical structures, lesion areas, and planned paths segmented from multimodal fusion images are overlaid and displayed as navigation views in the surgical field of vision. The planned paths include preoperatively planned osteotomy lines, implant paths, resection boundaries, or implant preparation trajectories. Real-time acquisition of actual surgical traces generated during actual operation; calculation of spatial deviation between the actual surgical traces and the planned path; generation of offset analysis results. When the deviation exceeds a preset threshold, a real-time deviation warning is provided to the operator through at least one of visual alerts, voice prompts, or tactile feedback. The navigation view selectively displays at least one of the following, depending on the type of image being merged and the current operation stage: Bone structure contours based on cone-beam computed tomography (CBCT) images or computed tomography images; Soft tissue layers, nerve and blood vessel course based on magnetic resonance imaging; Tumor extent and invasion boundaries based on magnetic resonance imaging or positron emission tomography (PET) images.

[0016] Preferably, in step S3, the full-process quantitative evaluation results generated by the evaluation engine are determined by the weighted sum of five categories of indicators: planning quality, anatomical protection capability, technical operation capability, functional results, and aesthetic results.

[0017] Furthermore, in step S4, the outcome prediction model includes an outcome prediction base model, whose inputs are the patient-specific surgical digital twin, the virtual surgical plan, and the behavioral sequence, and whose outputs include the postoperative outcomes, the aesthetic outcomes, and the risk of complications. The functional results include occlusal relationship, airway changes, neurological function, masticatory function, and motor function; The aesthetic results include facial symmetry, contour shape, soft tissue support, changes in the nasolabial angle, and improvement in facial proportions; The risks of complications include infection, bleeding, nerve damage, recurrence, soft tissue instability, and bone segment displacement.

[0018] Furthermore, in step S2, the interactive surgical simulation environment supports at least one of the following implementation methods: virtual reality, augmented reality, mixed reality, haptic feedback, surgical navigation, robot assistance, and physical model operation. When tactile feedback is used, the force between the virtual tissue and the instrument is calculated in real time based on the biomechanical characteristics and tissue response of the patient-specific surgical digital twin, and the force feedback is output. Different tissues have specific elastic modulus, density, cutting resistance, puncture resistance, tensile resistance and fracture characteristics.

[0019] Furthermore, in step S2, the surgical world model simulates the dynamic state transition by mapping the current patient state and current operational behavior to the next patient state, and supports counterfactual simulation to analyze the impact of changing operational strategies on the final outcome.

[0020] Furthermore, in step S3, when generating the full-process quantitative evaluation results, the evaluation engine combines the anatomical basic model and the surgical basic model, wherein: The surgical basic model is a motion-centered medical basic model. Its inputs include imaging data, anatomical structures, instruments and trajectories, surgical procedures and clinical outcomes. Its outputs include operation behavior identification or suggestions, risk prediction, and prediction of the next state or final outcome. The surgical basic model has the ability to identify the current structure, explain the reasons for the operation, understand the operation process, and predict the consequences of alternative behaviors. The anatomical model is used to construct a human anatomy cognitive map, defining anatomical risk codes for each structure, including damage probability, functional impact, recovery capacity, compensatory capacity, and severity of complications; the anatomical model also learns population anatomical variations to generate training scenarios that reflect clinical complexity.

[0021] Furthermore, in step S1, the multimodal medical data also includes pathological data and molecular diagnostic data; The patient-specific surgical digital twin further includes at least one of the following: tumor tissue, lesion boundaries, infiltration area, extent of bone destruction, nerve invasion, vascular invasion, and lymph node metastasis information, to support the navigation, simulation, and training of tumor resection, safety boundary assessment, and reconstruction plans.

[0022] Further, in step S5, the output of the cooperative control signal for controlling the physical interaction system further includes: During the simulation, the type, posture, tip position, depth of entry, contact force, and tissue resistance of the instruments are obtained through sensors and tracking systems configured on the surgical instruments. The acquired information is mapped to the patient-specific surgical digital twin, and the tissue-level status is updated in real time. When the surgical world model or the anatomical basis model detects that the instrument is approaching a preset high-risk structure or deviating from the planned path, it outputs warning and continuous guidance signals to the physical interaction system through at least one of the following methods: augmented reality display, voice prompts, color coding, haptic feedback, or instrument vibration.

[0023] Furthermore, in step S5, the output of the cooperative control signal for controlling the physical interaction system further includes: Establish digital twins of robotic arms and instruments, which operate together with the patient-specific surgical digital twin in a surgical world model; Under the operator's control, the robotic arm performs scanning, marking, navigation assistance, field of view adjustment, or instrument delivery tasks based on the patient-specific optimal surgical map; In shared control mode, the robotic arm maintains the stability of the instrument's trajectory, angle, and depth, and restricts entry into dangerous areas; In the supervised autonomous mode, authorized standardized automated operations are performed under safe and authorized conditions, and the operator retains the authority to take over or terminate the operation at any time.

[0024] Preferably, in step S5, the output of the collaborative control signal for controlling the physical interaction system further includes constructing a multi-agent collaborative environment. The multi-agent includes multiple agents such as patient agent, anatomy agent, surgical procedure agent, cognitive assistance agent, expert knowledge agent, teaching agent, navigation agent, robotic arm agent, and outcome prediction agent. Each agent shares the patient-specific surgical digital twin state and achieves real-time anatomical recognition, stage judgment, risk warning, knowledge push, teaching adaptation, spatial navigation, and outcome prediction through collaborative reasoning, forming a unified surgical cognitive and assistance environment. The generation of personalized training feedback or output of collaborative control signals is at least partially accomplished by one or more agents in the multi-agent collaborative environment.

[0025] More preferably, the method further includes: Record the complete operation sequence and results of multiple preoperative simulations under the digital twin of the same patient, and use the result prediction model to comprehensively evaluate the functional recovery, aesthetic results, degree of tissue damage, risk of complications and operation efficiency of each simulation. The simulation scheme with the highest comprehensive score or that meets the preset optimization target is automatically selected to generate the patient's specific optimal surgical map. The optimal surgical map includes the instrument trajectory, movement direction, speed, force, entry depth, tissue exposure sequence and risk avoidance path for each step. The output is used to control the collaborative control signals of the physical interaction system, including enabling the robotic arm or surgical robot to perform automatic teaching or assisted operations according to the patient-specific optimal surgical map.

[0026] On the other hand, the present invention provides a surgical training system based on a patient digital twin, comprising: The digital twin construction module is used to acquire multimodal medical data of the target patient, perform anatomical segmentation on the multimodal medical data, and perform multimodal registration to construct a patient-specific surgical digital twin that is aligned with the patient in real time. The multimodal registration adopts a non-invasive real-time registration process, uses an external reference tracking device to establish a patient coordinate system, and dynamically registers the multimodal medical images to the patient's physical space. An interactive operating environment and simulation module are used to establish a surgical world model based on the patient-specific surgical digital twin to generate an interactive surgical simulation environment; the operator's behavior sequence is recorded during simulation training, and the surgical world model is used to simulate the dynamic state transition of tissue morphology and anatomical risk under surgical operation. The navigation and quantitative assessment module is used to continuously maintain the real-time spatial correspondence between the patient-specific surgical digital twin and the patient during the operation, generate navigation information based on the correspondence, and input the behavior sequence, the real-time state of the patient-specific surgical digital twin, and the state transition results of the surgical world model into the assessment engine to generate a full-process quantitative assessment result of the operator's ability. The outcome prediction module is used to predict postoperative outcomes based on the patient-specific surgical digital twin, the executed virtual surgical plan, the behavioral sequence, and the state transition results of the surgical world model, using an outcome prediction model. The feedback and control output module is used to generate personalized feedback or output collaborative control signals for controlling the physical interaction system based on the quantitative evaluation results of the entire process and the predicted postoperative results.

[0027] Compared with the prior art, the beneficial effects of the present invention are: (1) The present invention adopts a non-invasive registration process based on external anatomical reference area, which does not require bone nails or intraoral fixation devices, avoids invasive trauma and marker drift problems, and improves the non-invasiveness, repeatability and intraoperative spatial correspondence accuracy of patient registration; (2) This invention simulates the dynamic state transfer of tissue morphology and anatomical risk through a surgical world model, which can predict the consequences of different operation behaviors before actual operation, reduce surgical risk, and enhance the training safety and decision-making fortification of complex surgery. (3) This invention inputs the operator’s behavior sequence and the real-time status of the digital twin into the evaluation engine to generate a full-process quantitative evaluation result covering multiple dimensions such as diagnosis, planning, anatomical protection and operation performance. This overcomes the shortcomings of traditional subjective evaluation and provides a standardized basis for surgical training and competency certification. (4) Based on patient-specific digital twins, surgical plans and actual operation processes, this invention predicts various postoperative outcomes such as function, aesthetics and complications, which helps to optimize surgical plans, reduce the incidence of postoperative complications and improve the quality of patient prognosis. (5) Based on the evaluation and prediction results, the present invention generates personalized feedback or collaborative control signals, which can drive physical interaction systems such as tactile feedback, augmented reality navigation, and robot assistance, seamlessly connecting preoperative training, real-time intraoperative navigation and postoperative evaluation, significantly improving the accuracy, collaborative efficiency and intelligence level of surgical operations. Attached Figure Description

[0028] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of a surgical training method based on a patient's digital twin according to the present invention; Figure 2 This is a schematic diagram of a non-intrusive real-time registration and digital twin dynamic maintenance logic according to the present invention; Figure 3 This is a schematic diagram of an intraoperative trace displacement analysis based on augmented reality according to the present invention. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0030] This invention proposes an AI-based surgical training method and system based on patient digital twins, named the Patient-Specific AI Surgical Training Operating System (PAISTOS). This system integrates multimodal data such as CBCT, CT, MRI, 3D facial scans, intraoral scans, ultrasound images, and electronic medical records. Through anatomical segmentation, multimodal registration, soft and hard tissue reconstruction, biomechanical modeling, surgical knowledge graph, interactive simulation, AI assessment, and postoperative outcome prediction, it constructs a patient-specific orthognathic surgery digital twin training environment. The system can not only simulate preoperative assessment, virtual surgical planning, surgical operation training, and postoperative outcome analysis, but also quantitatively evaluate the trainee's diagnostic ability, planning ability, anatomical preservation ability, operational performance, functional results, and aesthetic results throughout the entire process using a Surgical Foundation Model, a Surgical World Model, an Anatomical Foundation Model, a Digital Twin Foundation Model, and an Outcome Prediction Foundation Model.

[0031] The specific embodiments of the present invention will be described below with reference to the accompanying drawings and examples.

[0032] Example 1

[0033] This example uses orthognathic surgery. Orthognathic surgery involves complex and dynamic interactions between the skeleton, dentition, occlusion, neurovascular system, airway, soft tissue morphology, and facial aesthetics. Because different patients exhibit significant differences in skeletal morphology, dentition relationships, facial proportions, soft tissue thickness, neurovascular pathways, and anatomical variations, traditional training methods often fail to fully reflect the complexity of real clinical practice. Therefore, PAISTOS can further construct a patient-specific digital twin training environment for orthognathic surgery, achieving a complete training loop from preoperative analysis, virtual planning, intraoperative simulation to postoperative outcome prediction.

[0034] Please see Figure 1 The technical solution for a surgical training method based on a patient's digital twin provided in this embodiment includes the following steps: S1: Acquire multimodal medical data of the target patient, perform anatomical segmentation on the multimodal medical data, and perform multimodal registration to construct a patient-specific surgical digital twin that is aligned with the patient in real time. The multimodal registration adopts a non-invasive real-time registration process, uses an external reference tracking device to establish a patient coordinate system, and dynamically registers the multimodal medical images to the patient's physical space. S2: Establish a surgical world model based on the patient-specific surgical digital twin to generate an interactive surgical simulation environment; record the operator's behavior sequence during simulation training, and use the surgical world model to simulate the dynamic state transition of tissue morphology and anatomical risk under surgical operation. S3: During the operation, the real-time spatial correspondence between the patient-specific surgical digital twin and the patient is continuously maintained. Navigation information is generated based on this correspondence. The behavior sequence, the real-time status of the patient-specific surgical digital twin, and the state transition results of the surgical world model are input into the evaluation engine to generate a full-process quantitative evaluation result of the operator's ability. S4: Based on the patient-specific surgical digital twin, the executed virtual surgical plan, the behavioral sequence, and the state transition results of the surgical world model, the postoperative outcome is predicted using the outcome prediction model. S5: Based on the quantitative assessment results of the entire process and the predicted postoperative results, generate personalized feedback or output collaborative control signals for controlling the physical interaction system.

[0035] First, data acquisition is performed in step S1. The multimodal medical data includes several types of data, such as cone-beam computed tomography (CBCT) images, computed tomography (CT) images, magnetic resonance imaging (MRI) images, three-dimensional facial scan data, intraoral scan data, ultrasound images, and electronic medical record data. The dynamic alignment of the image space with the patient's physical space includes: Rigid registration and / or non-rigid registration are performed on images of different modalities to obtain multimodal fusion data in a unified image space; The external reference tracking device acquires the position and posture of the patient's head in physical space and calculates the transformation matrix from physical space to image space to complete real-time registration.

[0036] In this embodiment, PAISTOS first acquires the patient's CBCT, CT, MRI, 3D facial scan, intraoral scan, ultrasound images, and electronic medical record data. CBCT is mainly used to reconstruct the craniofacial skeletal structure, tooth root location, airway morphology, and bone density distribution. CT can be used to supplement a larger range of bony anatomy, organ structures, and lesion extent. MRI is used to obtain information on the layers of nerves, blood vessels, muscles, fascia, and soft tissues, and is particularly suitable for supplementing the deficiencies of CBCT and CT in displaying soft tissues. 3D facial scan is used to obtain skin surface, facial contours, facial symmetry, and expression status. Intraoral scan is used to obtain high-precision dentition, occlusal relationships, gingival contours, and dental arch morphology. Ultrasound images can be used to supplement dynamic soft tissue information. The electronic medical record provides diagnostic, medical history, surgical records, postoperative outcomes, and long-term follow-up information.

[0037] After multimodal data acquisition, the system uses a 3D medical basic model for automatic anatomical segmentation. For each voxel or surface point, the model predicts the probability of it belonging to different anatomical categories: ,in, Indicates the spatial location within the input image. This indicates the corresponding anatomical category. The structures the system needs to identify include bones, teeth, muscles, blood vessels, nerves, soft tissues, airways, nasal cavity, maxillary sinuses, and other organ structures. The segmentation result can be represented as: The goal of this phase is to establish a patient-specific unified anatomical atlas to provide a foundation for subsequent registration, risk modeling, and simulation training.

[0038] Because CBCT, CT, MRI, facial scans, and intraoral scans come from different devices, their coordinate systems, resolutions, and tissue representations are not consistent; therefore, PAISTOS requires registration. This invention employs a non-invasive real-time registration process for multimodal registration, which includes: Based on the stable external anatomical structure of the patient's target area at various locations, a non-invasive reference area is set, and the three-dimensional coordinate system of the patient's target area is established using the external reference tracking device; Images of different modalities are registered to the three-dimensional coordinate system of the target area of ​​the patient, so as to achieve dynamic alignment between the image space and the physical space of the patient; The segmented three-dimensional anatomical structures are superimposed and displayed in the field of view of virtual reality or augmented reality devices in the form of solid rendering, so that the operator can intuitively observe the spatial relationship between the anatomical structures in an immersive environment. The non-invasive real-time registration process does not rely on bone screws, bone markers, or intraoral fixation devices. The external reference tracking device includes at least one of an optical tracking device, an electromagnetic tracking device, or an inertial measurement unit; The three-dimensional coordinate system of the patient's target site can be repeatedly established by repeatedly identifying the surface geometric features of the non-invasive reference area or non-invasive markers fixed to the area.

[0039] In the orthognathic surgery of this embodiment, the system first performs rigid registration: , in, Represents the rotation matrix. This represents the translation vector. Rigid registration is used to align overall structures such as bones, dentition, and facial surfaces. Subsequently, the system performs non-rigid registration: This process addresses soft tissue deformities, facial surface differences, and local morphological inconsistencies between MRI and CBCT. Ultimately, all data is integrated into the Unified PatientCoordinate System (UPC) to create a unified patient model.

[0040] Based on the above cutting and registration, step S1, patient digital twin construction, is performed. The construction of the patient-specific surgical digital twin includes: By integrating the segmented and registered data, a multi-layered digital model is generated, which includes at least the bone layer, dental arch layer, neurovascular layer, soft tissue layer, and organ function layer. Each tissue is assigned specific elastic modulus, Poisson's ratio, and density parameters, and a biomechanical model is established using finite element analysis and / or deep learning prediction models to simulate tissue traction, compression, resection, reconstruction, bone segment movement, and soft tissue response.

[0041] In this embodiment, after segmentation and registration, PAISTOS constructs a patient-specific orthognathic surgical digital twin. This digital twin comprises five main layers. The first layer is the skeletal layer, describing the maxilla, mandible, zygomatic bone, nasal bone, skull base, and related bony supporting structures. The second layer is the dental arch layer, describing teeth, roots, periodontal structures, arch morphology, and occlusal relationships. The third layer is the neurovascular layer, describing important nerves and blood vessels and their spatial relationships with bony structures and roots. The fourth layer is the soft tissue layer, describing skin, fat, fascia, muscles, lip tissue, and nasal soft tissue. The fifth layer is the organ function layer, describing airway, nasal cavity, maxillary sinus, masticatory function, sensory function, and facial aesthetic parameters. The final generated patient digital twin can be represented as follows: ,in, Represents skeletal structure. Indicates the dental arch structure. Represents neural structures, Indicates vascular structure. Indicates soft tissue structure. Indicates the functional status.

[0042] Among them, the non-intrusive real-time registration and digital twin dynamic maintenance logic is as follows: Figure 2 As shown.

[0043] Secondly, the biomechanical model established in this implementation using finite element analysis and / or deep learning prediction models specifically includes: To simulate bone segment movement, soft tissue response, and postoperative morphological changes, PAISTOS employs finite element analysis and deep learning prediction models. Each tissue is assigned specific elastic modulus, Poisson's ratio, and density parameters, expressed as follows: Under linear approximation conditions, the relationship between tissue stress and strain can be expressed as: In real-world systems, soft tissues typically exhibit nonlinear, anisotropic, and viscoelastic properties, thus allowing for the use of more complex constitutive models to describe the dynamic responses of skin, fat, muscle, and fascia. This module simulates tissue traction, compression, resection, reconstruction, bone segment movement, soft tissue rebound, and postoperative recovery processes.

[0044] Based on this, the construction of patient-specific surgical digital twins is achieved through a basic digital twin model. This basic digital twin model has the ability to perform multimodal fusion, complete missing data, and model the time dimension, enabling the digital twin to cover the preoperative, intraoperative, early postoperative, recovery, and long-term follow-up stages to simulate the changes in the patient's condition over time.

[0045] Specifically, the Digital Twin Foundation Model is the foundational model in PAISTOS used to generate patient-level digital twins. Its inputs include CBCT, CT, MRI, 3D facial scans, intraoral scans, ultrasound images, and electronic medical records, and its output is a complete patient digital twin. The core capabilities of this model include multimodal fusion, missing data completion, and temporal modeling. When MRI is missing, the system can predict soft tissue structures based on other images and population statistical models; when intraoral scans are missing, the system can reconstruct dental morphology based on CBCT and dental arch statistical models; when facial scans are missing, the system can predict facial surface morphology based on images and soft tissue models. Digital twins should also have a temporal dimension: This time series can cover the preoperative, intraoperative, early postoperative, recovery, and long-term follow-up stages, enabling the system to not only simulate static anatomy but also predict the changes in the patient's condition over time.

[0046] In addition, the multimodal medical data also includes pathological data and molecular diagnostic data; The patient-specific surgical digital twin further includes at least one of the following: tumor tissue, lesion boundaries, infiltration area, extent of bone destruction, nerve invasion, vascular invasion, and lymph node metastasis information, to support the navigation, simulation, and training of tumor resection, safety boundary assessment, and reconstruction plans.

[0047] Next, step S2 involves generating the interactive surgical simulation environment, including: A surgical knowledge graph is established, which is represented as a graph structure, where nodes represent medical entities and edges represent relationships between entities. The medical entities include a variety of entities such as bones, tooth roots, nerves, blood vessels, soft tissues, instruments, training tasks, risk events, and postoperative indicators; the relationships include a variety of entities such as spatial proximity, connectivity, blood supply, nerve innervation, support, risk association, and outcome influence. The surgical knowledge graph is used to identify risky procedures, explain error events, and generate instructional feedback within the interactive surgical environment.

[0048] The generation of the interactive surgical simulation environment also includes: In the patient-specific surgical digital twin, a risk weight is assigned to each key anatomical structure, and the total risk is calculated based on the distance field between the virtual instrument, operation path, or bone segment movement and each key anatomical structure. The total risk is expressed as a weighted sum of the risk weight of each structure and its damage prediction probability. During interactive surgical simulations, key anatomical structures can be selectively displayed fully, semi-transparently, with only risk warnings, or hidden, depending on the training mode.

[0049] Specifically, in this embodiment, PAISTOS establishes a unified surgical knowledge graph to connect anatomical structures, instruments, operative tasks, risk events, complications, and functional indicators. This graph can be represented as: ,in, Indicates a medical entity. This represents the relationships between entities. Medical entities include the maxilla, mandible, tooth roots, nerves, blood vessels, soft tissues, instruments, training tasks, risk events, and postoperative indicators. Entity relationships include adjacent_to, connected_to, supplies, innervates, supports, risk_to, and effects_outcome. For example, the infraorbital nerve has a spatial proximity relationship with the maxilla, and the blood supply to the palate has a supply relationship with the soft tissues of the palate. This knowledge graph is not used to generate realistic surgical procedure guidelines, but rather to identify risks, interpret errors, and generate instructional feedback in a simulation environment.

[0050] In step S2, the behavioral sequence includes at least one of instrument trajectory, line-of-sight trajectory, tissue contact event, operation sequence, and decision path, wherein: The instrument trajectory is represented as a function of time and is used to describe the change of the instrument's position in space; The gaze trajectory is represented as a function of time and is used to describe the change in the operator's gaze point; The tissue contact event is represented as a function of time and is used to describe the contact state between the device and the virtual tissue; The operation sequence is represented as a function of time and is used to record the order of operation steps; The decision path is represented as a function of time, used to describe the choices made by the operator at different points in time; The above behavioral sequences together constitute the operator's behavioral time series, which can be used to assess their spatial understanding, risk identification, operational stability, efficiency, and decision-making logic.

[0051] In the virtual training environment of this embodiment, trainees can complete preoperative analysis, virtual planning, instrument selection, simulated operation, bone segment movement, occlusion assessment, fixation strategy assessment, and postoperative outcome analysis based on the patient's digital twin. The system records the trainee's complete behavioral sequence, including instrument trajectory: Eye movement trajectory: Organizational contact incidents: Operation sequence: and decision-making path: These data collectively constitute the trainee's behavioral time series, used to assess their spatial understanding, risk identification, operational stability, efficiency, and decision-making logic.

[0052] PAISTOS establishes risk weights for each key anatomical structure: Total risk can be defined as: ,in, Indicates the first The system predicts the probability of damage to individual structures. It can assign high-weighted risk objects to main nerves, major arteries, and important functional structures, and assign different levels of risk objects to tooth roots, nasal mucosa, maxillary sinus mucosa, soft tissue support structures, and areas of bone weakness. The system calculates the distance field between virtual instruments, operating paths, or bone segment movement and the risk structures in real time. The system can predict the probability of potential injury based on this information. If a trainee enters a risk area, the system can display fully visible, semi-transparently visible, risk area-only indication, or completely hidden anatomical structures depending on the training mode. This design allows the system to simultaneously support instructional training, advanced training, and examination assessment.

[0053] In step S2, the interactive surgical simulation environment supports at least one of the following implementation methods: virtual reality, augmented reality, mixed reality, haptic feedback, surgical navigation, robot assistance, and physical model operation. When tactile feedback is used, the force between the virtual tissue and the instrument is calculated in real time based on the biomechanical characteristics and tissue response of the patient-specific surgical digital twin, and the force feedback is output. Different tissues have specific elastic modulus, density, cutting resistance, puncture resistance, tensile resistance and fracture characteristics.

[0054] Specifically, force feedback is achieved through a multi-degree-of-freedom tactile feedback robotic arm. The operator's handheld instrument is connected to the end effector of the robotic arm, which integrates a high-precision position encoder and a servo motor. When the operator moves the instrument, the position encoder collects the instrument's spatial position and motion state in real time and transmits the data to a real-time control system. The control system calculates the required feedback force based on the tissue contact state in the virtual environment and then drives the servo motor to generate a corresponding reverse torque. This torque is transmitted to the handheld instrument through mechanical linkages or transmission mechanisms, allowing the operator to experience varying degrees of resistance and vibration. Unlike traditional devices that rely on springs or friction mechanisms to generate resistance, this system employs active force feedback control technology, simulating different tissue hardness and cutting characteristics by adjusting the motor output in real time. For example, when the instrument contacts high-hardness cortical bone, the system outputs a larger reverse force; when the instrument enters cancellous bone or a cavity area, the feedback force decreases rapidly, providing the operator with a tactile sensation close to that of real surgical procedures.

[0055] Furthermore, in step S2, the Surgical World Model simulates the dynamic state transition by mapping the current patient state and current operational behavior to the next patient state, and supports counterfactual simulation to analyze the impact of changing operational strategies on the final outcome.

[0056] Specifically, the Surgical World Model is the core module in PAISTOS used to simulate the dynamic world of surgery. Just as a language model learns patterns in language sequences, the Surgical World Model learns the dynamic changes in human tissues, instrument movements, and pathological states within the surgical environment. Its fundamental goal is to establish a predictive mechanism from the current patient state and the current procedure to the next patient state.

[0057] The patient's condition can be represented as: This state encompasses the skeletal, dental, nerve, blood vessel, soft tissue, organ, disease, functional, and risk states. Operational behavior can be represented as: This can be abstractly represented as simulated actions such as traction, separation, hemostasis, resection, suturing, fixation, reconstruction, and bone segment movement. The state transition model can be represented as: This model allows the system to predict the impact of trainee behavior on tissue morphology, anatomical risks, and final outcomes. The Surgical WorldModel also supports counterfactual simulations, allowing trainees or instructors to inquire how final functional outcomes, aesthetic results, and complication risks would change if the planning, bone segment movement direction, instrument path, or reconstruction strategy were altered.

[0058] In step S3, continuously maintaining the real-time spatial correspondence between the patient-specific surgical digital twin and the patient includes: During the surgery, the positional changes of the external reference tracking device are continuously tracked, and the patient coordinate system is dynamically updated. The key anatomical landmarks exposed are verified using structured light, laser scanning, or ultrasound surface information obtained during the operation. Calculate the deviation between the position of the marker point in the current patient coordinate system and the corresponding point in the digital twin. When the deviation exceeds a preset threshold, trigger local or global registration correction.

[0059] Secondly, in step S3, the generation of navigation information includes: S31: Using augmented reality or mixed reality devices, the anatomical structures, lesion areas and planned paths segmented from multimodal fusion images are overlaid and displayed as navigation views in the surgical field of vision. The planned paths include preoperatively planned osteotomy lines, implant paths, resection boundaries or implant preparation trajectories. S32: Real-time acquisition of actual surgical traces generated during actual operation, calculation of the spatial deviation between the actual surgical traces and the planned path, and generation of offset analysis results; S33: When the offset exceeds a preset threshold (judged by three dimensions: spatial deviation, angular deviation and depth deviation), a real-time deviation warning is provided to the operator through at least one of visual warning, voice prompt or tactile feedback; The navigation view selectively displays at least one of the following, depending on the type of image being merged and the current operation stage: Bone structure contours based on cone-beam computed tomography (CBCT) images or computed tomography images; Soft tissue layers, nerve and blood vessel course based on magnetic resonance imaging; Tumor extent and invasion boundaries based on magnetic resonance imaging or positron emission tomography (PET) images.

[0060] Based on this, the evaluation engine is introduced to score the entire process of step S3. The quantitative evaluation result generated by the evaluation engine is determined by the weighted sum of five categories of indicators: planning quality, anatomical protection capability, technical operation capability, functional results, and aesthetic results.

[0061] Specific augmented reality-based intraoperative trajectory deviation analysis, such as Figure 3 As shown.

[0062] In this embodiment, the final score can be expressed as: , in, Indicates the quality of the plan. Indicates anatomical protection capability. Indicates technical operational capability, Indicates the result of the function. Indicating aesthetic results, to This indicates the weight of each corresponding indicator. Planning quality includes diagnostic accuracy, deformity identification, occlusal goals, rationality of bone segment movement, and risk prediction. Anatomical protection capability includes the protection of key nerves, blood vessels, tooth roots, nasal cavity, maxillary sinus, and soft tissue blood supply. Technical operational capability includes movement stability, instrument selection, operational efficiency, path rationality, and tissue contact control. Functional outcomes include occlusal relationship, airway changes, sensory function, and motor function. Aesthetic outcomes include facial symmetry, contour harmony, soft tissue morphology, and improved profile.

[0063] When generating the full-process quantitative evaluation results, the evaluation engine combines the Anatomical Foundation Model and the Surgical Foundation Model, wherein: The surgical basic model is a motion-centered medical basic model. Its inputs include imaging data, anatomical structures, instruments and trajectories, surgical procedures and clinical outcomes. Its outputs include operation behavior identification or suggestions, risk prediction, and prediction of the next state or final outcome. The surgical basic model has the ability to identify the current structure, explain the reasons for the operation, understand the operation process, and predict the consequences of alternative behaviors. The anatomical model is used to construct a human anatomy cognitive map, defining anatomical risk codes for each structure, including damage probability, functional impact, recovery capacity, compensatory capacity, and severity of complications; the anatomical model also learns population anatomical variations to generate training scenarios that reflect clinical complexity.

[0064] Specifically, the Anatomical Foundation Model is used to build a unified cognitive model of human anatomy. This model not only identifies structure names but also learns the spatial, functional, supply, innervation, and variation patterns between different structures. Internally, it can build a Human Anatomy Graph. ,in, It represents anatomical entities such as bones, muscles, blood vessels, nerves, organs, and soft tissues. Relationships such as adjacency, connection, supply, dominance, support, and functional influence are represented. For each structure, the model also defines anatomy risk coding: This encoding incorporates the probability of injury, functional impact, recovery capacity, compensatory capacity, and severity of complications. One of the key capabilities of the Anatomical Foundation Model is its ability to learn anatomical variations within a population, including vascular variations, neural pathway variations, skeletal morphology variations, tooth root location variations, and soft tissue thickness variations. Through this model, PAISTOS can generate training scenarios that more closely reflect clinical complexity based on real patient data.

[0065] The Surgical Foundation Model is the core cognitive engine of PAISTOS. Traditional medical AI often focuses on image recognition, structural segmentation, or disease classification, while the Surgical Foundation Model aims to learn the causal relationships between surgical knowledge, surgical decisions, surgical actions, anatomical risks, and clinical outcomes. Therefore, this model is essentially an action-centric medical foundation model.

[0066] Training data for the Surgical Foundation Model should include intraoperative video, navigation system logs, robotic surgery kinematics logs, CBCT, CT, MRI, PET, ultrasound images, preoperative planning, intraoperative records, postoperative records, complication records, functional recovery records, and long-term follow-up data. Its inputs can be represented as: ,in, Represents image data, Indicates anatomical structure. Indicates the instrument and trajectory. Indicates the surgical procedure. This represents the clinical outcome. The model output can be represented as: ,in, This indicates a suggested or identified action. Indicates risk prediction, This represents a prediction of the next state or final outcome. The model needs to possess the ability to reason about what, why, how, and what-if simultaneously, that is, to identify the current structure, explain the reasons for the operation, understand the operation process, and predict the consequences of alternative behaviors.

[0067] Next, step S4 is performed to predict postoperative results. The result prediction model includes a result prediction base model, whose inputs are the patient-specific surgical digital twin, the virtual surgical plan, and the behavioral sequence. The output of the postoperative results includes functional results, aesthetic results, and complication risks. The functional results include occlusal relationship, airway changes, neurological function, masticatory function, and motor function; The aesthetic results include facial symmetry, contour shape, soft tissue support, changes in the nasolabial angle, and improvement in facial proportions; The risks of complications include infection, bleeding, nerve damage, recurrence, soft tissue instability, and bone segment displacement.

[0068] In this embodiment, PAISTOS establishes an Outcome Prediction Foundation Model to predict the postoperative outcomes that may result from the simulated surgical procedure and process. The model inputs include a digital twin of the patient, the virtual surgical plan, and the trainer's operational sequence; the output includes functional results. Aesthetic results: and the risk of complications: In the orthognathic surgery context, functional outcomes include occlusal relationship, airway morphology, neurological function, and masticatory function; aesthetic outcomes include facial symmetry, contour harmony, lateral profile improvement, nasolabial angle changes, and soft tissue support; complication risks include bleeding, infection, nerve injury, recurrence, bone segment instability, and soft tissue incoordination; and long-term prognosis includes stability, recurrence probability, and patient satisfaction. This model can compare trainee outcomes with expert plans, real postoperative results, and historical case databases.

[0069] Finally, step S5 involves feedback and control output, wherein the output is used to control the collaborative control signal of the physical interaction system, and further includes: S51: During the simulation operation, the type, posture, tip position, depth of entry, contact force and tissue resistance of the instrument are obtained through the sensors and tracking system configured on the surgical instrument; S52: Map the acquired information to the patient-specific surgical digital twin and update the tissue level status in real time; S53: When the surgical world model or the anatomical basis model detects that the instrument is approaching a preset high-risk structure or deviating from the planned path, it outputs warning and continuous guidance signals to the physical interaction system through at least one of augmented reality display, voice prompts, color coding, tactile feedback or instrument vibration.

[0070] The output of the cooperative control signal for controlling the physical interaction system also includes: S54: Establish a digital twin of the robotic arm and a digital twin of the instrument, which run together with the patient-specific surgical digital twin in the surgical world model; S55: Under operator control, the robotic arm performs scanning, marking, navigation assistance, field of view adjustment, or instrument delivery tasks based on the patient-specific optimal surgical map; S56: In shared control mode, the robotic arm maintains the stability of the instrument trajectory, angle, and depth, and restricts entry into hazardous areas; S57: In supervised autonomous mode, authorized standardized automated operations are performed under safe and authorized conditions, and the operator retains the authority to take over or terminate the operation at any time.

[0071] S58: The output of the collaborative control signal for controlling the physical interaction system further includes constructing a multi-agent collaborative environment, wherein the multi-agent includes multiple agents such as patient agent, anatomy agent, surgical procedure agent, cognitive assistance agent, expert knowledge agent, teaching agent, navigation agent, robotic arm agent, and outcome prediction agent; each agent shares the patient-specific surgical digital twin state, and through collaborative reasoning, achieves real-time anatomical recognition, stage judgment, risk warning, knowledge push, teaching adaptation, spatial navigation, and outcome prediction, forming a unified surgical cognitive and assistance environment; The generation of personalized training feedback or output of collaborative control signals is at least partially accomplished by one or more agents in the multi-agent collaborative environment.

[0072] Furthermore, the method of the present invention further includes: Record the complete operation sequence and results of multiple preoperative simulations under the digital twin of the same patient, and use the result prediction model to comprehensively evaluate the functional recovery, aesthetic results, degree of tissue damage, risk of complications and operation efficiency of each simulation. The simulation scheme with the highest comprehensive score or that meets the preset optimization target is automatically selected to generate the patient's specific optimal surgical map. The optimal surgical map includes the instrument trajectory, movement direction, speed, force, entry depth, tissue exposure sequence and risk avoidance path for each step. The output is used to control the collaborative control signals of the physical interaction system, including enabling the robotic arm or surgical robot to perform automatic teaching or assisted operations according to the patient-specific optimal surgical map.

[0073] In summary, this embodiment constructs a patient-level digital twin based on CBCT, CT, MRI, 3D facial scanning, intraoral scanning, and electronic medical records. The digital twin model includes not only the maxilla, mandible, dentition, tooth roots, neurovascular structures, and soft tissues, but also occlusal relationships, temporomandibular joint status, airway structure, masticatory function, and facial aesthetic parameters. The system further establishes a unified patient coordinate system, enabling all anatomical structures to be analyzed and simulated within the same spatial framework.

[0074] For maxillary surgery training scenarios, the system supports simulations of various surgical procedures, including Le Fort I osteotomy, single-segment Le Fort I, double-segment Le Fort I, triple-segment Le Fort I, multi-segment Le Fort I, horseshoe osteotomy, Surgically Assisted Rapid Palatal Expansion (SARPE), maxillary anterior, posterior, superior, inferior, rotational, and deviation correction, as well as complex craniofacial reconstruction. The system can analyze the impact of different osteotomy designs on root safety distance, preservation of palatal blood supply, nasal cavity morphology, maxillary sinus structure, occlusal relationship, midfacial support, and postoperative soft tissue changes.

[0075] Secondly, for mandibular surgery training scenarios, the method in this embodiment supports bilateral sagittal split osteotomy (BSSO), intraoral vertical ramus osteotomy (IVRO), high oblique ramus osteotomy (HORO), mandibular body osteotomy, and various complex mandibular reconstruction procedures. The system automatically reconstructs the inferior alveolar nerve, mental nerve, inferior alveolar vessels, and medullary cavity structure, and establishes a risk map. Trainees can analyze the impact of different bone incision paths, different bone segment movement directions, and different fixation strategies on nerve protection, bone contact area, bone healing stability, and postoperative functional recovery.

[0076] Furthermore, for chin augmentation training scenarios, the system supports various chin reconstruction strategies, including forward movement, backward movement, narrowing, widening, vertical elevation, vertical descent, skew correction, three-dimensional rotation, and chin wing osteotomy (also known as chin wing osteotomy). The system can predict the impact of different bone segment movement schemes on lower lip support, mentolabial sulcus morphology, mandibular border contour, cervicomental angle, and overall facial proportions.

[0077] Furthermore, for bimaxillary surgery training scenarios, the system can simultaneously simulate the coordinated movement relationships between the maxilla, mandible, and chin bone segments. Trainees can compare the effects of different bone segment movement combinations on occlusal reconstruction, facial proportions, airway volume, temporomandibular joint load, and soft tissue response. The system supports complex three-dimensional motion simulations such as anterior-posterior movement, vertical adjustment, lateral deviation correction, pitch rotation, roll rotation, and yaw rotation.

[0078] In complex deformity training scenarios, the system can further support complex cases such as facial asymmetry correction, long face syndrome, short face syndrome, open bite, deep overbite, skeletal Class II deformities, skeletal Class III deformities, syndrome-related craniofacial deformities, and post-traumatic deformity reconstruction. Through patient digital twins and anatomical baseline models, the system can automatically identify key anatomical variations and establish an individualized risk assessment system.

[0079] The shell world model establishes a dynamic relationship between bone segment movement and changes in patient condition. For any virtual surgical plan, the system can predict changes in bone position, soft tissue response, occlusal relationship, airway changes, neurological function effects, and long-term stability. Trainees can compare the differences in outcomes between different planning strategies through counterfactual simulations. For example, the system can predict the impact of different maxillary advancement, mandibular retraction, rotation centers, and chin movement schemes on the final facial morphology and functional outcomes.

[0080] The training process is then evaluated throughout by an assessment engine. The evaluation includes diagnostic accuracy, deformity analysis capability, planning rationality, bone segment movement design, neurovascular protection capability, instrument operation stability, fixation strategy design capability, and postoperative outcome prediction capability. The system not only evaluates the final planning outcome but also analyzes the trainee's decision-making path and cognitive process.

[0081] The predictive model further predicts key indicators such as postoperative occlusion, changes in airway volume, changes in facial soft tissue morphology, improvement in facial symmetry, changes in temporomandibular joint function, nerve recovery, long-term stability, and patient satisfaction. Through a digital twin-driven predictive mechanism, trainers can repeatedly validate different approaches and optimize the decision-making process before real patients enter the operating room.

[0082] Through the above methods, the present invention can construct a unified intelligent training platform for orthognathic surgery covering Le Fort series surgeries, BSSO, IVRO, Genioplasty, Chin Wing, SARPE, bimaxillary surgery, facial asymmetry correction, and complex craniofacial reconstruction, realizing a full-process digital training system from patient anatomy knowledge, preoperative planning, risk assessment, virtual operation to long-term outcome prediction.

[0083] Based on the above methods, this invention provides a surgical training system based on patient digital twins, comprising: The digital twin construction module is used to acquire multimodal medical data of the target patient, perform anatomical segmentation on the multimodal medical data, and perform multimodal registration to construct a patient-specific surgical digital twin that is aligned with the patient in real time. The multimodal registration adopts a non-invasive real-time registration process, uses an external reference tracking device to establish a patient coordinate system, and dynamically registers the multimodal medical images to the patient's physical space. An interactive operating environment and simulation module are used to establish a surgical world model based on the patient-specific surgical digital twin to generate an interactive surgical simulation environment; the operator's behavior sequence is recorded during simulation training, and the surgical world model is used to simulate the dynamic state transition of tissue morphology and anatomical risk under surgical operation. The navigation and quantitative assessment module is used to continuously maintain the real-time spatial correspondence between the patient-specific surgical digital twin and the patient during the operation, generate navigation information based on the correspondence, and input the behavior sequence, the real-time state of the patient-specific surgical digital twin, and the state transition results of the surgical world model into the assessment engine to generate a full-process quantitative assessment result of the operator's ability. The outcome prediction module is used to predict postoperative outcomes based on the patient-specific surgical digital twin, the executed virtual surgical plan, the behavioral sequence, and the state transition results of the surgical world model, using an outcome prediction model. The feedback and control output module is used to generate personalized feedback or output collaborative control signals for controlling the physical interaction system based on the quantitative evaluation results of the entire process and the predicted postoperative results.

[0084] It should be noted that the steps in the surgical training method based on patient digital twins provided in this embodiment can be implemented based on corresponding modules in the surgical training system based on patient digital twins. Those skilled in the art can refer to the technical solution of the system to implement the steps of the method. That is, the embodiments in the system can be understood as preferred examples of implementing the method, and will not be elaborated here.

[0085] Example 2

[0086] To validate the applicability of PAISTOS in complex craniofacial surgery training, a training scenario for temporomandibular joint disc reduction and fixation (Discopexy) based on patient-specific digital twins can be further constructed. This procedure is widely used in the treatment of anterior disc displacement, joint dysfunction, and related structural abnormalities. Its surgical outcome is highly dependent on the surgeon's understanding of disc morphology, glenoid anatomy, condylar kinematics, and surrounding neurovascular structures.

[0087] Traditional training methods primarily rely on surgical observation, cadaver dissection, intraoperative learning, and experience accumulation. However, the temporomandibular joint is an area with limited anatomical space, complex structure, and significant individual differences. Trainees often struggle to fully understand the three-dimensional spatial relationships between the articular disc, condyle, glenoid fossa, posterior disc tissue, and surrounding soft tissues. Particularly during disc reduction and fixation, assessment of disc mobility, determination of the extent of anterior tissue release, prediction of post-reduction stability, and selection of fixation position all heavily depend on experience, and significant differences may exist between different surgeons.

[0088] This invention provides a method for constructing a patient-specific digital twin of the temporomandibular joint (TMJ) based on patient MRI, CBCT, CT, ultrasound images, and electronic medical record data. The system reconstructs the articular disc, condyle, glenoid fossa, joint capsule, masticatory muscles, blood vessels, and nerve structures using multimodal anatomical segmentation and spatial registration techniques, and establishes a dynamic joint model in a unified patient coordinate system. This model not only describes the static anatomical morphology but also establishes the joint kinematic state by incorporating the patient's mouth opening, closing, and functional movement data, thus forming a temporomandibular joint digital twin with a time dimension.

[0089] In the training environment, trainees can simulate and virtually operate on disc reduction and fixation schemes. The system records the trainee's observation path, instrument trajectory, tissue contact events, anatomical identification process, and decision sequence, and uses a surgical world model to predict the impact of different manipulation strategies on disc position, joint stability, motor function, and potential complication risks. For example, when trainees select different degrees of anterior tissue release strategies, different reduction positions, or different fixation schemes, the system can predict in real time changes in disc stability after reduction, changes in condylar movement trajectory, and long-term recurrence risk.

[0090] Meanwhile, the Anatomical Foundation Model continuously assesses the trainee's ability to identify the articular disc, posterior disc tissue, joint capsule, and surrounding important anatomical structures, and calculates the probability of potential injury based on the risk model. The system can select to fully display, semi-transparently display, display risk warnings, or completely hide key anatomical structures according to the training mode, thereby achieving progressive training from basic teaching to advanced ability assessment.

[0091] The Outcome Prediction Foundation Model can further integrate the patient's digital twin status, simulated procedure, and historical case database to predict postoperative functional improvement, pain relief, recovery of joint range of motion, disc stability, and recurrence risk. Trainees can not only observe the immediate anatomical changes resulting from the current procedure but also explore the impact of different decision paths on the final outcome through counterfactual simulation, thereby building a deeper understanding of the causal mechanisms and long-term outcomes of disc reduction and fixation.

[0092] Through the above methods, we are no longer limited to traditional surgical skills training, but instead construct an intelligent training environment for temporomandibular joint surgery based on patient digital twins, anatomical cognition, risk assessment, and outcome prediction. This allows trainees to complete individualized preoperative rehearsals, risk identification, and plan optimization before real patients enter the operating room.

[0093] Example 3

[0094] The PAISTOS of this invention can be further extended to training environments for head and neck tumor surgery and craniomaxillofacial tumor surgery. Tumor surgery involves not only lesion resection itself, but also multiple complex decision-making processes such as tumor boundary determination, protection of important anatomical structures, resection margin design, reconstruction plan selection, functional recovery, and prediction of long-term survival and quality of life. Due to significant differences in tumor location, pathological type, extent of invasion, anatomical variations, and individual patient conditions, traditional training methods often fail to fully simulate the complexity and decision-making processes of real cases.

[0095] This invention can construct patient-specific digital twins of tumors based on CT, CBCT, MRI, PET, PET-CT, ultrasound images, 3D facial scans, pathological data, molecular diagnostic data, and electronic medical records. The system automatically identifies bones, dentition, nerves, blood vessels, muscles, salivary glands, airways, vital organs, and tumor tissue through multimodal anatomical segmentation, and establishes a 3D tumor model in a unified patient coordinate system. For different types of lesions, the system can further identify tumor boundaries, infiltration areas, extent of bone destruction, nerve invasion, vascular invasion, lymph node metastasis, and the risk of distant metastasis, thereby forming a complete patient-level digital twin of the tumor.

[0096] The system supports simulation training scenarios for various pathological types, including ameloblastoma, odontogenic keratocyst, odontogenic myxoma, central giant cell lesion, fibrous ossicular lesion, osteosarcoma, chondrosarcoma, Ewing sarcoma, squamous cell carcinoma, mucoepidermoid carcinoma, adenoid cystic carcinoma, adenocarcinoma, nasal cavity and sinus tumors, skull base tumors, salivary gland tumors, and metastatic lesions.

[0097] For mandibular tumor training scenarios, the system can simulate marginal mandibular resection, segmental mandibular resection, hemimandibular resection, and resection strategies for complex recurrent cases. Trainees can compare the impact of different safety boundary designs on tumor control rate, nerve preservation probability, bone defect extent, and reconstruction complexity. The system can calculate the spatial relationship between the resection boundary and the tumor invasion area in real time and predict potential residual risk.

[0098] For training scenarios involving maxillary and midface tumors, the system can simulate partial maxillary resection, basic maxillary resection, upper maxillary resection, total maxillary resection, orbital content-preserving resection, orbital content resection, and complex craniofacial resection. The system can analyze the impact of different resection extents on orbital support, nasal function, masticatory function, speech function, and facial appearance.

[0099] For training scenarios involving oral and head and neck cancer, the system can automatically reconstruct the primary tumor, cervical lymph nodes, neurovascular system, and surrounding organ structures. Trainees can simulate different treatment pathways such as local resection, extended resection, combined resection, and neck dissection. The system can assess the impact of different resection boundary designs on local control rate, functional preservation rate, and complication risk.

[0100] For reconstruction training scenarios, the system can further simulate free fibular flaps, free iliac flaps, free scapular flaps, radial forearm flaps, anterolateral femoral flaps, and other microsurgical reconstruction techniques. The system can automatically analyze bone defect length, soft tissue defect volume, vascular conditions, and future repair needs, and predict the impact of different reconstruction strategies on occlusal reconstruction, facial symmetry, speech function, swallowing function, and long-term stability.

[0101] The Surgical World Model establishes a dynamic relationship between tumor growth, tumor resection, tissue loss, and tissue reconstruction. For any resection approach, the system can predict the state of remaining tissue, reconstruction needs, potential complications, and long-term functional outcomes. Trainees can use counterfactual simulations to compare the differences in outcomes arising from different resection boundaries, reconstruction strategies, and treatment pathways.

[0102] The Anatomical Foundation Model continuously learns about anatomical variations, pathological invasion patterns, and recurrence patterns in cancer patients. The system can establish patient-specific risk models to perform real-time risk assessments of important nerves, major blood vessels, airway structures, orbital contents, and skull base structures, and predict the probability of potential damage and functional consequences.

[0103] The Outcome Prediction Foundation Model can further predict the risk of local recurrence, regional metastasis, distant metastasis, degree of functional recovery, changes in facial appearance, language function, swallowing function, chewing function, airway function, and long-term survival outcomes. The system can not only assess the current resection plan but also predict the impact of different treatment pathways on the patient's long-term prognosis.

[0104] Through the synergistic effect of patient digital twins, tumor anatomical modeling, surgical world models, and outcome prediction models, the present invention PAISTOS can construct a unified intelligent training platform for tumor surgery covering benign tumors, malignant tumors, craniofacial tumors, head and neck tumors, tumor resection, microscopic reconstruction, and long-term follow-up management, realizing a complete digital training closed loop from tumor identification, resection planning, anatomical protection, tissue reconstruction to long-term outcome prediction.

[0105] Example 4

[0106] In addition to orthognathic surgery, temporomandibular joint surgery, and oral implant surgery, the methods and systems of this invention can be further extended to a digital twin training environment covering the entire lifecycle of patients with cleft lip and palate. Cleft lip and palate is one of the most common congenital craniofacial deformities, and its treatment typically spans infancy, childhood, adolescence, and adulthood, involving multiple disciplines such as plastic surgery, oral and maxillofacial surgery, otolaryngology-head and neck surgery, orthodontics, speech therapy, implant prosthodontics, and psychological rehabilitation. Due to significant individual differences among patients in terms of the degree of anatomical defects, growth and development patterns, soft tissue condition, dentition development, nasal deformities, and long-term functional outcomes, traditional training methods struggle to fully represent the dynamic evolution and long-term decision-making logic of cleft lip and palate treatment.

[0107] PAISTOS can construct patient-specific digital twins for cleft lip and palate based on CT, CBCT, MRI, ultrasound images, 3D facial scans, intraoral scans, speech function assessments, nasal airway assessments, and electronic medical record data. The system can automatically identify lip tissues, orbicularis oris muscle, alar cartilage, nasal septum, alveolar process, hard palate, soft palate, pharyngeal lateral wall, pharyngeal posterior wall, palatopharyngeal muscle, levator palpebrae veli palatine muscle, dental structures, and related neurovascular systems, and establish a dynamic craniofacial model in a unified patient coordinate system.

[0108] Digital twins not only describe the current anatomical state but can also integrate patient age, growth and development stage, past treatment history, and long-term follow-up data to form a digital twin model of the patient with a time dimension. The system can simulate the craniofacial development process of patients from birth to adulthood and predict the impact of different treatment pathways on long-term functional and morphological outcomes.

[0109] For cleft lip repair training scenarios, the system supports various types of cleft lip, including unilateral cleft lip, bilateral cleft lip, incomplete cleft lip, complete cleft lip, microcleft lip, and cleft lip associated with complex syndromes. The training environment can simulate Millard Rotation Advancement, Mohler Modification, Fisher Anatomical Subunit Repair, Tennison-Randall Technique, Manchester Repair, and other cleft lip reconstruction strategies. The system can analyze the impact of different designs on lip peak reconstruction, philtrum morphology, nasal base symmetry, nasal alar position, scar distribution, and long-term facial growth.

[0110] For cleft palate repair training scenarios, the system supports various types of cleft palate according to the Veau classification, including submucosal cleft palate, soft palate cleft, combined hard and soft palate cleft, and cleft palate associated with complex syndromes. The training environment can simulate various cleft palate reconstruction methods. The system can analyze the long-term effects of different repair strategies on soft palate length, velopharyngeal closure function, speech outcomes, maxillary developmental limitations, and scar formation.

[0111] For alveolar cleft repair training scenarios, the system can automatically identify the cleft width, alveolar bone defect volume, canine eruption position, root relationship of adjacent teeth, and nasal floor support structures. Trainees can compare different strategies such as autologous iliac bone graft, tibial bone graft, skull graft, allogeneic bone graft, biomaterial graft, and tissue-engineered bone reconstruction. The system can predict the probability of bone bridge formation, canine eruption path, orthodontic feasibility, probability of implant formation, and long-term alveolar bone stability.

[0112] For nasal deformity reconstruction training scenarios, the system can reconstruct the alar cartilage, lateral crura, medial crura, nasal septum, nasal tip support structures, and nasal airway system. Trainees can analyze the impact of different nasal correction schemes on nasal symmetry, nasal tip projection, alar width, nasal floor morphology, and respiratory function.

[0113] For training scenarios involving the correction of secondary deformities, the system supports complex treatment scenarios such as alveolar bone defect reconstruction, correction of secondary deformities of the lip and nose, scar revision, nasal septum reconstruction, correction of maxillary hypoplasia, orthognathic surgery, distraction osteogenesis, and implant reconstruction. The system can establish a causal relationship model between a patient's past surgeries and current anatomical status, thereby helping trainees understand the decision-making logic in the long-term treatment chain.

[0114] For orthognathic surgery training scenarios related to cleft lip and palate, the system can further simulate Le Fort I, segmental Le Fort I osteotomy, bimaxillary surgery, maxillary advancement, midface reconstruction, and complex asymmetry correction. By combining the patient's growth and development trajectory and previous treatment history, the system can predict the impact of different bone segment movement schemes on facial morphology, occlusion, speech function, nasal airway, and long-term stability.

[0115] For training scenarios involving implant reconstruction related to cleft lip and palate, the system can assess alveolar bone quality, bone graft maturity, restorative space in the edentulous area, and occlusal conditions. Trainees can simulate different implant designs, different placement times, and different restorative plans, and analyze long-term osseointegration and restorative stability.

[0116] The Surgical World Model (SWM) can establish a long-term state transition model during cleft lip and palate treatment. The system can not only simulate the outcome of a single surgery but also predict the impact of multi-stage treatment pathways on the patient's final state. For example, trainees can compare the impact of different primary restorative strategies on subsequent alveolar bone graft requirements, orthodontic complexity, the probability of orthognathic surgery, and the final facial aesthetic outcome.

[0117] The Anatomical Foundation Model continuously learns about anatomical variations, growth and development patterns, and long-term outcomes of different treatment pathways in the cleft lip and palate patient population. The system can automatically identify high-risk anatomical areas, potential recurrence areas, and key structures affecting functional recovery.

[0118] The Outcome Prediction Foundation Model can further predict a patient's facial development trajectory, dentition status, occlusion, speech function, nasal breathing function, psychosocial adaptability, long-term aesthetic outcomes, and quality of life indicators at different ages. Trainees can not only observe the immediate impact of the current surgical plan but also predict potential outcomes years or even decades later, thereby establishing a treatment mindset oriented towards full life-cycle management.

[0119] Through the synergistic effect of patient digital twins, anatomical basic models, surgical world models, and outcome prediction models, PAISTOS can build a unified intelligent training platform for cleft lip and palate repair, including cleft lip repair, cleft palate repair, alveolar cleft bone grafting, nasal deformity reconstruction, secondary deformity correction, orthodontic treatment, orthognathic surgery, implant restoration, and long-term follow-up management. This platform enables a patient-specific digital training and competency assessment system throughout the entire process from birth to adulthood.

[0120] Example 5

[0121] Airway safety is a crucial component of perioperative management in craniomaxillofacial surgery. PAISTOS can create dynamic digital twin models of the airway, integrating imaging data, soft tissue condition, respiratory function, and perioperative parameters into a unified spatial framework.

[0122] The method of this invention can simulate different airway conditions under the Mallampati and Cormack-Lehane classifications, and covers airway management scenarios for patients with Pierre Robin sequence syndrome, Treacher Collins syndrome, Apert syndrome, Crouzon syndrome, and complex craniofacial deformities.

[0123] The method of this invention can predict the impact of treatment options such as orthognathic surgery, tumor resection, trauma reconstruction and distraction osteogenesis on upper airway volume, retrolingual space, pharyngeal cavity area and risk of sleep-disordered breathing, and establish a perioperative risk assessment model.

[0124] Meanwhile, complex craniofacial surgeries often involve a rich neurovascular system. PAISTOS can create patient-specific digital twin models of blood vessels, reconstructing the facial artery, lingual artery, maxillary artery, descending palatine artery, infraorbital artery, inferior alveolar artery, pterygoid venous plexus, and related venous systems.

[0125] This invention not only describes static vascular anatomy but also simulates real-time hemodynamic changes. For different vascular injury events, the system can predict bleeding velocity, decreased surgical field visibility, changes in tissue perfusion, and changes in circulatory status. Trainees can learn risk identification, crisis response, and decision-making processes in a virtual environment. The AI ​​assessment engine can analyze the trainee's reaction speed, decision-making logic, and risk control capabilities in the face of emergencies.

[0126] Example 6

[0127] In this embodiment, the PAISTOS of the present invention can be integrated with robotic arm systems, robot systems, navigation systems, tracking systems, augmented reality systems, mixed reality systems, and patient digital twin systems to construct a patient digital twin-driven robotic arm collaborative operation, intelligent execution, and supervised autonomous surgical platform. It can understand not only the patient's anatomical state but also the instrument state, robotic arm state, current surgical stage, tissue state, risk state, and expected outcome state, thereby achieving real-time collaborative control and intelligent assistance based on the patient digital twin.

[0128] The system establishes a three-layered digital model: a patient digital twin, an instrument digital twin, and a robot digital twin. The patient digital twin describes the patient's anatomical structure, diseased tissue, biomechanical properties, and tissue state; the instrument digital twin describes the instrument's geometry, working area, mechanical properties, and real-time position; and the robot digital twin describes the robot's kinematic model, joint states, range of motion, and execution capabilities. These digital models operate together within the Surgical WorldModel, forming a unified digital surgical environment.

[0129] In one implementation, the robotic arm can perform preoperative scanning tasks. The system acquires real-time anatomical information of the patient through structured light scanners, laser scanners, depth cameras, ultrasound probes, optical coherence tomography (OCT) systems, CBCT, CT, MRI, intraoperative imaging equipment, or other sensors, and automatically updates the patient's digital twin status. The robotic arm can automatically perform surface scanning, bone surface scanning, soft tissue scanning, surgical field reconstruction, tissue measurement, and spatial registration tasks, thereby establishing a real-time patient status model.

[0130] In another implementation, the robotic arm can perform automated marking tasks. Based on the patient's digital twin analysis, the system automatically generates incision lines, bone incision lines, implant paths, tumor resection boundaries, reconstruction areas, risk areas, and locations of important anatomical structures. The robotic arm can map this information onto the patient's surface, the surface of a physical model, or the surgical field using laser projection, augmented reality projection, optical marking, staining marking, or other methods. Marking content may include nerve locations, blood vessel locations, lesion boundaries, safety distances, expected resection margins, implant axis, osteotomy reference lines, and reconstruction reference points.

[0131] In a further embodiment, the robotic arm can perform automated navigation-assisted tasks. The system continuously analyzes the instrument position, patient status, digital twin status, and surgical stage, and calculates safe, risk, and target areas in real time based on the patient's digital twin. When the instrument deviates from the predetermined path, the system can generate visual, voice, tactile, navigation, or augmented reality cues to improve operational accuracy and spatial awareness.

[0132] In one shared control implementation, the robotic arm and the operator collaborate to complete the task. The operator is responsible for high-level decision-making and motion intention control, while the robotic arm is responsible for trajectory stabilization, angle maintenance, depth control, motion smoothing, and safety boundary protection. For example, in implant surgery, the system calculates nerve locations, blood vessel locations, bone distribution, and the target implantation path based on the patient's digital twin. The robotic arm monitors the drill bit position, angle, and depth in real time, automatically reducing its speed, increasing resistance feedback, restricting movement in dangerous directions, or pausing the operation when the drilling path approaches a high-risk area.

[0133] In orthognathic surgery, the system automatically generates the bone incision path, osteotomy depth, and risk areas based on preoperative planning. The robotic arm assists in maintaining the osteotome angle, limiting movement beyond the predetermined osteotomy depth, and displays the real-time positions of tooth roots, nerves, and blood vessels. Simultaneously, the system continuously updates the osteotomy progress based on the digital twin status and analyzes the deviation between the current operation and the planned operation.

[0134] In the oncology surgery implementation, the system can establish a dynamic resection margin analysis model based on the tumor model, pathological information, and imaging data in the patient's digital twin. The robotic arm continuously acquires instrument positions and calculates the relationship between the actual resection path and the planned resection margin. The system can display the expected safety boundary, the remaining tumor risk area, and the location of key anatomical structures in real time, and predict the possible differences in outcomes from different resection paths based on the Surgical World Model.

[0135] In another implementation, the system can establish an automatic step recognition mechanism. A robotic arm, instrument tracking system, vision system, eye-tracking system, voice system, and the patient's digital twin jointly analyze the operator's behavior to automatically determine the current surgical stage. These stages may include incision, flap elevation, exposure, dissection, resection, osteotomy, implantation, fixation, reconstruction, and suturing. The system dynamically adjusts the displayed content, risk warnings, teaching information, navigation information, and augmented reality information based on the current stage.

[0136] In a further implementation, the system can establish an automatic status update mechanism. After the instrument enters the tissue, the system automatically updates the patient's digital twin status based on instrument trajectory, cutting depth, tissue contact events, force feedback data, impedance data, and visual analysis results. For example, after incision is completed, the digital twin automatically records the incision location, incision depth, and tissue dissection range; after flap elevation is completed, the digital twin automatically updates the exposed area; after osteotomy is completed, the digital twin automatically updates the bone segment status; after resection is completed, the digital twin automatically updates the tissue defect status; and after reconstruction is completed, the digital twin automatically updates the tissue reconstruction results. These changes form a time-series model of the patient's status.

[0137] In another implementation, the system can establish an expert motion learning model. The robotic arm continuously records the expert operator's instrument trajectory, movement speed, acceleration, force changes, operation sequence, decision path, tissue handling methods, and risk avoidance strategies. This data is used to train the Surgical Foundation Model, forming an expert motion knowledge base and a standard operating procedure model. During subsequent operations by trainees, the system can automatically compare their motion patterns with the expert model and provide targeted cognitive feedback and skills assessment.

[0138] In a further implementation, the system can establish a patient-specific best surgical strategy learning mechanism. For the same patient digital twin, the operator can conduct multiple preoperative rehearsals in virtual training environments, augmented reality environments, mixed reality environments, haptic feedback environments, navigation environments, or robotic arm simulation environments. The system continuously records instrument trajectories, incision paths, flap elevation paths, dissection paths, osteotomy paths, resection paths, implantation paths, reconstruction paths, suture paths, tissue state changes, risk events, and outcome prediction data during each rehearsal.

[0139] The system further utilizes an Outcome Prediction Foundation Model, a Surgical Foundation Model, and a Surgical World Model to comprehensively evaluate the results of each simulation. Evaluation indicators may include functional recovery, aesthetic outcomes, degree of tissue damage, risk of nerve injury, risk of vascular injury, tumor margin safety, implant stability, occlusal stability, complication risk, operational efficiency, and long-term prognosis. The system automatically selects the simulation plan with the highest comprehensive score or that meets specific optimization goals and establishes a patient-specific best surgical map.

[0140] The optimal surgical map not only includes the final planning result but also complete operational process information, including the instrument trajectory, direction of movement, speed of movement, changes in force, depth of entry, order of tissue exposure, order of anatomical identification, risk avoidance path, and decision-making process for each step. The system can bind the above-mentioned optimal surgical map with the patient's digital twin, thereby forming a personalized optimal operation reference model for that patient.

[0141] In a further embodiment, when entering a real surgical environment, the system can access a patient-specific optimal surgical map and integrate it with the navigation system, augmented reality system, instrument tracking system, and robotic arm control system. The system continuously compares the real-time patient status with the optimal surgical map and automatically generates deviation analysis, risk analysis, spatial positioning analysis, and process prompts.

[0142] In one implementation, the robotic arm can perform automated teaching tasks based on a patient-specific optimal surgical map. The robotic arm can automatically reproduce the optimal instrument trajectory, optimal incision path, optimal flap path, optimal osteotomy path, optimal implantation path, optimal resection path, optimal reconstruction path, and optimal suture sequence in a physical model, training platform, or digital twin environment, thereby serving as a standardized teaching template for trainees to learn and verify.

[0143] In a further implementation, when safety, authorization, and human supervision conditions are met, the robotic arm can use the patient-specific optimal surgical map as the target trajectory. The system continuously compares the real-time patient state with the preoperative simulation state and dynamically corrects the target trajectory using the Anatomical Foundation Model and the Surgical World Model. When there are differences between the patient's anatomical structure, tissue state, bleeding, tumor boundaries, tissue displacement, or intraoperative environment and the preoperative simulation results, the system automatically adjusts the trajectory parameters, depth parameters, velocity parameters, angle parameters, and safety boundary parameters to ensure that the execution process remains consistent with the current actual patient state.

[0144] In another implementation, the system can establish an Expert Surgical Experience Transfer Mechanism. The system continuously collects high-quality case data from expert operators in real surgeries, simulation training, and preoperative drills, and builds an expert digital surgical knowledge base. The system can automatically retrieve expert cases most similar to the current case and map the expert's operational strategies to the current patient's digital twin, generating expert reference paths, expert reference incisions, expert reference osteotomy plans, expert reference resection plans, expert reference implantation plans, and expert reference reconstruction plans.

[0145] In a further implementation, the system can simultaneously integrate patient-specific optimal surgical maps and expert experience knowledge bases. When discrepancies exist between the two, the system can conduct a comprehensive assessment based on anatomical matching, risk level, outcome prediction, long-term stability, and historical success rate, and generate the most suitable individualized execution strategy for the current patient.

[0146] In a further implementation, the system can establish a supervised autonomous mode. The patient digital twin, instrument digital twin, robotic arm digital twin, and surgical world model together constitute a closed-loop control system. The robotic arm can autonomously perform scanning, positioning, measurement, path alignment, tissue marking, navigation verification, field of view adjustment, instrument delivery, and other standardized tasks. When safety and authorization conditions are met, the robotic arm can also perform specific automated operations, such as implant path alignment, osteotomy trajectory maintenance, resection margin distance monitoring, tissue scanning, and patient-specific optimal trajectory execution. All automated behaviors are under human supervision, and the operator retains the authority to take over, modify, or terminate the operation at any time.

[0147] In a further implementation, the system can establish a continuous learning mechanism driven by digital twins. Preoperative planning data, intraoperative robotic arm data, postoperative outcome data, and long-term follow-up data are continuously fed back to the Digital Twin Foundation Model, Anatomical Foundation Model, Surgical Foundation Model, Surgical World Model, and Outcome Prediction Foundation Model, thereby achieving the co-evolution and continuous optimization of the patient digital twin, instrument digital twin, and robotic arm digital twin.

[0148] Through the above implementation methods, PAISTOS can not only operate as a training system, but also as a robotic arm collaboration platform, navigation platform, augmented reality platform, intelligent execution platform, and supervised autonomous surgery platform. The patient digital twin, instrument digital twin, and robotic arm digital twin together constitute a unified digital surgical ecosystem, realizing a complete closed loop from preoperative planning, preoperative rehearsal, optimal strategy learning, intraoperative assistance, intelligent execution to postoperative learning.

[0149] Besides implementing the system and its various devices provided by this invention in purely computer-readable program code, the same functions can be achieved by logically programming the method steps, making the system and its various devices of this invention appear as logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices provided by this invention can be considered as a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0150] Finally, it should be noted that the above description is only a preferred embodiment of the present invention, and the scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be pointed out that for those skilled in the art, any improvements and modifications made without departing from the principle of the present invention should also be considered within the scope of protection of the present invention.

[0151] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A surgical training method based on a patient's digital twin, characterized in that, Includes the following steps: S1: Acquire multimodal medical data of the target patient, perform anatomical segmentation on the multimodal medical data, and perform multimodal registration to construct a patient-specific surgical digital twin that is aligned with the patient in real time. The multimodal registration adopts a non-invasive real-time registration process, uses an external reference tracking device to establish a patient coordinate system, and dynamically registers the multimodal medical images to the patient's physical space. S2: Establish a surgical world model based on the patient-specific surgical digital twin to generate an interactive surgical simulation environment; record the operator's behavior sequence during simulation training, and use the surgical world model to simulate the dynamic state transition of tissue morphology and anatomical risk under surgical operation. S3: During the operation, the real-time spatial correspondence between the patient-specific surgical digital twin and the patient is continuously maintained. Navigation information is generated based on this correspondence. The behavior sequence, the real-time status of the patient-specific surgical digital twin, and the state transition results of the surgical world model are input into the evaluation engine to generate a full-process quantitative evaluation result of the operator's ability. S4: Based on the patient-specific surgical digital twin, the executed virtual surgical plan, the behavioral sequence, and the state transition results of the surgical world model, the postoperative outcome is predicted using the outcome prediction model. S5: Based on the quantitative assessment results of the entire process and the predicted postoperative results, generate personalized feedback or output collaborative control signals for controlling the physical interaction system.

2. The surgical training method based on patient digital twins according to claim 1, characterized in that, In step S1, the multimodal medical data includes several types of cone-beam computed tomography (CBCT) images, computed tomography (CT) images, magnetic resonance imaging (MRI) images, three-dimensional facial scan data, intraoral scan data, ultrasound images, and electronic medical record data. The dynamic alignment of the image space with the patient's physical space includes: Rigid registration and / or non-rigid registration are performed on images of different modalities to obtain multimodal fusion data in a unified image space; The external reference tracking device acquires the position and posture of the patient's head in physical space and calculates the transformation matrix from physical space to image space to complete real-time registration.

3. The surgical training method based on patient digital twins according to claim 2, characterized in that, In step S1, the non-intrusive real-time registration process includes: Based on the stable external anatomical structure of the patient's target area in each part, a non-invasive reference area is set, and the three-dimensional coordinate system of the patient's target area is established using the external reference tracking device; Images of different modalities are registered to the three-dimensional coordinate system of the target area of ​​the patient, so as to achieve dynamic alignment between the image space and the physical space of the patient; The segmented three-dimensional anatomical structures are superimposed and displayed in the field of view of virtual reality or augmented reality devices in the form of solid rendering, so that the operator can intuitively observe the spatial relationship between the anatomical structures in an immersive environment. The non-invasive real-time registration process does not rely on bone screws, bone markers, or intraoral fixation devices. The external reference tracking device includes at least one of an optical tracking device, an electromagnetic tracking device, or an inertial measurement unit; The three-dimensional coordinate system of the patient's target site can be repeatedly established by repeatedly identifying the surface geometric features of the non-invasive reference area or non-invasive markers fixed to the area.

4. The surgical training method based on patient digital twins according to claim 1, characterized in that, In step S1, the construction of the patient-specific surgical digital twin includes: Integrate the segmented and registered data to generate a multi-layered digital model that includes at least the bone layer, dental arch layer, neurovascular layer, soft tissue layer, and organ function layer; Each tissue is assigned specific elastic modulus, Poisson's ratio, and density parameters, and a biomechanical model is established using finite element analysis and / or deep learning prediction models to simulate tissue traction, compression, resection, reconstruction, bone segment movement, and soft tissue response.

5. The surgical training method based on patient digital twins according to claim 1, characterized in that, In step S2, the generation of the interactive surgical simulation environment further includes: A surgical knowledge graph is established, which is represented as a graph structure, where nodes represent medical entities and edges represent relationships between entities. The medical entities include a variety of entities such as bones, tooth roots, nerves, blood vessels, soft tissues, instruments, training tasks, risk events, and postoperative indicators; the relationships include a variety of entities such as spatial proximity, connectivity, blood supply, nerve innervation, support, risk association, and outcome influence. The surgical knowledge graph is used to identify risky procedures, explain error events, and generate instructional feedback within the interactive surgical environment.

6. The surgical training method based on patient digital twins according to claim 1, characterized in that, In step S2, the behavioral sequence includes at least one of instrument trajectory, line-of-sight trajectory, tissue contact event, operation sequence, and decision path, wherein: The instrument trajectory is represented as a function of time and is used to describe the change of the instrument's position in space; The gaze trajectory is represented as a function of time and is used to describe the change in the operator's gaze point; The tissue contact event is represented as a function of time and is used to describe the contact state between the device and the virtual tissue; The operation sequence is represented as a function of time and is used to record the order of operation steps; The decision path is represented as a function of time, used to describe the choices made by the operator at different points in time; The above behavioral sequences together constitute the operator's behavioral time series, which can be used to assess their spatial understanding, risk identification, operational stability, efficiency, and decision-making logic.

7. The surgical training method based on patient digital twins according to claim 1, characterized in that, In step S2, the generation of the interactive surgical simulation environment further includes: In the patient-specific surgical digital twin, a risk weight is assigned to each key anatomical structure, and the total risk is calculated based on the distance field between the virtual instrument, operation path, or bone segment movement and each key anatomical structure. The total risk is expressed as a weighted sum of the risk weight of each structure and its damage prediction probability. During interactive surgical simulations, key anatomical structures can be selectively displayed fully, semi-transparently, with only risk warnings, or hidden, depending on the training mode.

8. The surgical training method based on patient digital twins according to claim 1, characterized in that, In step S3, continuously maintaining the real-time spatial correspondence between the patient-specific surgical digital twin and the patient includes: During the surgery, the positional changes of the external reference tracking device are continuously tracked, and the patient coordinate system is dynamically updated. The key anatomical landmarks exposed are verified using structured light, laser scanning, or ultrasound surface information obtained during the operation. Calculate the deviation between the position of the marker point in the current patient coordinate system and the corresponding point in the digital twin. When the deviation exceeds a preset threshold, trigger local or global registration correction.

9. The surgical training method based on patient digital twins according to claim 3, characterized in that, In step S3, generating navigation information includes: Using augmented reality or mixed reality devices, the anatomical structures, lesion areas, and planned paths segmented from multimodal fusion images are overlaid and displayed as navigation views in the surgical field of vision. The planned paths include preoperatively planned osteotomy lines, implant paths, resection boundaries, or implant preparation trajectories. Real-time acquisition of actual surgical traces generated during actual operation; calculation of spatial deviation between the actual surgical traces and the planned path; generation of offset analysis results. When the deviation exceeds a preset threshold, a real-time deviation warning is provided to the operator through at least one of visual alerts, voice prompts, or tactile feedback. The navigation view selectively displays at least one of the following, depending on the type of image being merged and the current operation stage: Bone structure contours based on cone-beam computed tomography (CBCT) images or computed tomography images; Soft tissue layers, nerve and blood vessel course based on magnetic resonance imaging; Tumor extent and invasion boundaries based on magnetic resonance imaging or positron emission tomography (PET) images.

10. The surgical training method based on patient digital twins according to claim 1, characterized in that, In step S3, the full-process quantitative evaluation results generated by the evaluation engine are determined by the weighted sum of five categories of indicators: planning quality, anatomical protection capability, technical operation capability, functional results, and aesthetic results.

11. The surgical training method based on patient digital twins according to claim 1, characterized in that, In step S4, the outcome prediction model includes an outcome prediction base model, whose inputs are the patient-specific surgical digital twin, the virtual surgical plan, and the behavioral sequence, and whose outputs include the postoperative outcomes, the aesthetic outcomes, and the risk of complications. The functional results include occlusal relationship, airway changes, neurological function, masticatory function, and motor function; The aesthetic results include facial symmetry, contour shape, soft tissue support, changes in the nasolabial angle, and improvement in facial proportions; The risks of complications include infection, bleeding, nerve damage, recurrence, soft tissue instability, and bone segment displacement.

12. The surgical training method based on patient digital twins according to claim 1, characterized in that, In step S2, the interactive surgical simulation environment supports at least one of the following implementation methods: virtual reality, augmented reality, mixed reality, haptic feedback, surgical navigation, robot assistance, and physical model operation. When tactile feedback is used, the force between the virtual tissue and the instrument is calculated in real time based on the biomechanical characteristics and tissue response of the patient-specific surgical digital twin, and the force feedback is output. Different tissues have specific elastic modulus, density, cutting resistance, puncture resistance, tensile resistance and fracture characteristics.

13. The surgical training method based on patient digital twins according to claim 1, characterized in that, In step S2, the surgical world model simulates the dynamic state transition by mapping the current patient state and current operational behavior to the next patient state, and supports counterfactual simulation to analyze the impact of changing operational strategies on the final outcome.

14. The surgical training method based on patient digital twins according to claim 1, characterized in that, In step S3, when generating the full-process quantitative evaluation results, the evaluation engine combines the anatomical basic model and the surgical basic model, wherein: The surgical basic model is a motion-centered medical basic model. Its inputs include imaging data, anatomical structures, instruments and trajectories, surgical procedures and clinical outcomes. Its outputs include operation behavior identification or suggestions, risk prediction, and prediction of the next state or final outcome. The surgical basic model has the ability to identify the current structure, explain the reasons for the operation, understand the operation process, and predict the consequences of alternative behaviors. The anatomical model is used to construct a human anatomy cognitive map, defining anatomical risk codes for each structure, including damage probability, functional impact, recovery capacity, compensatory capacity, and severity of complications; the anatomical model also learns population anatomical variations to generate training scenarios that reflect clinical complexity.

15. The surgical training method based on patient digital twins according to claim 1, characterized in that, In step S1, the construction of the patient-specific surgical digital twin is achieved through a digital twin basic model. The digital twin basic model has the ability to perform multimodal fusion, missing data completion, and time dimension modeling, so that the digital twin covers the preoperative, intraoperative, early postoperative, recovery, and long-term follow-up stages to simulate the changes in the patient's condition over time.

16. The surgical training method based on patient digital twins according to claim 1, characterized in that, In step S1, the multimodal medical data also includes pathological data and molecular diagnostic data; The patient-specific surgical digital twin further includes at least one of the following: tumor tissue, lesion boundaries, infiltration area, extent of bone destruction, nerve invasion, vascular invasion, and lymph node metastasis information, to support the navigation, simulation, and training of tumor resection, safety boundary assessment, and reconstruction plans.

17. The surgical training method based on patient digital twins according to claim 14, characterized in that, In step S5, the output of the cooperative control signal for controlling the physical interaction system further includes: During the simulation, the type, posture, tip position, depth of entry, contact force, and tissue resistance of the instruments are obtained through sensors and tracking systems configured on the surgical instruments. The acquired information is mapped to the patient-specific surgical digital twin, and the tissue-level status is updated in real time. When the surgical world model or the anatomical basis model detects that the instrument is approaching a preset high-risk structure or deviating from the planned path, it outputs warning and continuous guidance signals to the physical interaction system through at least one of the following methods: augmented reality display, voice prompts, color coding, haptic feedback, or instrument vibration.

18. The surgical training method based on a patient digital twin according to claim 17, characterized in that, In step S5, the output of the cooperative control signal for controlling the physical interaction system further includes: Establish digital twins of robotic arms and instruments, which operate together with the patient-specific surgical digital twin in a surgical world model; Under the operator's control, the robotic arm performs scanning, marking, navigation assistance, field of view adjustment, or instrument delivery tasks based on the patient-specific optimal surgical map; In shared control mode, the robotic arm maintains the stability of the instrument's trajectory, angle, and depth, and restricts entry into dangerous areas; In the supervised autonomous mode, authorized standardized automated operations are performed under safe and authorized conditions, and the operator retains the authority to take over or terminate the operation at any time.

19. The surgical training method based on patient digital twins according to claim 1, characterized in that, In step S5, the output of the collaborative control signal for controlling the physical interaction system further includes constructing a multi-agent collaborative environment, wherein the multi-agent includes multiple agents such as patient agent, anatomical agent, surgical procedure agent, cognitive assistance agent, expert knowledge agent, teaching agent, navigation agent, robotic arm agent, and outcome prediction agent. Each intelligent agent shares the patient-specific surgical digital twin state and achieves real-time anatomical recognition, stage judgment, risk warning, knowledge push, teaching adaptation, spatial navigation and outcome prediction through collaborative reasoning, forming a unified surgical cognition and assistance environment; The generation of personalized training feedback or output of collaborative control signals is at least partially accomplished by one or more agents in the multi-agent collaborative environment.

20. The surgical training method based on patient digital twins according to claim 1, characterized in that, The method further includes: Record the complete operation sequence and results of multiple preoperative simulations under the digital twin of the same patient, and use the result prediction model to comprehensively evaluate the functional recovery, aesthetic results, degree of tissue damage, risk of complications and operation efficiency of each simulation. The simulation scheme with the highest comprehensive score or that meets the preset optimization target is automatically selected to generate the patient's specific optimal surgical map. The optimal surgical map includes the instrument trajectory, movement direction, speed, force, entry depth, tissue exposure sequence and risk avoidance path for each step. The output is used to control the collaborative control signals of the physical interaction system, including enabling the robotic arm or surgical robot to perform automatic teaching or assisted operations according to the patient-specific optimal surgical map.

21. A surgical training system based on a patient's digital twin, characterized in that, include: The digital twin construction module is used to acquire multimodal medical data of the target patient, perform anatomical segmentation on the multimodal medical data, and perform multimodal registration to construct a patient-specific surgical digital twin that is aligned with the patient in real time. The multimodal registration adopts a non-invasive real-time registration process, uses an external reference tracking device to establish a patient coordinate system, and dynamically registers the multimodal medical images to the patient's physical space. An interactive operating environment and simulation module are used to establish a surgical world model based on the patient-specific surgical digital twin to generate an interactive surgical simulation environment; the operator's behavior sequence is recorded during simulation training, and the surgical world model is used to simulate the dynamic state transition of tissue morphology and anatomical risk under surgical operation. The navigation and quantitative assessment module is used to continuously maintain the real-time spatial correspondence between the patient-specific surgical digital twin and the patient during the operation, generate navigation information based on the correspondence, and input the behavior sequence, the real-time state of the patient-specific surgical digital twin, and the state transition results of the surgical world model into the assessment engine to generate a full-process quantitative assessment result of the operator's ability. The outcome prediction module is used to predict postoperative outcomes based on the patient-specific surgical digital twin, the executed virtual surgical plan, the behavioral sequence, and the state transition results of the surgical world model, using an outcome prediction model. The feedback and control output module is used to generate personalized feedback or output collaborative control signals for controlling the physical interaction system based on the quantitative evaluation results of the entire process and the predicted postoperative results.

Images