Haptic editing system capable of being mixed with quick virtual editing function of real surgery

Abstract

The invention belongs to the technical field of medical intelligent auxiliary systems, and particularly relates to a haptic editing system capable of being mixed with a quick virtual editing function of a real surgery, which comprises a data acquisition module used for acquiring multi-source volume data of a human body, reconstructing a digital three-dimensional model containing a bony structure, soft tissue and vascular nerves based on the multi-source data, and outputting the reconstructed digital three-dimensional model; in combination with a mixed reality space mapping technology, generating a virtual surgery scene with virtual-real fusion; the interaction control module is used for carrying out organization layering and deformation parameter space construction on the virtual operation scene data to obtain three-dimensional operation scene analysis data; and the application output module is used for analyzing the three-dimensional operation scene analysis data and the operation interaction mechanics data to generate a standard degree evaluation report and risk early warning information. According to the system, a brand-new virtual operation training platform is provided for operation workers through the mixed reality (MR) technology and the haptic interaction technology.

Description

Technical Field

[0001] This invention relates to the field of medical intelligent auxiliary system technology, specifically a force-tactile editing system with mixed reality surgical rapid virtual editing function. Background Technology

[0002] Mixed reality is a technology that combines virtual information with the real environment, allowing users to experience an interactive experience that lies between reality and virtuality. Force-haptic interaction technology, on the other hand, simulates tactile sensations, creating realistic feelings when users interact with virtual objects, thus providing a more natural interactive experience.

[0003] Traditional surgical training and assessment have relied primarily on actual surgical procedures and traditional teaching methods, which have numerous limitations, such as difficulty in accessing resources, high costs, high risks, and difficulty in quantifying the effectiveness of preoperative training. With technological advancements, mixed reality (MR) and force-haptic technologies are gradually becoming effective means to improve this situation. By applying these advanced technologies to surgical training and assessment, highly realistic simulation experiences can be provided in a safe environment, reducing learning and operational costs while improving training effectiveness.

[0004] Traditional surgical training and assessment often lack systematic and quantitative indicators, making it difficult to objectively and accurately evaluate trainees' skill levels. The introduction of mixed reality and force-haptic interaction technologies can virtualize complex surgical procedures, making the training process more vivid and concrete through real-time feedback and interaction. Force-haptic feedback technology, in particular, can realistically simulate the tactile sensations during surgery, enabling trainees to better understand and master the techniques, thereby improving their operational skills.

[0005] Currently, while some surgical training systems based on virtual reality and force-haptic technology exist, they still have shortcomings in system integration, flexible customization of functions, and comprehensive evaluation systems. Therefore, this invention aims to combine mixed reality and force-haptic interaction technologies to develop a complete virtual surgical training and evaluation platform, covering multiple aspects such as curriculum design, operational training, and examination evaluation. By comprehensively utilizing these two advanced technologies, it addresses many problems in traditional surgical training and evaluation.

[0006] For the reasons mentioned above, we propose a force-haptic editing system that enables rapid virtual editing of mixed reality surgery. Summary of the Invention

[0007] The purpose of this invention is to provide a force-tactile editing system with rapid virtual editing capabilities for mixed reality surgery. This system provides surgical personnel with a brand-new virtual surgical training platform through mixed reality (MR) technology and force-tactile interaction technology.

[0008] To achieve the above objectives, the present invention provides the following technical solution: A force-haptic editing system capable of rapid virtual editing in mixed reality surgery includes: The data acquisition module is used to acquire multi-source volumetric data of the human body, including multi-modal volumetric data from MRI, CT, and ultrasound. The modeling module reconstructs a digital 3D model containing bony structures, soft tissues, blood vessels, and nerves based on multi-source data, and combines it with mixed reality spatial mapping technology to generate a virtual surgical scene that blends the virtual and real worlds. The interactive control module performs tissue layering and deformation parameter space construction on the virtual surgical scene data to obtain three-dimensional surgical scene analysis data; it extracts nonlinear transmission characteristics and calculates stress distribution of mechanical parameters in the mixed reality interactive data to form surgical interactive mechanical data. The application output module analyzes 3D surgical scene analysis data and surgical interaction mechanics data to generate a standardization assessment report and risk warning information.

[0009] Specifically, the modeling module includes: The model building unit acquires multimodal volumetric data through MRI, CT, and ultrasound, and simultaneously acquires spatial information of the surgical environment, on-site video stream, and lesion-specific images; it registers multi-source data with markers, reconstructs an integrated three-dimensional model of "bone structure-soft tissue-blood vessels and nerves", optimizes details, and assigns physical parameters; The spatial mapping unit performs deep segmentation and spatiotemporal calibration of surgical environment data to generate mixed reality spatial mapping data, and completes the registration of the model with the world coordinate system and the operating platform. The scene fusion unit overlays virtual models and live video streams, configures multi-view displays on master and slave HMDs, integrates collision detection and force feedback functions, and ultimately generates a virtual-real fusion surgical scene with "visual alignment and multi-role collaboration".

[0010] Specifically: the model building unit includes: Bony structure reconstruction: Based on CT registration data, a bony structure model containing bone surface texture and internal pore structure is generated through threshold segmentation and three-dimensional surface rendering; Soft tissue reconstruction: Based on nuclear magnetic resonance registration data, the region growing algorithm is used to segment muscle, fascia and adipose soft tissue, and volume rendering technology is used to restore the hierarchical distribution of soft tissue and generate a three-dimensional soft tissue structure; Vascular and nerve reconstruction: Combining MRI and ultrasound contrast data, the course of vascular and nerve structures is extracted using a vascular tracking algorithm to generate a tubular vascular and nerve model, clarifying its spatial interpenetration relationship with bony structures and soft tissues.

[0011] The spatial mapping unit employs a region growing algorithm to segment the preprocessed point cloud data. Using known geometric features in the surgical scene as seed points, it clusters based on depth similarity and spatial continuity to separate key targets such as the operating table, surgical instruments, and patient surface manipulation areas, generating surgical scene depth data. Semantic annotation is applied to each segmented region to clarify its spatial extent and functional attributes, providing a target benchmark for subsequent spatiotemporal calibration. High-precision timestamps are added to the surgical scene depth data, surgical environment positioning data, and on-site video stream data. Using the timestamps from the spatial positioning sensor as a benchmark, linear interpolation is used to align the time series data from other data sources.

[0012] The generation of mixed reality spatial mapping data includes: Step 1: Extract video feature point data: Spatial feature points are extracted from the live video stream. Corner detection algorithms are used to identify significant feature points in the video frames, including the edge of the operating table, the outline of instruments, and anatomical landmarks on the patient's body surface. The optical flow method is used to achieve temporal tracking of feature points between consecutive frames to generate video feature point data. Step 2: Calculate the spatial registration mapping relationship: Perform spatial registration calculation between the spatiotemporally calibrated "scene feature matrix" and "video feature point data". The registration formula is as follows: Where n is the number of feature points, The weight coefficient for the i-th feature point (the weight of anatomical landmarks > the weight of instrument feature points). For rotation matrix, These are the spatial coordinates of the feature points (from the scene feature matrix). The translation vector is used; the spatial registration mapping relationship is obtained by minimizing the registration error. Step 3: Integrate real-time environmental monitoring data: Collect real-time monitoring data of the surgical environment, including light intensity, temperature and humidity, and spatial noise; perform feature fusion processing on the "spatial registration mapping relationship" and the "real-time environmental monitoring data" to provide a foundation for subsequent virtual-real registration.

[0013] The scene fusion unit includes: The hardware and permission configuration of the master and slave HMDs involves deploying head-mounted visual devices and using the camera of the master HMD to capture video footage within its field of view. This allows for real-time updates of the virtual model's display angle on the master HMD, automatically switching the virtual model displayed on the HMD to a side view angle to match the student's actual observation position.

[0014] The interactive control module assigns core deformation parameters to each layer of tissue based on "tissue parameter distribution data". It constructs a three-dimensional deformation parameter space according to the "spatial coordinates of the layered tissue". That is, any spatial point (x, y, z) of each tissue layer corresponds to a unique set of deformation parameters, forming a "parameter-space" mapping relationship. Based on the mapping matrix of the three-dimensional deformation parameter space and the parameter statistics of each layer of tissue, it provides a quantitative basis for subsequent surgical process simulation.

[0015] The interactive control module extracts the nonlinear transmission characteristics and calculates the stress distribution of the mechanical parameters in the mixed reality interactive data, forming surgical interactive mechanical data acquisition of the contact force of the instrument operation, the interactive pressure between the hand and the instrument, the real-time contact force between the instrument and the virtual tissue, and the direction vector of the force. It then extracts the core mechanical parameters—the magnitude of the force (F), the direction of the force (θ, φ), and the coordinates of the point of application of the force (θ, φ). The rate of change of force (dF / dt) forms a "time series of mechanical parameters". Based on the nonlinear mechanical response of the tissue, strain data of the surrounding tissue are collected with the point of force application as the center to establish a force-strain curve. Through curve fitting, the nonlinear transmission coefficient is extracted, which reflects the transmission and attenuation law of mechanical parameters in the tissue.

[0016] The biofeedback system also includes a data visualization module, which displays the analyzed results in at least one of the following forms: charts, curves, numerical values, and grade labels, through an MR head-mounted display or an external display device.

[0017] The process of extracting nonlinear transmission characteristics and calculating stress distribution from the mechanical parameters in the mixed reality interactive data to form surgical interactive mechanical data includes: preprocessing the original interactive data, removing noise, filling in missing values ​​to ensure the data accuracy required for the surgical scenario, using numerical calculation methods such as the finite element method to construct a mechanical model of instrument-tissue contact in the mixed reality scenario, and summarizing the nonlinear transmission characteristic parameters and stress distribution results to form a core dataset.

[0018] Compared with the prior art, the beneficial effects of the present invention are: This solution focuses on the unique operational scenarios at sea, characterized by high humidity, frequent turbulence, and complex electromagnetic environments. By integrating mixed reality (MR), multi-source sensing, data fusion analysis, and intelligent feedback technologies, it enables comprehensive collection, precise analysis, and real-time feedback of the physiological and psychological states, operational behaviors, and environmental parameters of surgical personnel at sea. This enhances operational stability and emergency response capabilities, providing technical support for the safety of surgical procedures at sea.

[0019] This solution enables multi-dimensional data collection and integrated analysis, filling gaps in data correlation for maritime scenarios. It simultaneously collects four types of related data: physiological, psychological, operational force / tactile, and environmental data. Through correlated storage and integrated analysis, it establishes an environment-personnel status correspondence model, quantifying the impact coefficient of environmental disturbances on operations and providing a scientific basis for status assessment.

[0020] Enhanced adaptability to marine environments and improved data and operational stability. The turbulence compensation module incorporates an IMU and Kalman filter algorithm to calculate compensation amounts, simultaneously counteracting the interference of hull sway on force and tactile feedback and visual jitter in MR scenes, ensuring operational continuity and data accuracy.

[0021] A real-time, multi-channel biofeedback mechanism is constructed to reduce the physiological and psychological burden on personnel. Based on an MR head-mounted display, visual, auditory, and tactile feedback are provided, with guidance tiered from safety to mild alerts to severe warnings.

[0022] To ensure data security and emergency response efficiency, and reduce surgical risks, a dual architecture of local redundant storage and off-site cloud backup is adopted. Local industrial-grade waterproof equipment ensures that data is not lost during network outages, while the cloud-based maritime dedicated cloud platform supports categorized indexing. Transmission uses AES-256 encryption and low-latency maritime frequency bands, and critical data is authorized in a hierarchical manner. Simultaneously, it links with the ship command center and shore-based expert system, and can initiate remote expert connections to achieve rapid intervention in cases of critical personnel data or operational anomalies.

[0023] Achieving seamless integration of virtual and real elements with a user-friendly design optimizes the operational experience. The MR head-mounted display integrates virtual surgical scenes, the real environment, and layered data, avoiding a disconnect between the virtual and real worlds; the ergonomically designed, non-slip, and sweat-resistant operating handles are suitable for one- or two-handed operation at sea, and the haptic feedback supports multi-mode gradient adjustment, further reducing operator fatigue.

[0024] The system can be linked with maritime emergency platforms, shore-based medical centers, and personnel training platforms. The collected operational and status data can be used to optimize personalized training programs. The data visualization module supports exporting PDF / Excel reports, which can serve as a basis for evaluating operators and optimizing maritime surgical procedures, thereby enhancing the reusability and promotional value of the technology. Attached Figure Description

[0025] Figure 1 This is a system logic block diagram of the present invention. Detailed Implementation

[0026] The purpose of this invention is to develop a virtual force-haptic interactive surgical training platform that utilizes mixed reality (MR) technology to enable training, simulation, and assessment of surgical procedures in a virtual environment. This platform aims to improve surgeons' adaptability and operational skills in complex surgical tasks, enhance the surgical experience through force-haptic feedback, and accelerate the learning and mastery of surgical techniques. For various surgical tasks, relevant virtual force-haptic interactive surgical platforms, adaptive courses, and assessment systems will be developed. A comprehensive assessment system based on mixed reality and force-haptic interaction technologies will be organically integrated, incorporating surgical training and assessment systems, as well as quantitative assessment indicators of psychological, physiological, and practical skills during surgical procedures. This will lead to the development of an intelligent virtual interactive system based on mixed reality and force-haptic interaction technologies.

[0027] Based on various surgical tasks, the surgical tasks are streamlined using Mixed Reality (MR) technology. In the surgical process, virtual force-touch technology is used to develop a virtual surgical procedure with force-touch feedback interaction in an MR environment. Surgical personnel can complete adaptive courses, training, and assessment through a surgical platform with force-touch interaction technology in an MR environment.

[0028] The research and development of virtual interactive system software should at least include two major systems: a surgical training assessment system based on mixed reality technology and force-tactile interaction technology, and a quantitative assessment system for psychological, physiological, and practical skills indicators during surgical operations.

[0029] Example 1 The force-tactile editing system with rapid virtual editing function for mixed reality surgery provided in this application embodiment includes a data acquisition module 201, a modeling module 202, an interactive control module 203, and an application output module 204. The data acquisition module 201 is responsible for acquiring multi-source volumetric data of the human body, including MRI, CT, and ultrasound multimodal volumetric data. The modeling module 202 reconstructs a digital three-dimensional model based on this data and generates a virtual surgical scene that blends virtual and real elements. The interactive control module 203 processes the virtual surgical scene data to obtain three-dimensional surgical scene analysis data and surgical interaction mechanics data. The application output module 204 analyzes this data to generate a standardization assessment report and risk warning information. This achieves the effect of comprehensively and accurately simulating surgical scenes and providing assessment and warnings for surgery. This is because multi-source data can cover more human body information, and the collaborative processing of each module makes the simulation closer to the real surgical situation. The assessment and warning information can assist doctors in better planning and executing surgery.

[0030] Specifically, the data acquisition module 201 includes devices for acquiring MRI data, CT data, and ultrasound data. The MRI data acquisition device is typically a specialized MRI scanner, a diagnostic method that uses signals generated by the resonance of atomic nuclei within a magnetic field to reconstruct images. In this system, it can acquire detailed information about human soft tissues. An alternative could be a more advanced high-field MRI scanner, providing higher resolution images. The CT data acquisition device is a CT scanner, which uses X-rays to perform tomographic scanning of a specific part of the body to obtain information about the bony structures. An alternative could be a dual-source CT scanner, offering faster scanning speeds and higher image quality. The ultrasound data acquisition device is an ultrasound diagnostic instrument, which uses the reflection principle of ultrasound waves to observe the morphology and function of internal organs and tissues. An alternative could be a 3D ultrasound diagnostic instrument, providing more intuitive 3D images. These devices connect to the system via wired or wireless means, transmitting the acquired data to the subsequent processing module. The combination of these devices enables comprehensive acquisition of multi-source volumetric data of the human body, with data from different modalities complementing each other, providing rich and accurate information for subsequent modeling. MRI data can clearly show soft tissues, CT data can accurately present bony structures, and ultrasound data can dynamically observe the condition of some organs and blood vessels.

[0031] Specifically, modeling module 202 includes a model building unit, a spatial mapping unit, and a scene fusion unit. The model building unit includes: Bony structure reconstruction: Based on CT registration data, a bony structure model containing bone surface texture and internal pore structure is generated through threshold segmentation and three-dimensional surface rendering; Soft tissue reconstruction: Based on nuclear magnetic resonance registration data, the region growing algorithm is used to segment muscle, fascia and adipose soft tissue, and volume rendering technology is used to restore the hierarchical distribution of soft tissue and generate a three-dimensional soft tissue structure; Vascular and nerve reconstruction: Combining MRI and ultrasound contrast data, the course of vascular and nerve structures is extracted using a vascular tracking algorithm to generate a tubular vascular and nerve model, clarifying its spatial interpenetration relationship with bony structures and soft tissues.

[0032] Bony structure reconstruction is based on CT registration data. Through thresholding segmentation and 3D surface rendering, a bony structure model including bone surface texture and internal pore structure is generated. Thresholding segmentation can employ a grayscale-based algorithm, segmenting based on the grayscale differences between bone and surrounding tissues in the CT image. 3D surface rendering can use a moving cube algorithm to convert 3D volume data into a 2D surface mesh model. Alternative algorithms include region growing-based thresholding segmentation algorithms and the Marching Tetrahedra algorithm. Soft tissue reconstruction is based on MRI registration data. A region growing algorithm is used to segment muscle, fascia, and adipose tissue. Volume rendering techniques are then used to reconstruct the hierarchical distribution of soft tissue, generating a 3D soft tissue structure. The region growing algorithm starts from a seed point and continuously expands the region based on pixel similarity. Volume rendering techniques can use a ray casting algorithm to directly project the 3D volume data onto a 2D plane for display. Alternative algorithms include the watershed algorithm and the maximum density projection algorithm. Vascular and nerve reconstruction combines magnetic resonance angiography and ultrasound contrast imaging data. The course of vascular and nerve is extracted through a vascular tracking algorithm to generate a tubular vascular and nerve model, clarifying its spatial interpenetration relationship with bony structures and soft tissues.

[0033] The blood vessel tracking algorithm can employ a skeleton extraction-based approach, first extracting the blood vessel skeleton and then generating a tubular model based on the skeleton. An alternative algorithm is a graph-based blood vessel tracking algorithm. These components cooperate through data sharing and seamless processing to jointly build the model. The combination of these components in the model building unit can accurately reconstruct a digital 3D model including bony structures, soft tissues, blood vessels, and nerves. Different reconstruction methods are tailored to different tissue characteristics, ensuring the accuracy and completeness of the model.

[0034] The spatial mapping unit includes components for depth segmentation of surgical environment data, spatiotemporal calibration, and generation of mixed reality spatial mapping data. Depth segmentation uses a region growing algorithm to segment preprocessed point cloud data. Using known geometric features in the surgical scene as seed points, clustering is performed based on depth similarity and spatial continuity to separate key targets such as the operating table, surgical instruments, and patient surface manipulation areas, generating surgical scene depth data. Seed points for the region growing algorithm can be determined based on geometric features such as the edge of the operating table and the shape of surgical instruments. Spatiotemporal calibration adds high-precision timestamps to the surgical scene depth data, surgical environment positioning data, and on-site video stream data. Using the timestamps from the spatial positioning sensors as a reference, linear interpolation is used to align the time series from other data sources. Generating mixed reality spatial mapping data includes extracting video feature point data, calculating spatial registration mapping relationships, and fusing real-time environmental monitoring data. Extracting video feature point data uses a corner detection algorithm to identify salient feature points in video frames, such as the edge of the operating table, instrument outlines, and anatomical landmarks on the patient's surface. Optical flow is used to track the temporal sequence of these feature points across consecutive frames.

[0035] Specifically, the generation of mixed reality space mapping data includes: Step 1: Extract video feature point data: Spatial feature points are extracted from the live video stream. Corner detection algorithms are used to identify significant feature points in the video frames, including the edge of the operating table, the outline of instruments, and anatomical landmarks on the patient's body surface. The optical flow method is used to achieve temporal tracking of feature points between consecutive frames to generate video feature point data. Step 2: Calculate the spatial registration mapping relationship: Perform spatial registration calculation between the spatiotemporally calibrated "scene feature matrix" and "video feature point data". The registration formula is as follows: Where n is the number of feature points, The weight coefficient for the i-th feature point (the weight of anatomical landmarks > the weight of instrument feature points). For rotation matrix, These are the spatial coordinates of the feature points (from the scene feature matrix). The translation vector is used; the spatial registration mapping relationship is obtained by minimizing the registration error. Step 3: Integrate real-time environmental monitoring data: Collect real-time monitoring data of the surgical environment, including light intensity, temperature and humidity, and spatial noise; perform feature fusion processing on the "spatial registration mapping relationship" and the "real-time environmental monitoring data" to provide a foundation for subsequent virtual-real registration.

[0036] The spatial registration mapping calculation involves spatially registering the spatiotemporally calibrated "scene feature matrix" with the "video feature point data," minimizing the registration error to obtain the spatial registration mapping relationship. Real-time environmental monitoring data, such as light intensity, temperature, humidity, and spatial noise, is collected and fused with the "spatial registration mapping relationship" for feature fusion processing. These components collaborate sequentially, first performing depth segmentation, then spatiotemporal calibration, and finally generating the mapping data. The combined components of the spatial mapping unit accurately map the virtual model to the real surgical scene, aligning the virtual surgical scene with the actual surgical environment in both space and time, providing an accurate foundation for subsequent interaction and simulation.

[0037] Specifically, the scene fusion unit includes hardware and permission configuration components for both master and slave HMDs. By deploying head-mounted displays, the camera on the master HMD captures video footage within its field of view, updating the display angle of the virtual model on the master HMD in real time. The virtual model displayed on the master HMD automatically switches to a side-view angle to match the student's actual observation position. The head-mounted displays can be common VR or MR headsets, connected to the system wirelessly or via wired connections. Permission configuration allows setting different operation permissions based on different user roles and needs. The components of the scene fusion unit, combined, enable the overlay display of virtual models and live video streams, and adjust the display angle of the virtual model in real time according to the user's observation position, enhancing the user's immersion and interactive experience.

[0038] Specifically, the interactive control module 203 includes components for tissue layering and deformation parameter space construction of virtual surgical scene data, and components for nonlinear transmission characteristic extraction and stress distribution calculation of mechanical parameters in mixed reality interactive data. The component for tissue layering and deformation parameter space construction of virtual surgical scene data assigns core deformation parameters to each layer of tissue based on "tissue parameter distribution data," and constructs a three-dimensional deformation parameter space according to the "spatial coordinates of the layered tissues." That is, any spatial point (x, y, z) in each tissue layer corresponds to a unique set of deformation parameters, forming a "parameter-space" mapping relationship. Here, a mesh-based method can be used to layer the tissues, assigning deformation parameters according to the physical properties and structural characteristics of the tissues. The component for nonlinear transmission characteristic extraction and stress distribution calculation of mechanical parameters in mixed reality interactive data collects the contact force of instrument operation, the interactive pressure between the hand and the instrument, the real-time contact force between the instrument and the virtual tissue, and the direction vector of the force, extracting core mechanical parameters—the magnitude of the force (F), the direction of the force (θ, φ), and the coordinates of the point of application of the force (θ, φ). The force and the rate of change of force (dF / dt) are used to form a "time series of mechanical parameters". Based on the nonlinear mechanical response of the tissue, strain data of the surrounding tissue is collected with the point of force application as the center to establish a force-strain curve. Through curve fitting, the nonlinear transmission coefficient is extracted, which reflects the transmission attenuation law of mechanical parameters in the tissue. At the same time, the original interactive data is preprocessed to remove noise and fill in missing values ​​to ensure the data accuracy required for the surgical scenario. Numerical calculation methods such as the finite element method are used to construct a mechanical model of instrument-tissue contact in a mixed reality scenario. The nonlinear transmission characteristic parameters and stress distribution results are summarized to form a core dataset. These components of the interactive control module are combined to effectively process virtual surgical scenario data and mechanical parameters, providing a quantitative basis for surgical simulation, making the simulation more realistic and accurate, and better reflecting the mechanical changes and tissue deformation during the surgical process.

[0039] Specifically, the application output module 204 includes components for analyzing data and generating reports and early warning information. The data analysis component receives 3D surgical scene analysis data and surgical interaction mechanics data, and processes this data using data analysis algorithms. The report and early warning information generation component generates a standardization assessment report and risk warning information based on the analysis results. The standardization assessment report evaluates and scores various parameters and operations during the surgical process according to the standard requirements of the surgical procedure. The risk warning information issues early warnings based on abnormalities shown in the data, such as excessively large mechanical parameters or unreasonable tissue deformation. The components of the application output module 204, combined, provide valuable assessment and early warning information for surgery, helping doctors better plan surgical procedures and improve the safety and success rate of the surgery. The implementation principle of this embodiment is as follows: This system comprehensively collects multi-source volumetric data of the human body through the data acquisition module, providing a rich and accurate information foundation for subsequent modeling. The modeling module processes this data, reconstructs a digital 3D model containing various tissues, and generates a virtual surgical scene that blends the real and virtual elements, allowing doctors to more intuitively understand the surgical situation. The interactive control module processes scene data and mechanical parameters to provide quantitative data for surgical simulation, making the simulation closer to real surgery. The application output module 204 generates evaluation reports and early warning information based on the processed data, assisting doctors in better planning and executing surgeries. Compared with existing technologies, it can simulate surgical scenarios more comprehensively and accurately, providing more effective support for surgery and improving the precision and safety of the operation.

[0040] Example 2 The difference between this embodiment and the previous embodiment is that the model building unit in the modeling module 202 uses a deep learning-based segmentation algorithm instead of a threshold segmentation algorithm when reconstructing bony structures. The deep learning-based segmentation algorithm can use convolutional neural networks, such as the U-Net network, which learns bone features and boundaries by training on a large amount of CT image data, thereby more accurately segmenting bony structures. The implementation principle of this embodiment is that using a deep learning-based segmentation algorithm can improve the accuracy and efficiency of bony structure reconstruction because deep learning algorithms can automatically learn the complex features of bones, reducing manual intervention and errors. Compared with the traditional threshold segmentation algorithm, it can better adapt to the differences in bone features and CT images of different patients, making the reconstructed bony structure model more accurate, thereby improving the accuracy of the entire virtual surgical scene, providing a more reliable basis for surgical planning and simulation, and further improving the quality and safety of the surgery. The force-tactile editing system with the rapid virtual editing function of mixed reality surgery also includes a data visualization module, used to display the analyzed results in at least one of the following forms: charts, curves, numerical values, and grade labels, through an MR head-mounted display or an external display device.

[0041] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. It will be apparent to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or basic characteristics of the present invention. Therefore, the embodiments should be regarded as exemplary and non-limiting in all respects. The scope of the present invention is defined by the appended claims rather than the foregoing description, and therefore all changes falling within the meaning and scope of the equivalents of the claims are intended to be included within the present invention.

[0042] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A force-haptic editing system with rapid virtual editing capabilities for mixed reality surgery, characterized in that, include: The data acquisition module is used to acquire multi-source volumetric data of the human body, including multi-modal volumetric data from MRI, CT, and ultrasound. The modeling module reconstructs a digital 3D model containing bony structures, soft tissues, blood vessels, and nerves based on multi-source data, and combines it with mixed reality spatial mapping technology to generate a virtual surgical scene that blends the virtual and real worlds. The interactive control module performs tissue layering and deformation parameter space construction on the virtual surgical scene data to obtain three-dimensional surgical scene analysis data; The mechanical parameters in the mixed reality interactive data are extracted for nonlinear transmission characteristics and stress distribution is calculated to form surgical interactive mechanical data. The application output module analyzes 3D surgical scene analysis data and surgical interaction mechanics data to generate a standardization assessment report and risk warning information.

2. The force-tactile editing system with rapid virtual editing function for mixed reality surgery according to claim 1, characterized in that, The modeling module includes: The model building unit acquires multimodal volumetric data through MRI, CT, and ultrasound, and simultaneously acquires spatial information of the surgical environment, on-site video stream, and lesion-specific images; it registers multi-source data with markers, reconstructs an integrated three-dimensional model of "bone structure-soft tissue-blood vessels and nerves", optimizes details, and assigns physical parameters; The spatial mapping unit performs deep segmentation and spatiotemporal calibration of surgical environment data to generate mixed reality spatial mapping data, and completes the registration of the model with the world coordinate system and the operating platform. The scene fusion unit overlays virtual models and live video streams, configures multi-view displays of master and slave HMDs, integrates collision detection and force feedback functions, and finally generates a virtual-real fusion surgical scene with "visual alignment and multi-role collaboration".

3. The force-tactile editing system for rapid virtual editing of mixed reality surgery according to claim 1, characterized in that: The model building unit includes: Bony structure reconstruction: Based on CT registration data, a bony structure model containing bone surface texture and internal pore structure is generated through threshold segmentation and three-dimensional surface rendering; Soft tissue reconstruction: Based on nuclear magnetic resonance registration data, the region growing algorithm is used to segment muscle, fascia and adipose soft tissue, and volume rendering technology is used to restore the hierarchical distribution of soft tissue and generate a three-dimensional soft tissue structure; Vascular and nerve reconstruction: Combining MRI and ultrasound contrast data, the course of vascular and nerve structures is extracted using a vascular tracking algorithm to generate a tubular vascular and nerve model, clarifying its spatial interpenetration relationship with bony structures and soft tissues.

4. The force-tactile editing system for rapid virtual editing of mixed reality surgery according to claim 2, characterized in that: The spatial mapping unit uses a region growing algorithm to segment the preprocessed point cloud data. Using known geometric features in the surgical scene as seed points, it clusters according to depth similarity and spatial continuity to separate key targets such as the operating table, surgical instruments, and patient surface operation area, generating surgical scene depth data. Semantic annotation is performed on each segmented region to clarify the spatial range and functional attributes of each region, providing a target benchmark for subsequent spatiotemporal calibration; high-precision timestamps are added to the surgical scene depth data, surgical environment positioning data, and on-site video stream data respectively. Using the timestamps of spatial positioning sensors as a reference, time series from other data sources are aligned using linear interpolation.

5. The force-tactile editing system for rapid virtual editing of mixed reality surgery according to claim 4, characterized in that: Mixed reality spatial mapping data generation includes: Step 1: Extract video feature point data: Extract spatial feature points from the live video stream, use corner detection algorithm to identify significant feature points in the video frames, including the edge of the operating table, instrument outline, and anatomical landmarks on the patient's body surface, and use optical flow method to achieve temporal tracking of feature points between consecutive frames to generate video feature point data; Step 2: Calculate the spatial registration mapping relationship: Perform spatial registration calculation between the spatiotemporally calibrated "scene feature matrix" and "video feature point data". The registration formula is as follows: Where n is the number of feature points. The weight coefficient for the i-th feature point (anatomical landmark weight > instrument feature point weight). For rotation matrix, These are the spatial coordinates of the feature points (from the scene feature matrix). The translation vector is used; the spatial registration mapping relationship is obtained by minimizing the registration error. Step 3: Integrate real-time environmental monitoring data: Collect real-time monitoring data of the surgical environment, including light intensity, temperature and humidity, and spatial noise; perform feature fusion processing on the "spatial registration mapping relationship" and "real-time environmental monitoring data" to provide a foundation for subsequent virtual-real registration.

6. The force-tactile editing system for rapid virtual editing of mixed reality surgery according to claim 2, characterized in that: The scene fusion unit includes: The hardware and permission configuration of the master and slave HMDs involves deploying head-mounted visual devices and using the camera of the master HMD to capture video footage within its field of view. This allows for real-time updates of the virtual model's display angle on the master HMD, automatically switching the virtual model displayed on the HMD to a side view angle to match the student's actual observation position.

7. The force-tactile editing system for rapid virtual editing of mixed reality surgery according to claim 1, characterized in that: The interactive control module assigns core deformation parameters to each layer of tissue based on "tissue parameter distribution data". It constructs a three-dimensional deformation parameter space according to the "spatial coordinates of the layered tissues". That is, any spatial point (x, y, z) of each tissue layer corresponds to a unique set of deformation parameters, forming a "parameter-space" mapping relationship. Based on the mapping matrix of the three-dimensional deformation parameter space and the parameter statistics of each layer of tissue, it provides a quantitative basis for subsequent surgical procedure simulation.

8. The force-tactile editing system for rapid virtual editing of mixed reality surgery according to claim 7, characterized in that: The interactive control module extracts the nonlinear transmission characteristics and calculates the stress distribution of the mechanical parameters in the mixed reality interactive data, forming surgical interactive mechanical data acquisition of the contact force of the instrument operation, the interactive pressure between the hand and the instrument, the real-time contact force between the instrument and the virtual tissue, and the direction vector of the force. It then extracts the core mechanical parameters—the magnitude of the force (F), the direction of the force (θ, φ), and the coordinates of the point of application of the force (θ, φ). The rate of change of force (dF / dt) forms a "mechanical parameter time sequence". Based on the nonlinear mechanical response of the tissue, strain data of the surrounding tissue is collected with the point of application of force as the center, and a force-strain curve is established. By curve fitting, the nonlinear transmission coefficient is extracted, which reflects the attenuation law of mechanical parameters in the tissue.

9. The force-tactile editing system for rapid virtual editing of mixed reality surgery according to claim 8, characterized in that: The biofeedback system also includes a data visualization module, which displays the analyzed results in at least one of the following forms: charts, curves, numerical values, and grade labels, via an MR head-mounted display or an external display device.

10. The force-tactile editing system for rapid virtual editing of mixed reality surgery according to claim 1, characterized in that: The process of extracting nonlinear transmission characteristics and calculating stress distribution from the mechanical parameters in the mixed reality interactive data to form surgical interactive mechanical data includes: preprocessing the original interactive data, removing noise, filling in missing values ​​to ensure the data accuracy required for the surgical scenario, using numerical calculation methods such as the finite element method to construct a mechanical model of instrument-tissue contact in the mixed reality scenario, and summarizing the nonlinear transmission characteristic parameters and stress distribution results to form a core dataset.

Images