Virtual operation analogue simulation method and system based on mixed reality technology

Abstract

The invention relates to the technical field of image processing, and discloses a virtual operation analogue simulation method and system based on a mixed reality technology. The method comprises the steps that three-dimensional space information is collected, feature extraction and registration are conducted on a real operation environment and video data, and mixed reality mapping and monitoring data are generated; performing medical image registration processing based on mixed reality mapping to obtain virtual scene data; processing instrument sensing data according to the mapping data and the monitoring parameters to obtain interaction data; performing surgical process simulation according to the virtual scene and the interaction data; performing operation trajectory analysis and identification based on the simulation data, and generating evaluation and early warning information; and analyzing the evaluation and early warning information, and outputting an analysis report. Through multi-source data fusion and a real-time evaluation feedback mechanism, the authenticity and accuracy of operation simulation training are effectively improved.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a virtual surgery simulation method and system based on mixed reality technology. Background Art

[0002] With the continuous development of medical technology, surgical simulation training plays an increasingly important role in medical education and surgical skills training. Traditional surgical simulation training mainly relies on physical models and virtual reality technology. Although physical models can provide real tactile feedback, they are expensive and lack repeatability. Although virtual reality technology has good repeatability and customizability, it has obvious deficiencies in realism and interactivity. In recent years, the emergence of mixed reality technology has provided a new technical path for surgical simulation training. By integrating the real surgical environment with the virtual surgical scene, it can provide a rich virtual interactive experience while ensuring the realism of training. However, the current mixed reality surgical simulation system still faces many technical challenges in terms of real-time acquisition of the surgical environment, precise registration of virtual and real scenes, dynamic tracking of surgical instruments, and evaluation and feedback of the surgical process.

[0003] The existing mixed reality surgical simulation system has the following shortcomings: First, in terms of environmental perception, it is difficult to accurately obtain the dynamic change information of the surgical space, resulting in insufficient accuracy in virtual and real scene registration; second, in terms of surgical process simulation, the existing system does not accurately simulate the physical characteristics of tissue deformation and cannot truly reflect the tissue response during the surgical operation; third, in terms of surgical evaluation, there is a lack of accurate analysis and real-time evaluation mechanism for surgical operation trajectories, making it difficult to provide timely and effective feedback guidance for trainees. These technical problems have seriously restricted the practical effect of the mixed reality surgical simulation system. Summary of the invention

[0004] The present application provides a virtual surgery simulation method and system based on mixed reality technology, which is used to solve the problem of how to achieve high-precision virtual-real scene registration and real-time surgical process evaluation in the mixed reality surgical simulation process, and effectively improve the authenticity and accuracy of surgical simulation training through multi-source data fusion and real-time evaluation feedback mechanism.

[0005] In a first aspect, the present application provides a virtual surgery simulation method based on mixed reality technology, and the virtual surgery simulation method based on mixed reality technology includes: based on the collected three-dimensional spatial information, performing spatial feature extraction and alignment and calibration processing on the real surgical environment data and the on-site video data, and generating mixed reality space mapping data and environmental real-time monitoring data; according to the mixed reality space mapping data, performing virtual-real alignment and physical property mapping processing on the patient's medical image, and obtaining virtual surgical scene data and real-time correction parameters; based on the mixed reality space mapping data, the real-time environmental monitoring data and the real-time correction parameters, performing time series fusion processing on the multi-source sensor data of the surgical instrument, and obtaining mixed reality interaction data; according to the virtual surgical scene data and the mixed reality interaction data, performing surgical process simulation through tissue deformation simulation calculation, and obtaining surgical simulation data and process monitoring data; based on the surgical simulation data, the process monitoring data and the real-time environmental monitoring data, performing feature analysis and pattern recognition processing on the surgical operation trajectory, and generating surgical evaluation data and real-time warning information; performing operation suggestion analysis on the surgical evaluation data and the real-time warning information, and obtaining a surgical simulation process analysis report and target suggestion data.

[0006] In a second aspect, the present application provides a virtual surgery simulation system based on mixed reality technology, and the virtual surgery simulation system based on mixed reality technology includes: The acquisition module is used to extract spatial features and perform registration and calibration processing on the real surgical environment data and on-site video data based on the acquired three-dimensional spatial information, and generate mixed reality space mapping data and real-time environmental monitoring data; A mapping module, used to perform virtual-real registration and physical property mapping processing on the patient's medical image according to the mixed reality space mapping data, so as to obtain virtual surgical scene data and real-time correction parameters; A fusion module, configured to perform time-series fusion processing on multi-source sensor data of the surgical instrument according to the mixed reality space mapping data, the real-time environmental monitoring data and the real-time correction parameters, so as to obtain mixed reality interaction data; A simulation module, used to simulate the surgical process through tissue deformation simulation calculation according to the virtual surgical scene data and the mixed reality interaction data, so as to obtain surgical simulation data and process monitoring data; An identification module, configured to perform feature analysis and pattern recognition processing on the surgical operation trajectory based on the surgical simulation data, the process monitoring data and the environmental real-time monitoring data, and generate surgical evaluation data and real-time warning information; The analysis module is used to perform operation suggestion analysis on the surgery evaluation data and the real-time warning information to obtain a surgery simulation process analysis report and target suggestion data.

[0007] In the technical solution provided by this application, the accurate reconstruction of the surgical scene is achieved by extracting spatial features and aligning and calibrating the real surgical environment data and the on-site video data, which improves the realism and immersion of the mixed reality environment. The virtual surgical scene is closer to the real surgical environment through virtual-real registration and physical property mapping, which enhances the authenticity of the simulation training. In terms of surgical instrument interaction, the time-series fusion processing of multi-source sensor data is adopted to improve the accuracy and stability of surgical instrument position tracking. At the same time, through tissue deformation simulation calculation, a realistic surgical process simulation effect is achieved. In the surgical operation evaluation link, the surgical operation trajectory is subjected to feature analysis and pattern recognition processing, and an objective and comprehensive evaluation system is established. Through the real-time early warning mechanism, potential risks are discovered in time, which improves the safety of surgical training. Finally, through the in-depth analysis of surgical evaluation data and early warning information, a detailed analysis report and targeted improvement suggestions are generated, which improves the effect and efficiency of surgical training. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0009] Figure 1 A schematic diagram of an embodiment of a virtual surgery simulation method based on mixed reality technology in an embodiment of the present application; Figure 2 This is a schematic diagram of an embodiment of a virtual surgery simulation system based on mixed reality technology in an embodiment of the present application. DETAILED DESCRIPTION

[0010] The embodiment of the present application provides a virtual surgery simulation method and system based on mixed reality technology. The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described here can be implemented in an order other than that illustrated or described here. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0011] For ease of understanding, the specific process of the embodiment of the present application is described below. Figure 1 In the embodiment of the present application, an embodiment of a virtual surgery simulation method based on mixed reality technology includes: Step S101: Based on the collected three-dimensional spatial information, spatial feature extraction and registration and calibration processing are performed on the real surgical environment data and the on-site video data to generate mixed reality space mapping data and real-time environmental monitoring data; Step S102: performing virtual-real registration and physical property mapping processing on the patient's medical image according to the mixed reality space mapping data to obtain virtual surgical scene data and real-time correction parameters; Step S103: performing time series fusion processing on multi-source sensor data of the surgical instrument according to the mixed reality space mapping data, the real-time environmental monitoring data and the real-time correction parameters to obtain mixed reality interaction data; Step S104: simulating the surgical process through tissue deformation simulation calculation according to the virtual surgical scene data and the mixed reality interaction data, to obtain surgical simulation data and process monitoring data; Step S105: Based on the surgery simulation data, process monitoring data and environmental real-time monitoring data, feature analysis and pattern recognition processing are performed on the surgery operation trajectory to generate surgery evaluation data and real-time warning information; Step S106: Perform operation suggestion analysis on the surgical evaluation data and the real-time warning information to obtain a surgical simulation process analysis report and target suggestion data.

[0012] It is understandable that the execution subject of the present application can be a virtual surgery simulation system based on mixed reality technology, or a terminal or a server, which is not limited here. The present application embodiment is described by taking the server as the execution subject as an example.

[0013] Specifically, three-dimensional spatial information is collected to process the real surgical environment data and on-site video data. The surgical space image data is collected by three-dimensional camera, and the depth feature segmentation is performed to obtain the depth data of the surgical scene. In this process, the point cloud data collected by the depth camera is subjected to noise reduction and registration processing to ensure the accurate extraction of scene features. For the collected video data, the key feature points are extracted by feature point detection technology. These feature points include spatial reference information such as the edge of the operating table, the position of the instrument, and the anatomical landmarks. By performing time-series tracking and spatial registration on these feature points, a stable mixed reality space mapping relationship is established. In the virtual-real registration link, the patient's medical imaging data is accurately aligned with the mixed reality space. Medical imaging data includes three-dimensional imaging data such as CT and MRI. First, multi-scale feature extraction is performed on these data to identify tissue boundaries and anatomical structures. The extracted features are registered with the reference markers in the actual surgical environment through the feature matching algorithm, and the spatial transformation matrix is ​​calculated. On this basis, the corresponding physical property parameters, including elastic modulus, density, etc., are assigned to different tissue structures to construct a virtual surgical scene. At the same time, according to the ambient lighting conditions and changes in viewing angle, the rendering parameters of the virtual scene are dynamically adjusted to ensure visual realism.

[0014] The real-time tracking of surgical instruments adopts a multi-source sensor data fusion method. The position and posture data of the instruments are collected by optical markers and inertial sensors arranged on the surgical instruments. These data are aligned with the mixed reality space mapping data in time and space to calculate the precise position of the instrument in the virtual scene. The tracking results are dynamically corrected according to the real-time monitoring data of the environment to eliminate the interference caused by environmental factors. The real-time correction parameters are used to compensate for the accumulated errors in the tracking process to ensure the tracking accuracy during long-term operations. In the simulation stage of the surgical process, the tissue deformation simulation calculation is performed based on the acquired virtual surgical scene data and mixed reality interaction data. By establishing a mechanical model of the tissue, the tissue deformation response during the surgical operation is simulated. When the surgical instrument comes into contact with the virtual tissue, the deformation distribution is calculated based on the contact force and the physical properties of the tissue. This process needs to consider the nonlinear characteristics and boundary constraints of the tissue to ensure the accuracy of the deformation calculation. At the same time, the key data of the entire surgical process are recorded, including deformation, stress distribution, operation trajectory, etc.

[0015] During the surgical evaluation phase, the collected surgical data is analyzed in depth. The surgical operation trajectory is analyzed through feature extraction algorithms to identify key operation steps and surgical skills. The extracted features are compared with standard surgical specifications to evaluate the standardization and proficiency of the operation. Combined with environmental monitoring data, potential risks during the operation are warned. This includes real-time monitoring of parameters such as instrument position, operation force, and tissue stress, and warning information is generated when the parameters exceed the safety threshold. Finally, a detailed analysis report is generated based on the surgical evaluation data and real-time warning information. The report content includes surgical operation scores, skill analysis, risk warnings, and improvement suggestions. Through the quantitative analysis of operation data, an objective evaluation basis is provided for surgical training. The suggestions point out the operation details and precautions that need to be improved in a targeted manner to help improve surgical skills.

[0016] For example, when performing intracranial tumor resection in neurosurgery simulation training, mixed reality space mapping data shows the three-dimensional spatial position of the surgical approach, and the virtual surgical scene data contains the precise anatomical structure and physical property parameters of the patient's brain tissue. When using surgical instruments for operation, mixed reality interactive data records the movement trajectory and operation force of the instrument, and tissue deformation simulation calculation displays the deformation response of brain tissue in real time. The surgical evaluation system analyzes the deformation amount and boundary stress of brain tissue during the operation, and issues a warning in time when it is close to important neurovascular structures. The analysis report points out the stability of instrument control and the appropriateness of tissue traction during the operation, and provides specific suggestions for improvement.

[0017] In the embodiment of the present application, the accurate reconstruction of the surgical scene is achieved by extracting spatial features and aligning and calibrating the real surgical environment data and the on-site video data, which improves the realism and immersion of the mixed reality environment. The virtual surgical scene is closer to the real surgical environment through virtual-real registration and physical property mapping, which enhances the authenticity of the simulation training. In terms of surgical instrument interaction, the time-series fusion processing of multi-source sensor data is adopted to improve the accuracy and stability of surgical instrument position tracking. At the same time, through tissue deformation simulation calculation, a realistic surgical process simulation effect is achieved. In the surgical operation evaluation link, the surgical operation trajectory is subjected to feature analysis and pattern recognition processing, and an objective and comprehensive evaluation system is established. Through the real-time early warning mechanism, potential risks are discovered in time, which improves the safety of surgical training. Finally, through the in-depth analysis of surgical evaluation data and early warning information, a detailed analysis report and targeted improvement suggestions are generated, which improves the effect and efficiency of surgical training.

[0018] In a specific embodiment, the process of executing step S101 may specifically include the following steps: (1) Collect surgical space image data, perform depth feature segmentation on the surgical space image data, and obtain surgical scene depth data; (2) The surgical environment positioning data is collected through the spatial positioning sensor, and the surgical scene depth data and the surgical environment positioning data are calibrated in the time domain to obtain the scene feature matrix; (3) Obtaining a real-time video stream of the surgical scene according to the video acquisition unit, extracting spatial feature points from the video stream, and obtaining video feature point data; (4) Perform spatial registration calculation on the scene feature matrix and the video feature point data to obtain a spatial registration mapping relationship; (5) Perform real-time monitoring and calculation of surgical environment parameters based on the spatial registration mapping relationship to obtain real-time environmental monitoring data; (6) Perform feature fusion processing on the spatial registration mapping relationship and the real-time environmental monitoring data to generate mixed reality spatial mapping data and real-time environmental monitoring data.

[0019] Specifically, the surgical space image data is collected, and the three-dimensional point cloud information of the surgical scene is collected by the depth camera. Each point contains the spatial coordinates and depth value. The collected point cloud data is segmented by depth features, and the point cloud is clustered using the regional growing algorithm to identify the spatial distribution of key objects such as the operating table, instruments, and markers to form the depth data of the surgical scene. In the process of deep feature segmentation, adjacent points are classified into the same area according to the spatial continuity and depth similarity of the point cloud, and a hierarchical representation of the scene is established. In the process of the spatial positioning sensor collecting the surgical environment positioning data, infrared positioning markers and optical tracking devices are used to obtain the real-time three-dimensional coordinates of the key reference points in the surgical scene. The surgical scene depth data and the surgical environment positioning data are calibrated for time domain features, the two data streams are aligned by timestamps, and a unified time reference system is established. On this basis, a scene feature matrix is ​​constructed, which contains the correspondence between the spatial position, depth value, and positioning reference point.

[0020] The video acquisition device obtains the real-time video stream of the surgical scene and extracts spatial feature points from the video data. The feature point extraction process uses a corner detection algorithm to identify significant features in the video frame, such as the surgical area boundary, instrument outline, anatomical markers, etc. The extracted feature points are tracked in time sequence to ensure the correspondence between feature points in consecutive frames and obtain video feature point data.

[0021] Spatial registration calculation is the process of accurately aligning the scene feature matrix with the video feature point data. The mathematical model of spatial registration uses the following formula: in, represents the registration mapping matrix, n is the number of feature points, is the weight coefficient of the i-th feature point, is the rotation matrix, is the spatial coordinate of the feature point, is the translation vector. The optimal spatial registration mapping relationship is obtained by minimizing the registration error. The acquisition of real-time environmental monitoring data covers multiple aspects: light intensity monitoring uses photosensors to collect ambient light values ​​for adjusting the rendering parameters of virtual objects; temperature and humidity monitoring uses temperature and humidity sensors to collect environmental parameters for compensating sensor drift; spatial noise monitoring evaluates the intensity of environmental interference by analyzing the jitter degree of point cloud data. These monitoring data are feature fused with the spatial registration mapping relationship to establish a correlation model between environmental parameters and spatial positioning accuracy.

[0022] Feature fusion processing uses a multi-level data fusion method to integrate the spatial registration mapping relationship with the real-time environmental monitoring data. At the feature level, the key features of spatial position and environmental parameters are extracted; at the decision level, the spatial positioning results are dynamically corrected according to the environmental parameters. The fused data forms mixed reality space mapping data, which includes spatial coordinate transformation relationships, environmental parameter distributions, and correction parameters.

[0023] For example, the depth camera collects point cloud data of the operating table to identify the entry point of the surgical instrument and the position of the laparoscope. The spatial positioning sensor tracks the markers on the surgical instrument to obtain the real-time position of the instrument. The video acquisition device records the surgical field of view and extracts the anatomical feature points in the laparoscope image. Environmental monitoring data shows that the light intensity in the operating room is 500 lux, the temperature is 24°C, and the humidity is 50%. Through spatial registration calculation, the depth data, positioning data, and video feature points are aligned to the same coordinate system to achieve accurate fusion of the virtual surgical scene and the real environment. When the ambient light changes, the system adjusts the rendering parameters of the virtual object according to the change in light intensity; when the temperature change causes sensor drift, the spatial positioning data is corrected according to the temperature compensation model. This real-time fusion and dynamic correction of multi-source data ensures the stability and accuracy of the mixed reality surgical simulation system.

[0024] In a specific embodiment, the process of executing step S102 may specifically include the following steps: (1) Through feature analysis and calculation, the tissue interface information in the medical image data is processed at multiple scales to obtain anatomical feature distribution data; (2) Using the mixed reality space mapping data as a reference, the anatomical feature distribution data is subjected to pose estimation and error compensation to obtain initial registration data; (3) Perform key point screening and spatial feature matching operations on the initial registration data, iteratively optimize the spatial correspondence, and obtain virtual-real fusion mapping data; (4) Perform physical parameter calculation and hierarchical analysis on different tissue regions in the virtual-reality fusion mapping data to obtain tissue parameter distribution data; (5) Dynamically correct the deformation response in the virtual environment based on the tissue parameter distribution data to obtain real-time correction parameters; (6) The virtual-reality fusion mapping data and tissue parameter distribution data are fused and superimposed according to the topological structure to generate virtual surgical scene data and real-time correction parameters.

[0025] Specifically, medical image data is processed by multi-scale analysis methods, and wavelet transform is used to perform multi-layer decomposition of images to extract tissue boundary information at different resolutions. At each layer of the image pyramid, the tissue interface is identified by edge detection algorithms to form a multi-level feature description. These features include the boundary contours, internal structures, and spatial position relationships of the tissue, forming anatomical feature distribution data.

[0026] The pose estimation and error compensation process uses the following mathematical model: in, represents the pose estimation error, m is the number of feature points, is the weight factor of the kth feature point, is the homography matrix, are the feature point coordinates, is the position offset, is the error compensation coefficient, n is the number of compensation items, is the compensation weight, is the error term. By minimizing Obtain the optimal initial registration data.

[0027] In the process of key point screening and spatial feature matching, the importance of feature points is evaluated for the initial registration data, and points with significant features are selected as matching benchmarks. The iterative closest point algorithm is used to register the feature points, and the spatial transformation parameters are continuously optimized until the convergence conditions are reached to obtain virtual-real fusion mapping data.

[0028] The following model is used to calculate the physical parameters of different tissue regions: in, is the tissue physical parameter, p is the number of tissue regions, is the regional weight coefficient, is the basic physical parameter matrix, is the correction factor, is the level adjustment factor, q is the number of levels, is the level weight, is the hierarchical parameter. The distribution data of tissue parameters is calculated through this model.

[0029] Based on the tissue parameter distribution data, the deformation response in the virtual environment is dynamically corrected to establish the corresponding relationship between tissue deformation and external force. By calculating the deviation between the tissue deformation variable and the theoretical prediction value in real time, real-time correction parameters are generated to adjust the mechanical response of the virtual tissue. Finally, the virtual-real fusion mapping data and the tissue parameter distribution data are fused and superimposed, and the corresponding relationship between the virtual and real scenes is established based on the topological structure to generate virtual surgical scene data containing geometric information and physical characteristics.

[0030] For example, in liver surgery simulation, the patient's CT image data is first processed, and anatomical features such as liver contour, blood vessel direction and tumor location are identified through multi-scale analysis. These features are estimated with the actual surgical area in the mixed reality space, and the spatial correspondence between the virtual liver model and the real surgical environment is established. By analyzing the relationship between CT value and tissue density, the physical parameters of different areas such as liver parenchyma, blood vessels and tumors are calculated. When the surgeon operates in the virtual environment, the tissue deformation response is calculated in real time according to the contact between the instrument and the tissue, and dynamic correction is performed according to the deformation characteristics in the actual surgical environment. During the whole process, the physical parameters and deformation response of the tissue are dynamically updated to ensure the consistency of the virtual surgical scene with the real surgical environment. For example, when the surgical instrument applies pressure on the liver surface, by measuring the difference between the deformation of the real tissue and the deformation of the virtual model, the elastic modulus and other parameters are dynamically adjusted to make the deformation behavior of the virtual tissue closer to the real situation.

[0031] In a specific embodiment, the process of executing step S103 may specifically include the following steps: (1) Extract the spatiotemporal distribution and perform feature analysis on the multi-source sensor data of surgical instruments to obtain the initial state data of the instruments; (2) Perform spatial transformation processing on the mixed reality space mapping data and the real-time environmental monitoring data, and obtain the device space coordinate data through projection mapping; (3) Calibrate and synchronize the spatial coordinate data of the instrument based on the real-time correction parameters to obtain the corrected instrument trajectory data; (4) Performing motion feature analysis and interaction state recognition processing on the corrected instrument trajectory data to obtain instrument interaction feature data; (5) Calculate constraints and perform spatial detection on the device interaction feature data to obtain device interaction response data; (6) The device interaction feature data and the device interaction response data are fused in the time domain and the space domain to generate mixed reality interaction data.

[0032] Specifically, the spatiotemporal distribution of data from different sensors is extracted. These data include the spatial position information of optical markers, the posture data of the inertial measurement unit, and the contact force data of the force sensor. Through feature analysis and processing, the characteristic parameters such as the position, posture, and movement speed of the instrument are extracted, and the six-degree-of-freedom state description of the instrument is constructed to form the initial state data of the instrument. In the spatial transformation processing stage, the mixed reality space mapping data is used as the reference coordinate system, and the environmental parameters such as lighting conditions, temperature and humidity in the real-time environmental monitoring data are combined to perform a unified transformation of the coordinate system. The position information of the instrument in different coordinate systems is converted into a unified reference coordinate system through the projection mapping method to obtain the instrument space coordinate data, which accurately describes the spatial position and posture information of the instrument in the mixed reality environment.

[0033] In the calibration and synchronization processing stage, the spatial coordinate data of the instrument is dynamically corrected based on the real-time correction parameters. The real-time correction parameters include the system error caused by environmental factors, the sensor drift error, and the error compensation value caused by the timing asynchrony. By applying these correction parameters to the original coordinate data, the accurate calibration of the instrument position is achieved, while ensuring the timing synchronization between different data sources, and finally the corrected instrument trajectory data is obtained. In the motion feature analysis and interaction state recognition processing stage, the corrected instrument trajectory data is deeply analyzed. First, the kinematic characteristics of the instrument are extracted, including parameters such as position change, posture change, speed and acceleration. At the same time, the interaction state between the instrument and the virtual tissue is identified to determine whether contact, puncture, cutting and other operations occur, so as to obtain the instrument interaction feature data. The instrument interaction feature data contains information such as motion trajectory characteristics, interaction type identification, contact force magnitude and direction.

[0034] Constraint calculation and spatial detection processing are the processes of further analyzing the instrument interaction feature data. Based on the safety boundary of the surgical operation, avoidance area and other constraints, the legality of the instrument movement is calculated. Through the spatial collision detection algorithm, the positional relationship between the instrument and the virtual tissue is judged to predict the potential collision risk. These processing results form the instrument interaction response data, which includes constraint violation signs, collision warning information, force feedback parameters, etc. The final time domain fusion and space domain fusion processing systematically integrate the instrument interaction feature data and the instrument interaction response data. Time domain fusion ensures the continuity and consistency of data at different time scales, and space domain fusion ensures the accurate correspondence of spatial position and posture information. The fused mixed reality interaction data contains a complete description of the interaction process, including the movement trajectory of the instrument, interaction status, force feedback information and safety warning data.

[0035] For example, in laparoscopic surgery simulation training, the optical markers installed on the surgical instruments collect position data through infrared cameras, while the inertial measurement unit records the posture changes of the instruments, and the force sensor collects the contact force between the instruments and the tissues. These multi-source data are processed to first obtain the position and posture information of the instruments, and then these information are converted into the unified coordinate system of the mixed reality environment through the space transformation matrix. During the calibration process, the sensor drift caused by the temperature change in the operating room is taken into account and compensated by real-time correction parameters. When the doctor uses the instrument to perform tissue separation operations, the operation type is analyzed based on the instrument trajectory data, and the required force feedback size is calculated in combination with the physical properties of the virtual tissue. At the same time, it detects whether the instrument is close to dangerous areas such as important blood vessels and generates corresponding warning information. The final generated mixed reality interaction data not only reflects the precise position and operation status of the instrument, but also contains force feedback control signals and safety warning information.

[0036] In a specific embodiment, the process of executing step S104 may specifically include the following steps: (1) Organize and layer virtual surgical scene data and construct deformation parameter space to obtain three-dimensional surgical scene analysis data; (2) Extract nonlinear transfer characteristics and calculate stress distribution of mechanical parameters in mixed reality interaction data to form surgical interaction mechanical data; (3) Perform surgical cutting path planning and tissue deformation compensation calculation based on 3D surgical scene analysis data and surgical interaction mechanics data, and output surgical operation prediction data; (4) Perform tissue interface deformation analysis on the surgical operation prediction data through rigid body transformation and elastic deformation coupling calculation to obtain tissue interface contact data; (5) Perform multi-level physical deformation and stress propagation analysis on tissue interface contact data to generate surgical process deformation data; (6) Perform state fusion and process recording processing on the surgical deformation data and tissue interface contact data to generate surgical simulation data and process monitoring data.

[0037] Specifically, different types of tissues are layered according to anatomical structures, including epidermis, fat layer, muscle layer, and internal organs. Corresponding physical parameters, such as elastic modulus, density, Poisson's ratio, etc., are set for each layer of tissue to construct a deformation parameter space. The combination of these parameters forms three-dimensional surgical scene analysis data, providing a basis for subsequent deformation calculations.

[0038] The mechanical parameter processing in mixed reality interaction data adopts the following nonlinear stress distribution calculation model: in, is the stress distribution, is the number of force application points, is the nonlinear transfer coefficient, is the magnitude of the force, is the direction of action, is the material characteristic factor, is the number of stress propagation layers, is the interlayer transfer coefficient, is the interlaminar stress attenuation factor. The surgical interaction mechanical data is obtained through this calculation.

[0039] In surgical cutting path planning, tissue deformation compensation calculation is performed by combining 3D surgical scene analysis data and surgical interaction mechanics data: in, is the deformation compensation amount, is the number of compensation points, is the compensation weight, is the initial deformation, is the compensation coefficient, is the global compensation factor, s is the number of compensation layers, is the level weight, is the level compensation amount.

[0040] The following calculation model is used for the analysis of tissue interface deformation: in, is the interface deformation, d is the number of interface points, is the interface weight, is the rigid body deformation component, is the elastic deformation component, is the coupling coefficient, g is the number of interface layers, is the interlayer coupling coefficient, is the interlayer deformation.

[0041] The tissue interface contact data is subjected to multi-level physical deformation analysis, the mechanical conduction characteristics between different levels of tissues are considered, and the propagation law of stress within the tissue is calculated. This process forms complete deformation data of the surgical process, including the instantaneous deformation state of the tissue, internal stress distribution, and deformation propagation process. The final state fusion integrates the surgical process deformation data and tissue interface contact data to generate surgical simulation data and process monitoring data. These data contain a complete record of the surgical operation process, providing a basis for surgical simulation evaluation.

[0042] For example, in liver surgery simulation, liver tissue is layered, including the capsule layer, the parenchyma layer, and the vascular layer. Through nonlinear stress distribution calculation, the stress propagation characteristics of the surgical instrument when applying pressure on the liver surface are analyzed. When the instrument moves along the preset cutting path, the cutting path is adjusted in real time according to the tissue deformation compensation calculation. During the cutting process, the deformation state of the tissue separation surface is calculated through interface deformation analysis, and the tissue stress distribution and deformation amount during the cutting process are recorded. These data are finally integrated into a complete surgical simulation record, which is used to evaluate the accuracy and safety of the surgical operation. For example, when the surgical instrument contacts the liver surface, the stress distribution at the contact point is first calculated, and then the propagation of stress inside the tissue is analyzed. Finally, the tissue deformation effect is generated according to the deformation response characteristics of tissues at different levels.

[0043] In a specific embodiment, the process of executing step S105 may specifically include the following steps: (1) Extract trajectory features and segment surgical operations from surgical simulation data to separate spatial motion feature data; (2) Perform node analysis and behavioral feature recognition on the process monitoring data according to the key surgical steps to form surgical step sequence data; (3) The real-time environmental monitoring data is used to limit the operating space and dynamically track the boundaries of the surgical area to obtain the surgical area constraint data; (4) Compare and analyze the spatial motion feature data with the standard operation sequence through spatial trajectory similarity calculation to obtain operation standardization evaluation data; (5) Based on the surgical step sequence data and surgical area constraint data, safety boundary determination and dangerous area warning calculation are performed to form surgical risk prediction data; (6) Comprehensive feature analysis and risk level determination are performed based on the operation standardization assessment data and surgical risk prediction data to generate surgical assessment data and real-time warning information.

[0044] Specifically, trajectory feature extraction and surgical operation segmentation are performed. Trajectory feature extraction contains information such as the position, posture, speed and acceleration of surgical instruments in three-dimensional space. The continuous surgical operation trajectory is segmented according to the operation characteristics, and the characteristics of different operation stages are identified. During the segmentation process, the starting point, key turning point and end point of the operation are marked by analyzing the continuity and characteristic changes of the instrument movement, so as to obtain spatial motion feature data. The processing of process monitoring data focuses on the identification and analysis of surgical steps. For each key step in the surgical process, its characteristic parameters are extracted, including operation time, operation force, instrument movement range, etc. By analyzing these characteristic parameters, the key nodes in the standard surgical process are identified, such as surgical approach establishment, tissue separation, suturing and other steps. These nodes are organized according to the time sequence relationship to form a complete surgical step sequence data.

[0045] The processing of real-time environmental monitoring data first determines the spatial scope of the surgical operation. According to the type of surgery and the characteristics of the anatomical structure, the boundaries of the safe operation area are delineated. During the operation, the relative position relationship between the surgical instruments and the key anatomical structures is dynamically tracked, and the boundary constraints of the operation area are updated in real time. This information constitutes the surgical area constraint data, which is used to ensure the safety of the surgical operation. The spatial trajectory similarity calculation method is used to compare and analyze the spatial motion feature data with the standard operation sequence. The degree of standardization of the surgical operation is evaluated by calculating the similarity between the actual operation trajectory and the standard trajectory. The similarity calculation takes into account multiple dimensions such as the spatial shape of the trajectory, the movement speed, and the smoothness of the operation, and comprehensively forms the operation standardization evaluation data.

[0046] In the safety boundary determination and dangerous area warning calculation, the surgical step sequence data and surgical area constraint data are combined to analyze the risk level of the surgical operation. The potential risk level is evaluated by calculating the shortest distance between the instrument and the important anatomical structure. When the instrument approaches the dangerous area or the operation trajectory deviates from the safe path, the corresponding level of warning signal is triggered. This information constitutes the surgical risk prediction data.

[0047] Integrate the operation standardization assessment data and surgical risk prediction data. By establishing an evaluation index system, conduct a comprehensive assessment of surgical operations from the dimensions of operation proficiency, safety, efficiency, etc. At the same time, based on the results of the risk level determination, generate corresponding early warning information to provide timely feedback and guidance for surgical training.

[0048] For example, in the simulation of ventriculoscopy surgery, the instrument trajectory in the surgical simulation data is first analyzed to identify key operation stages such as entering the skull, passing through brain tissue, and entering the ventricle. Through process monitoring data analysis, standard operating steps for each stage are determined, such as determining the entry point, controlling the entry angle, adjusting the working channel, etc. Environmental monitoring data shows the real-time status of the surgical area, including the distance from important blood vessels and nerves. When analyzing the doctor's operation trajectory, it is compared with the standard operation sequence to evaluate whether parameters such as the entry angle and entry depth meet the specifications. If it is found that the instrument deviates from the preset path or approaches an important structure, the risk level is immediately calculated and an early warning is issued.

[0049] In a specific embodiment, the process of executing step S106 may specifically include the following steps: (1) Quantify and analyze the correlation of surgical evaluation data to generate evaluation indicator matrix data; (2) Real-time warning information is processed through risk level classification and risk factor identification to obtain warning analysis data; (3) Mapping and transforming the evaluation index matrix data according to the standard data in the expert knowledge base to form initial recommended data; (4) Classify and process early warning analysis data and generate countermeasures through risk prevention and control strategies, and output risk response data; (5) Perform multi-layer optimization calculations and solution integration on the initial recommendation data and risk response data to obtain target recommendation data; (6) Integrate and analyze the evaluation indicator matrix data, early warning analysis data, and target recommendation data according to the evaluation specifications to generate a surgical simulation process analysis report and target recommendation data.

[0050] Specifically, the sub-indicators are quantified and analyzed for correlation. The sub-indicators include dimensions such as surgical operation accuracy, timing rhythm, movement fluency, instrument control ability, and tissue protection awareness. Accurate quantitative standards are used for each indicator. For example, operation accuracy is quantified through parameters such as spatial position error and angle deviation; timing rhythm is evaluated through the time allocation ratio and transition connection of each surgical step; and movement fluency is measured based on the continuity and volatility of the speed curve. These quantitative indicators are organized into evaluation indicator matrix data, and each element in the matrix represents the evaluation score of a specific dimension. The processing of real-time warning information involves multi-level risk level division. First, the warning information is classified, including anatomical structure damage risk, instrument operation risk, and surgical step disorder risk. According to the urgency and potential harm of the risk, the risk level is divided into three levels: mild warning, moderate warning, and severe warning. For each warning event, in-depth risk factor identification is carried out, and the direct causes and potential factors leading to the risk are analyzed to form structured warning analysis data.

[0051] In the application of the expert knowledge base, standard surgical data is organized in the form of knowledge graphs. The evaluation indicator matrix data is mapped and compared with the expert experience data to identify key issues and improvement space in the evaluation results. Through the knowledge reasoning engine, the expert experience is converted into specific improvement suggestions, including operation skill optimization, surgical path adjustment, instrument use points, etc., to form initial recommendation data. The formulation of risk prevention and control strategies adopts a hierarchical and graded processing method. Design corresponding prevention and control measures for different types of risks identified in the early warning analysis data. For tissue damage risks, formulate precise safety boundaries and buffer areas; for instrument operation risks, design detailed operation specifications and correction plans; for step disorder risks, establish a complete process correction mechanism. These strategies are systematically organized to form risk response data.

[0052] The integration and optimization of initial recommendation data and risk response data is a multi-level process. First, at the strategic level, the feasibility and effectiveness of the recommendations are evaluated; second, at the tactical level, the specific implementation steps are refined; and finally, at the operational level, a detailed implementation plan is formulated. Through multi-layer optimization calculations, the best improvement plans are screened out and these plans are integrated to form target recommendation data.

[0053] The evaluation indicator matrix data, early warning analysis data and target recommendation data are systematically integrated according to the unified evaluation specifications. The content structure of the analysis report includes: overall evaluation results, detailed indicator analysis, risk early warning summary, improvement suggestion list, follow-up training plan, etc. Each part has clear data support and specific improvement measures.

[0054] For example, in the cardiac surgery simulation evaluation, the evaluation indicator matrix data showed that during the suturing operation, the doctor's needle distance uniformity and suturing depth control needed to be improved. The early warning analysis data pointed out that some suturing points were too close to the coronary artery, posing a risk of injury. By comparing with the expert knowledge base, specific suggestions for improving suturing skills were generated, including the adjustment of the needle insertion angle and the optimization of tension control. The risk prevention and control strategy formulated a detailed safety area division plan for the protection of the coronary artery. The final analysis report not only pointed out the specific links that need to be improved, but also provided a systematic training plan, such as adding special exercises for suturing depth control, and strengthening safety awareness training for operations around the coronary arteries.

[0055] The virtual surgery simulation method based on mixed reality technology in the embodiment of the present application is described above. The virtual surgery simulation system based on mixed reality technology in the embodiment of the present application is described below. Figure 2 In the embodiment of the present application, an embodiment of a virtual surgery simulation system based on mixed reality technology includes: The acquisition module is used to extract spatial features and perform registration and calibration processing on the real surgical environment data and on-site video data based on the acquired three-dimensional spatial information, and generate mixed reality space mapping data and real-time environmental monitoring data; A mapping module, used to perform virtual-real registration and physical property mapping processing on the patient's medical image according to the mixed reality space mapping data, so as to obtain virtual surgical scene data and real-time correction parameters; A fusion module, configured to perform time-series fusion processing on multi-source sensor data of the surgical instrument according to the mixed reality space mapping data, the real-time environmental monitoring data and the real-time correction parameters, so as to obtain mixed reality interaction data; A simulation module, used to simulate the surgical process through tissue deformation simulation calculation according to the virtual surgical scene data and the mixed reality interaction data, so as to obtain surgical simulation data and process monitoring data; An identification module, configured to perform feature analysis and pattern recognition processing on the surgical operation trajectory based on the surgical simulation data, the process monitoring data and the environmental real-time monitoring data, and generate surgical evaluation data and real-time warning information; The analysis module is used to perform operation suggestion analysis on the surgery evaluation data and the real-time warning information to obtain a surgery simulation process analysis report and target suggestion data.

[0056] Through the collaboration of the above components, the accurate reconstruction of the surgical scene is achieved by extracting spatial features and aligning and calibrating the real surgical environment data and the on-site video data, which improves the realism and immersion of the mixed reality environment. Through virtual-real alignment and physical property mapping, the virtual surgical scene is closer to the real surgical environment, which enhances the authenticity of the simulation training. In terms of surgical instrument interaction, the time series fusion processing of multi-source sensor data is used to improve the accuracy and stability of surgical instrument position tracking. At the same time, through tissue deformation simulation calculation, a realistic surgical process simulation effect is achieved. In the surgical operation evaluation link, the surgical operation trajectory is analyzed and pattern recognized, and an objective and comprehensive evaluation system is established. Through the real-time early warning mechanism, potential risks are discovered in time, which improves the safety of surgical training. Finally, through the in-depth analysis of surgical evaluation data and early warning information, detailed analysis reports and targeted improvement suggestions are generated, which improves the effect and efficiency of surgical training.

[0057] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the aforementioned embodiments can still be modified, or some of the technical features therein can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A virtual surgery simulation method based on mixed reality technology, characterized in that: The virtual surgery simulation method based on mixed reality technology includes: Based on the collected 3D spatial information, the real surgical environment data and on-site video data are processed for spatial feature extraction and registration and calibration to generate mixed reality space mapping data and real-time environmental monitoring data; According to the mixed reality space mapping data, the patient's medical image is processed by virtual-real registration and physical property mapping to obtain virtual surgical scene data and real-time correction parameters; According to the mixed reality space mapping data, the real-time environment monitoring data and the real-time correction parameters, time-series fusion processing is performed on the multi-source sensor data of the surgical instrument to obtain mixed reality interaction data; According to the virtual surgical scene data and the mixed reality interaction data, the surgical process is simulated by tissue deformation simulation calculation to obtain surgical simulation data and process monitoring data; Based on the surgical simulation data, the process monitoring data and the real-time environmental monitoring data, feature analysis and pattern recognition processing are performed on the surgical operation trajectory to generate surgical evaluation data and real-time warning information; The surgical evaluation data and the real-time warning information are analyzed for operation suggestions to obtain a surgical simulation process analysis report and target suggestion data.

2. The virtual surgery simulation method based on mixed reality technology according to claim 1, characterized in that: Based on the collected three-dimensional spatial information, the real surgical environment data and the on-site video data are subjected to spatial feature extraction and registration and calibration processing to generate mixed reality space mapping data and real-time environmental monitoring data, including: Collecting surgical space image data, performing depth feature segmentation on the surgical space image data, and obtaining surgical scene depth data; Collecting surgical environment positioning data through a spatial positioning sensor, and performing time domain feature calibration on the surgical scene depth data and the surgical environment positioning data to obtain a scene feature matrix; Acquire a real-time video stream of the surgical scene according to the video acquisition unit, extract spatial feature points from the video stream, and obtain video feature point data; Performing spatial registration calculation on the scene feature matrix and the video feature point data to obtain a spatial registration mapping relationship; Performing real-time monitoring and calculation of surgical environment parameters according to the spatial registration mapping relationship to obtain real-time environmental monitoring data; The spatial registration mapping relationship and the environmental real-time monitoring data are subjected to feature fusion processing to generate the mixed reality spatial mapping data and the environmental real-time monitoring data.

3. The virtual surgery simulation method based on mixed reality technology according to claim 1, characterized in that: The method of performing virtual-real registration and physical property mapping processing on the patient's medical image according to the mixed reality space mapping data to obtain virtual surgical scene data and real-time correction parameters includes: Through feature analysis and calculation, the tissue interface information in the medical image data is processed at multiple scales to obtain anatomical feature distribution data; Using the mixed reality space mapping data as a reference benchmark to perform pose estimation and error compensation processing on the anatomical feature distribution data to obtain initial registration data; The initial registration data is used to perform key point screening and spatial feature matching operations, and the spatial correspondence is iteratively optimized to obtain virtual-real fusion mapping data; Performing physical parameter calculation and hierarchical analysis processing on different tissue regions in the virtual-reality fusion mapping data to obtain tissue parameter distribution data; Performing dynamic correction calculation on the deformation response in the virtual environment according to the tissue parameter distribution data to obtain real-time correction parameters; The virtual-reality fusion mapping data and the tissue parameter distribution data are fused and superimposed according to the topological structure to generate the virtual surgery scene data and the real-time correction parameters.

4. The virtual surgery simulation method based on mixed reality technology according to claim 1, characterized in that: The method of performing time series fusion processing on multi-source sensor data of surgical instruments according to the mixed reality space mapping data, the real-time environmental monitoring data and the real-time correction parameters to obtain mixed reality interaction data includes: Extract the time-space distribution and feature analysis of the multi-source sensor data of the surgical instrument to obtain the initial state data of the instrument; Performing spatial transformation processing on the mixed reality space mapping data and the real-time environmental monitoring data, and obtaining the instrument space coordinate data through projection mapping; Calibrate and synchronize the instrument spatial coordinate data based on the real-time correction parameters to obtain corrected instrument trajectory data; Performing motion feature analysis and interaction state recognition processing on the corrected instrument trajectory data to obtain instrument interaction feature data; Performing constraint condition calculation and space detection processing on the device interaction feature data to obtain device interaction response data; The device interaction feature data and the device interaction response data are subjected to time domain fusion and space domain fusion processing to generate the mixed reality interaction data.

5. The virtual surgery simulation method based on mixed reality technology according to claim 1, characterized in that: The step of simulating the surgical process by tissue deformation simulation calculation based on the virtual surgical scene data and the mixed reality interaction data to obtain surgical simulation data and process monitoring data includes: Performing tissue stratification and deformation parameter space construction on the virtual surgical scene data to obtain three-dimensional surgical scene analysis data; Extracting nonlinear transfer characteristics and calculating stress distribution of mechanical parameters in the mixed reality interaction data to form surgical interaction mechanical data; Perform surgical cutting path planning and tissue deformation compensation calculation based on the three-dimensional surgical scene analysis data and the surgical interactive mechanics data, and output surgical operation prediction data; Performing tissue interface deformation analysis on the surgical operation prediction data by rigid body transformation and elastic deformation coupling calculation to obtain tissue interface contact data; Performing multi-level physical deformation and stress propagation analysis on the tissue interface contact data to form surgical process deformation data; The surgical process deformation data and the tissue interface contact data are subjected to state fusion and process recording processing to generate the surgical simulation data and the process monitoring data.

6. The virtual surgery simulation method based on mixed reality technology according to claim 1, characterized in that: The method of performing feature analysis and pattern recognition processing on the surgical operation trajectory based on the surgical simulation data, the process monitoring data and the real-time environmental monitoring data to generate surgical evaluation data and real-time warning information includes: Extracting trajectory features and performing segmented processing on the surgical simulation data to separate and obtain spatial motion feature data; Perform node analysis and behavior feature recognition on the process monitoring data according to key surgical steps to form surgical step sequence data; Performing operation space limitation and surgical area boundary dynamic tracking processing on the real-time environmental monitoring data to obtain surgical area constraint data; Comparing and analyzing the spatial motion feature data with the standard operation sequence through spatial trajectory similarity calculation to obtain operation standardization evaluation data; Performing safety boundary determination and dangerous area warning calculation according to the surgical step sequence data and the surgical area constraint data to form surgical risk prediction data; Comprehensive feature analysis and risk level determination are performed based on the operation standardization assessment data and the surgical risk prediction data to generate the surgical assessment data and the real-time warning information.

7. The virtual surgery simulation method based on mixed reality technology according to claim 1, characterized in that: The operation suggestion analysis is performed on the operation evaluation data and the real-time warning information to obtain a surgery simulation process analysis report and target suggestion data, including: Quantifying and analyzing the correlation of sub-items of the surgical evaluation data to generate evaluation indicator matrix data; The real-time warning information is processed through risk level classification and risk factor identification to obtain warning analysis data; Mapping and transforming the evaluation index matrix data according to the standard data in the expert knowledge base to form initial recommendation data; The early warning analysis data is graded and countermeasures are generated through risk prevention and control strategies, and risk response data is output; Perform multi-layer optimization calculation and solution integration on the initial suggestion data and the risk response data to obtain target suggestion data; The evaluation index matrix data, the early warning analysis data and the target recommendation data are integrated and analyzed according to the evaluation specifications to generate the surgical simulation process analysis report and the target recommendation data.

8. A virtual surgery simulation system based on mixed reality technology, used to implement the virtual surgery simulation method based on mixed reality technology as described in any one of claims 1 to 7, characterized in that: The virtual surgery simulation system based on mixed reality technology includes: The acquisition module is used to extract spatial features and perform registration and calibration processing on the real surgical environment data and on-site video data based on the acquired three-dimensional spatial information, and generate mixed reality space mapping data and real-time environmental monitoring data; A mapping module, used to perform virtual-real registration and physical property mapping processing on the patient's medical image according to the mixed reality space mapping data, so as to obtain virtual surgical scene data and real-time correction parameters; A fusion module, configured to perform time-series fusion processing on multi-source sensor data of the surgical instrument according to the mixed reality space mapping data, the real-time environmental monitoring data and the real-time correction parameters, so as to obtain mixed reality interaction data; A simulation module, used to simulate the surgical process through tissue deformation simulation calculation according to the virtual surgical scene data and the mixed reality interaction data, so as to obtain surgical simulation data and process monitoring data; An identification module, configured to perform feature analysis and pattern recognition processing on the surgical operation trajectory based on the surgical simulation data, the process monitoring data and the environmental real-time monitoring data, and generate surgical evaluation data and real-time warning information; The analysis module is used to perform operation suggestion analysis on the surgery evaluation data and the real-time warning information to obtain a surgery simulation process analysis report and target suggestion data.

Images