Surgical minimally invasive surgery data processing method and system based on AR technology

Abstract

The invention relates to the technical field of image processing, and discloses a surgical minimally invasive surgery data processing method and system based on an AR technology. The method comprises the following steps: performing three-dimensional reconstruction and lesion segmentation on medical image data, and extracting feature points; performing position acquisition and space mapping on the surgical instrument by using the feature point data; reconstructing a light field according to the feature points and the position data and generating an augmented reality scene; performing trajectory analysis and path planning based on augmented reality; performing real-time monitoring and risk early warning on the operation path; and finally, extracting an operation rule and establishing a knowledge mapping relation. According to the method, association analysis and fusion processing of multi-source heterogeneous data in the operation process are achieved, a complete technical link from data collection to knowledge extraction is established, and the operation safety and the intelligent level are improved.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a method and system for processing surgical minimally invasive surgery data based on AR technology. Background Art

[0002] In existing minimally invasive surgical procedures, the application of AR technology is mainly concentrated in surgical navigation and real-time imaging during surgery. Traditional AR surgical navigation technology provides doctors with intuitive surgical guidance by superimposing preoperative imaging data with real-time surgical scenes. At the same time, existing technologies also include spatial positioning of surgical instruments based on optical tracking, electromagnetic tracking and other methods to achieve real-time navigation of surgical paths. In terms of surgical data processing, existing methods mainly use image registration, three-dimensional reconstruction, trajectory planning and other technologies to collect and analyze various types of data during the operation.

[0003] However, existing technologies have obvious deficiencies in data processing and risk warning. First, existing data processing methods often separate and process various types of data, lacking systematic analysis and correlation mining of data from the entire surgical process. Second, in terms of surgical risk warning, existing technologies mainly rely on simple threshold judgments and cannot effectively identify and predict complex risk evolution processes. In addition, existing technologies are also deficient in knowledge accumulation and experience conversion, making it difficult to effectively convert surgical data into a reusable knowledge system. Summary of the invention

[0004] The present application provides a surgical minimally invasive surgery data processing method and system based on AR technology, which is used to realize the correlation analysis and fusion processing of multi-source heterogeneous data during the operation, establish a complete technical link from data collection to knowledge extraction, and improve the safety and intelligence level of the operation.

[0005] In a first aspect, the present application provides a method for processing surgical minimally invasive surgery data based on AR technology, and the method for processing surgical minimally invasive surgery data based on AR technology includes: performing three-dimensional reconstruction processing on medical imaging data, performing lesion segmentation on the reconstructed data, processing the segmented data through feature extraction, and obtaining three-dimensional feature point data; using the three-dimensional feature point data to collect and process the position of surgical instruments, spatially mapping the collected position data, and aligning and fusing the mapped data to form spatial positioning data; reconstructing light field data based on the three-dimensional feature point data and the spatial positioning data, deforming the reconstructed data, combining the processed data with real-scene data to obtain augmented reality data; performing operation trajectory analysis based on the augmented reality data, performing path planning on the analysis data, and obtaining optimal path data through optimization processing; real-time monitoring of the optimal path data, performing risk analysis on the monitoring data, and performing early warning processing on the analysis results through threshold judgment to obtain risk data; extracting surgical rules using the risk data, performing multi-dimensional analysis on the rule data, establishing a mapping relationship between data through association processing, and forming target knowledge data.

[0006] In a second aspect, the present application provides a minimally invasive surgery data processing system based on AR technology, and the minimally invasive surgery data processing system based on AR technology includes: The reconstruction module is used to perform three-dimensional reconstruction on the medical image data, segment the lesions on the reconstructed data, process the segmented data by feature extraction, and obtain three-dimensional feature point data; An acquisition module is used to acquire and process the position of the surgical instrument using the three-dimensional feature point data, spatially map the acquired position data, and register and fuse the mapped data to form spatial positioning data; A processing module, used to reconstruct light field data according to the three-dimensional feature point data and the spatial positioning data, perform deformation processing on the reconstructed data, and combine the processed data with the real scene data to obtain augmented reality data; An analysis module is used to analyze the operation trajectory according to the augmented reality data, perform path planning on the analyzed data, and obtain the optimal path data through optimization processing; A monitoring module is used to monitor the optimal path data in real time, perform risk analysis on the monitoring data, perform early warning processing on the analysis results through threshold judgment, and obtain risk data; A module is established to extract surgical rules using the risk data, perform multi-dimensional analysis on the rule data, establish a mapping relationship between the data through association processing, and form target knowledge data.

[0007] In the technical solution provided by this application, significant technical effects are achieved through data processing in multiple links: by performing three-dimensional reconstruction processing on medical image data and implementing lesion segmentation, combining feature extraction technology to obtain three-dimensional feature point data, the precise positioning and feature expression of the surgical target area are achieved; the three-dimensional feature point data is used to collect and spatially map the position of surgical instruments in real time, and spatial positioning data is formed through registration fusion, ensuring the accuracy and real-time positioning of surgical instruments; light field data is reconstructed based on three-dimensional feature point data and spatial positioning data, combined with deformation processing and scene fusion technology, to achieve the natural presentation of augmented reality scenes; operation trajectory analysis and path planning are performed based on augmented reality data, and optimal path data is obtained through optimization processing, which improves the accuracy and safety of surgical operations; real-time monitoring and risk analysis of optimal path data are performed, and early warning processing is achieved through threshold judgment to form risk data, which enhances the safety of the surgical process; finally, risk data is used to extract surgical rules, multi-dimensional analysis is performed and data mapping relationships are established to form target knowledge data, realizing the effective accumulation of surgical experience and knowledge transformation. A complete technical chain from data collection, processing, analysis to knowledge extraction is realized, which significantly improves the accuracy, safety and intelligence level of minimally invasive surgery. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0009] Figure 1 This is a schematic diagram of an embodiment of a method for processing minimally invasive surgery data based on AR technology in an embodiment of the present application; Figure 2 This is a schematic diagram of an embodiment of a minimally invasive surgical data processing system based on AR technology in an embodiment of the present application. DETAILED DESCRIPTION

[0010] The embodiments of the present application provide a method and system for processing surgical minimally invasive surgery data based on AR technology. The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described here can be implemented in an order other than that illustrated or described here. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0011] For ease of understanding, the specific process of the embodiment of the present application is described below. Figure 1 In the embodiment of the present application, an embodiment of the method for processing surgical minimally invasive surgery data based on AR technology includes: Step S101, performing three-dimensional reconstruction on the medical image data, performing lesion segmentation on the reconstructed data, processing the segmented data by feature extraction, and obtaining three-dimensional feature point data; Step S102: using the three-dimensional feature point data to collect and process the position of the surgical instrument, spatially mapping the collected position data, registering and fusing the mapped data, and forming spatial positioning data; Step S103, reconstructing light field data according to the three-dimensional feature point data and the spatial positioning data, performing deformation processing on the reconstructed data, and combining the processed data with the real scene data to obtain augmented reality data; Step S104: Analyze the operation trajectory according to the augmented reality data, plan the path of the analyzed data, and obtain the optimal path data through optimization processing; Step S105: monitor the optimal path data in real time, perform risk analysis on the monitoring data, perform early warning processing on the analysis results through threshold judgment, and obtain risk data; Step S106: extract surgical rules using risk data, perform multi-dimensional analysis on the rule data, establish a mapping relationship between data through association processing, and form target knowledge data.

[0012] It is understandable that the execution subject of the present application can be a surgical minimally invasive surgery data processing system based on AR technology, or a terminal or a server, which is not limited here. The present application embodiment is described by taking the server as the execution subject as an example.

[0013] Specifically, a three-dimensional reconstruction process of medical image data is performed. Medical image data generated by CT or nuclear magnetic resonance is received, and grayscale value processing is performed on the original two-dimensional slice image. In the grayscale value processing stage, the brightness value of each pixel is converted into a standardized value, and the converted data will be used for subsequent three-dimensional reconstruction. In the three-dimensional reconstruction process, the processed two-dimensional slice data is stacked in space and spatially reconstructed based on voxel density. After the reconstruction is completed, the data is segmented into lesion areas. The segmentation process is mainly based on the difference in image density values ​​to distinguish the target area from the surrounding normal tissue. In the segmentation process, the boundary points of the lesion area are first marked, and then the entire lesion area is completely outlined by the region growing algorithm. Feature extraction is performed on the segmented data, mainly including the extraction of edge features, texture features and morphological features. The edge feature reflects the contour information of the lesion area, the texture feature characterizes the internal structural characteristics of the lesion tissue, and the morphological feature describes the geometric shape characteristics of the lesion area. Through the combined analysis of these features, the three-dimensional feature point data of the lesion area is obtained. The acquired three-dimensional feature point data is used to collect and process the position of the surgical instrument. In the acquisition process, marker points are first arranged on the surface of the surgical instrument, and these marker points have a specific geometric distribution form. During the acquisition and processing phase, optical sensors are used to track the marker points in real time and record the coordinate information of the marker points in three-dimensional space. The acquired position data is then spatially mapped to convert the position information in the instrument coordinate system to the patient coordinate system. The mapping process requires the establishment of a transformation relationship between the two coordinate systems, including the calculation of the rotation matrix and the translation vector. The mapped data is registered and fused. The registration process achieves precise alignment by minimizing the spatial distance error between the two sets of data. The fusion process integrates the registered data to form unified spatial positioning data.

[0014] Light field data reconstruction is performed based on three-dimensional feature point data and spatial positioning data. The light field data reconstruction process aligns the two sets of data in the same coordinate system, and then constructs the light field distribution of the surgical scene based on the light propagation theory. The reconstruction process considers the emission, propagation and reception characteristics of light, and also includes the reflection characteristics of the surfaces of each object in the scene. The reconstructed data is deformed, and the deformation processing is mainly aimed at the deformation characteristics of soft tissue during surgical operations. The processing process includes establishing a deformation model, calculating deformation parameters, and predicting tissue deformation trends. The processed data is combined with the real scene data. The combination process considers factors such as perspective changes, lighting conditions, and occlusion relationships. The image fusion algorithm is used to achieve seamless integration of virtual information and real scenes to generate augmented reality data. The surgical operation trajectory analysis is performed based on the augmented reality data. The trajectory analysis first decomposes the motion path of the surgical instrument and extracts key motion features. The path planning is performed on the analyzed data. The planning process considers the constraints of the surgical space, including avoiding important anatomical structures and maintaining a safe operating distance. The path planning adopts a multi-objective optimization method, while considering multiple objectives such as path length, operation safety, and energy consumption. The optimal path data is obtained through optimization processing. The optimization process adopts an iterative calculation method to continuously adjust the path parameters until all constraints are met and the optimal goal is achieved.

[0015] The optimal path data is monitored in real time. The monitoring process includes real-time collection and analysis of parameters such as position deviation, speed change, and acceleration fluctuation during the path execution process. Risk analysis is performed on the monitoring data. The analysis process mainly examines the relationship between the current operation state and the preset safety threshold. The analysis results are pre-warned through threshold judgment. When the monitoring parameters exceed the preset threshold, the system will generate a corresponding level of pre-warning information and give a risk assessment result. The pre-warning process adopts a multi-level pre-warning mechanism to generate pre-warning signals of different levels according to the degree of risk to form risk data. Finally, the risk data is used to extract surgical rules. The rule extraction process first performs a time series analysis on the risk data to identify the occurrence mode and time characteristics of risk events. The extracted rule data is analyzed in multiple dimensions, and the analysis dimensions include spatial dimension, time dimension, operation dimension, etc. The mapping relationship between data is established through association processing. The association processing mainly examines the mutual influence and dependency between different data features. The target knowledge data is formed, which contains information on multiple aspects such as experience knowledge of surgical operations, risk prevention measures, and emergency treatment plans.

[0016] For example, the CT image data of patients is processed. The original CT data contains multiple groups of slice images, each with a resolution of 512×512 pixels and a grayscale value range of 0-4095. These slice data are reconstructed into a three-dimensional model through a three-dimensional reconstruction algorithm. During the reconstruction process, about 2,000 feature points of the tumor area are extracted, which accurately describe the spatial morphology and location information of the tumor. During the operation, the position data is obtained by tracking the surgical instruments, with a sampling frequency of 60Hz and a position accuracy of sub-millimeter level. Combining the spatial position information of the patient's anatomical structure and the surgical instruments, the system generates an augmented reality view and superimposes the virtual navigation information on the real-time surgical field of view. Based on this information, path planning is performed, and the optimal path obtained by planning avoids important blood vessels and ensures the shortest path for surgical operations. During the operation, the real-time monitoring system tracks the position of the instrument, and immediately issues a risk warning when it is found that the deviation from the preset path exceeds the safety threshold. Through data analysis of the entire surgical process, the surgical rules are extracted and a knowledge base is formed.

[0017] In the embodiment of the present application, significant technical effects are achieved through data processing in multiple links: by performing three-dimensional reconstruction processing on medical image data and implementing lesion segmentation, combining feature extraction technology to obtain three-dimensional feature point data, accurate positioning and feature expression of the surgical target area are achieved; the three-dimensional feature point data is used to collect and spatially map the position of surgical instruments in real time, and spatial positioning data is formed through registration fusion, ensuring the accuracy and real-time positioning of surgical instruments; light field data is reconstructed based on three-dimensional feature point data and spatial positioning data, combined with deformation processing and scene fusion technology, to achieve the natural presentation of augmented reality scenes; operation trajectory analysis and path planning are performed based on augmented reality data, and optimal path data is obtained through optimization processing, which improves the accuracy and safety of surgical operations; real-time monitoring and risk analysis of optimal path data are performed, and early warning processing is achieved through threshold judgment to form risk data, which enhances the safety of the surgical process; finally, risk data is used to extract surgical rules, multi-dimensional analysis is performed and data mapping relationships are established to form target knowledge data, which realizes the effective accumulation of surgical experience and knowledge transformation. A complete technical chain from data acquisition, processing, analysis to knowledge extraction is achieved, which significantly improves the accuracy, safety and intelligence level of minimally invasive surgery.

[0018] In a specific embodiment, the process of executing step S101 may specifically include the following steps: (1) Project the grayscale values ​​in the medical image data into three-dimensional space and obtain the initial voxel data through pixel density threshold processing; (2) Performing region growing processing on the initial voxel data, separating the target area according to the grayscale gradient value, and obtaining separated and reconstructed data; (3) The separated and reconstructed data are analyzed for connected domains through region labeling, and the boundaries of the connected domains are extracted to form reconstructed data; (4) The reconstructed data is subjected to regional segmentation to extract the lesion boundary, and feature enhancement processing is performed based on the boundary curvature to obtain lesion segmentation data; (5) Perform edge detection on the lesion segmentation data, extract key point coordinates based on the detection results, and obtain segmentation data; (6) Calculate the feature point density of the segmented data, filter the feature points through spatial distance constraints, and obtain three-dimensional feature point data.

[0019] Specifically, the two-dimensional slice images generated by CT or MRI are processed. The pixels in each slice image contain specific grayscale values, which reflect the density characteristics of different tissues. The grayscale value information of these two-dimensional slices is projected into three-dimensional space, taking into account the spacing between slices and the actual physical size of the pixels. The projected data is screened by setting a density threshold. The selection of the density threshold is based on the attenuation characteristics of different tissues. For example, bone tissue, soft tissue and diseased tissue have different density ranges. After threshold processing, the initial voxel data obtained retains the basic characteristics of the tissue structure. The initial voxel data is subjected to region growth processing. Region growth is a process that starts from a seed point and gradually incorporates surrounding voxels with similar characteristics into the target region. In this process, the grayscale gradient value around each voxel point is first calculated. The gradient value reflects the severity of the grayscale change. Different tissue regions are separated based on the change characteristics of the gradient value. During the separation process, the growth criteria are set, including grayscale value similarity, spatial continuity and other conditions. When the adjacent voxels meet the growth conditions, they are incorporated into the current region, thereby obtaining separated and reconstructed data.

[0020] Connected domain analysis is performed on the separated reconstructed data. Connected domain analysis is used to determine the set of interconnected voxels in three-dimensional space, and each independent connected region is numbered through a marking algorithm. In the marking process, a recursive scanning method is used to check the 26 neighborhood points of each voxel to determine whether they belong to the same connected region. The boundary extraction process is performed on the marked connected domain. The boundary extraction process determines whether it is a boundary point by detecting the neighborhood features of the voxel to form the reconstructed data.

[0021] The reconstructed data is used to extract the lesion boundary. The data is processed by the regional segmentation method, which is based on the grayscale features, texture features and morphological features of the image. In the process of boundary extraction, the curvature value of the boundary point is calculated, and the curvature value reflects the local geometric features of the boundary. The boundary curvature is subjected to feature enhancement processing, which includes normalization and highlighting of the curvature value to obtain the lesion segmentation data. The lesion segmentation data is subjected to edge detection. Edge detection is mainly based on the gradient features of the grayscale value, and the edge is detected by calculating the gradient operator in the three-dimensional space. In the detection process, the gradient amplitude and direction of each point are first calculated, and then the non-edge points are removed by non-maximum suppression, and finally the edge points are determined by double threshold processing. The coordinates of the key points are extracted according to the detected edge points. The selection of the key points is based on the curvature, connectivity and other features of the edge points to form the segmentation data.

[0022] The process of feature point density calculation can be expressed by the following formula: in: Represents the density value of the feature point at the coordinate point (x, y, z); Represents the weight coefficient of the i-th feature point; Represents the position vector of the current calculation point; Represents the position vector of the i-th feature point; represents the standard deviation parameter of the Gaussian kernel function; n represents the total number of feature points.

[0023] For example, when processing CT image sequences. The grayscale value distribution of each CT slice reflects the attenuation characteristics of different tissues. The grayscale value range of liver tissue is significantly different from that of surrounding tissues, while the tumor area shows grayscale characteristics different from those of normal liver tissue. The initial three-dimensional voxel model is constructed through grayscale value projection. In the process of regional growth, the feature points of the tumor area are selected as seed points, and the region is separated based on the grayscale gradient value, which can accurately delineate the boundary of the tumor. Connected domain analysis ensures the integrity of the segmented area, and the precise contour of the tumor is obtained through boundary extraction. Feature enhancement processing highlights the features of the tumor boundary, and the most representative three-dimensional feature points are screened out through feature point density calculation. These feature points accurately describe the spatial morphological characteristics of the tumor.

[0024] In a specific embodiment, the process of executing step S102 may specifically include the following steps: (1) Projecting the 3D feature point data into the surgical space, calibrating the projection results, and obtaining the calibrated coordinate data; (2) Optically track the position of the surgical instrument and perform spatial fixed-point processing on the tracking data to obtain fixed-point data; (3) Convert the position parameters in the fixed-point data into three-dimensional space coordinates, perform noise filtering on the coordinate data, and form the collected position data; (4) Perform coordinate system transformation on the collected position data, project the transformed data into the calibration space, and obtain mapping data; (5) Calculate the overlap between the mapping data and the calibration coordinate data, perform error compensation on the calculation results, and form registration data; (6) Perform spatial registration processing on the registration data, perform multi-dimensional fusion of the registration parameters, and obtain spatial positioning data.

[0025] Specifically, the process of projecting three-dimensional feature point data into the surgical space involves coordinate transformation and calibration operations. The three-dimensional feature point data contains key position information of the surgical target area, and these data need to be converted from the image space to the actual surgical operation space. The projection process uses a ray tracing method to project each feature point into the surgical space along a specific direction. When calibrating the projection results, it is necessary to arrange a calibration plate in the surgical space, on which there is an array of marker points with known spacing. By identifying the correspondence between the actual position and the theoretical position of these marker points, a coordinate transformation matrix is ​​established. During the calibration process, repeated measurements at multiple angles and multiple positions are used, and the calibration parameters obtained by the least squares method are optimized to form calibration coordinate data. Optical tracking of the position of surgical instruments is achieved by installing reflective marker points on the surface of the instrument. These marker points have a specific geometric distribution pattern, which is convenient for tracking camera recognition. During the optical tracking process, multiple high-speed cameras simultaneously capture the spatial position of the marker points, and calculate the three-dimensional coordinates of the marker points by the principle of triangulation. The tracking data undergoes spatial fixed-point processing, including steps such as removing outliers, smoothing trajectories, and compensating for motion delays. Fixed-point processing uses the median filter method to remove instantaneous noise, and the Kalman filter is used to smooth the motion trajectory to obtain stable fixed-point data. When converting fixed-point data into three-dimensional space coordinates, the local coordinate system of the marker point needs to be determined first. The position parameters of each marker point include its relative position and direction information in the local coordinate system of the instrument. These local coordinates are converted to the global surgical space coordinate system through the rigid body transformation matrix. The data after coordinate conversion often contains measurement noise, which requires noise filtering. The filtering process uses a Butterworth low-pass filter, and the cutoff frequency is set according to the characteristic frequency of the surgical operation to ensure that high-frequency noise is filtered out while retaining effective motion information. The data after filtering is used as the collected position data for subsequent processing. When performing coordinate system transformation on the collected position data, the mapping relationship between different reference systems is considered. The movement of surgical instruments is usually described in their own coordinate system, while surgical navigation needs to be performed in the patient coordinate system. Therefore, it is necessary to establish a transformation matrix between the two coordinate systems, including rotation and translation. The transformation process uses quaternion representation to avoid the universal joint deadlock problem that may occur in Euler angle representation. The transformed data is mapped to the calibration space through projection transformation to form mapping data.

[0026] Overlap calculation is a key step in evaluating the accuracy of spatial registration, and its mathematical expression is as follows: in: Indicates the degree of overlap between two sets of point cloud data A and B; Represents the weight coefficient between the i-th point and the j-th point; Represents the position vector of the i-th point in point cloud A; Represents the position vector of the jth point in point cloud B; represents the bandwidth parameter of the Gaussian kernel function; N represents the number of points in the point cloud.

[0027] The result obtained by overlap calculation is used to evaluate the registration quality, and the calculation result is error compensated. The compensation process considers two parts: systematic error and random error. The systematic error mainly comes from the measurement deviation of the equipment itself, and the compensation parameters are obtained by pre-calibration; the random error is estimated and corrected by statistical methods. The data after error compensation forms the registration data. The registration data is spatially registered, and the iterative nearest point algorithm is used to accurately align the two sets of data. The matching degree of both position and direction is considered in the registration process, and the iterative process is controlled by setting the convergence threshold. The multi-dimensional fusion of the registration parameters adopts the weighted average method, and the weight value is dynamically adjusted according to the reliability of the data in each dimension. The fused data is the spatial positioning data.

[0028] For example, three-dimensional feature points of the tumor area were extracted from the preoperative CT reconstruction data, and these feature points were projected into the actual surgical space through the optical positioning equipment in the operating room. During the calibration process, a calibration plate with 64 markers was used, and sub-millimeter calibration accuracy was achieved through multi-angle acquisition. Four reflective ball markers were installed on the surgical instruments to form a specific geometric configuration. During the movement of the instrument, the optical tracking system collected the position of the markers at a frequency of 60 frames per second, and the real-time position data of the instrument was obtained through the spatial fixed-point algorithm. After the original tracking data was filtered for noise, the position accuracy was improved to within 0.3 mm. The processed position data was projected into the patient coordinate system and aligned with the preoperatively planned target. The alignment process evaluated the matching quality through overlap calculation. After multiple rounds of iterative optimization, the precise spatial positioning required for real-time intraoperative navigation was achieved.

[0029] In a specific embodiment, the process of executing step S103 may specifically include the following steps: (1) Perform spatial registration on the 3D feature point data and the spatial positioning data, and uniformly process the registration results through coordinate transformation to obtain unified coordinate data; (2) Perform ray tracing on the unified coordinate data, convert the tracing results into brightness value distribution, and obtain the initial light field data; (3) Perform density calculation on the initial light field data and interpolate the calculation results to form interpolated light field data; (4) Calculate the tissue deformation of the interpolated light field data, perform numerical iteration on the deformation parameters, and obtain the reconstructed data; (5) Performing surface fitting on the reconstructed data and processing the fitting results by calculating the shape variables to form processed data; (6) Perform deep alignment on the processed data and spatially fuse the alignment results with the real-scene data to obtain augmented reality data.

[0030] Specifically, the three-dimensional feature point data and spatial positioning data are processed first. The three-dimensional feature point data comes from the reconstruction of preoperative medical images, while the spatial positioning data is the position information of the surgical instrument in the actual space. The registration process uses the rigid body transformation method to convert the two sets of data into the same coordinate system. The coordinate transformation includes two parts: rotation and translation. The transformation parameters are optimized by minimizing the distance between the corresponding points of the two sets of points. The transformation process also considers the coordinate system differences of different devices, unifies all data into the global coordinate system of the operating room, and obtains unified coordinate data. Ray tracing is a key step in converting unified coordinate data into light field representation. The tracing process starts from the virtual camera position, emits light along the preset direction, and calculates the intersection with the surface of the object in the scene. Each intersection of a ray and the surface of the object records information such as position, normal vector, and material properties. This information is used to calculate the propagation characteristics of light, including effects such as reflection, refraction, and scattering. The interaction between light and the surface of the object produces brightness values, which form the initial data of the light field according to the spatial distribution.

[0031] When performing density calculation on the initial light field data, it is necessary to evaluate the light field density distribution at each sampling point in space. Density calculation takes into account the spatial distribution characteristics of light and the law of energy attenuation. The calculated density distribution may have the problem of uneven sampling, and interpolation processing is required to supplement the missing data points. The interpolation process uses radial basis functions to perform continuous processing on sparse data points in space to form interpolated light field data.

[0032] The mathematical expression of tissue deformation calculation is as follows: in: Represents the deformation field at a spatial point (x, y, z); represents the weight coefficient of the kth control point; represents radial basis function; Represents the position vector of the current calculation point; represents the position vector of the kth control point; represents the affine transformation parameters; Represents the spatial coordinate component; represents the translation parameter; K represents the total number of control points.

[0033] The deformation calculation is based on the physical model, taking into account the elastic properties of the tissue and the external forces. The deformation parameters are solved by an iterative optimization method, and each iteration updates the node position and internal stress distribution until an equilibrium state is reached to obtain the reconstructed data. When performing surface fitting on the reconstructed data, a three-dimensional spline interpolation method is used. The fitting process first constructs a control point grid, and then optimizes the fitting parameters by minimizing the surface energy functional. The deformation variable calculation is based on the local deformation characteristics of the fitted surface, including indicators such as curvature change and area change. The processed data retains the key features of tissue deformation.

[0034] The processed data is registered with the real scene image. Depth alignment is based on the principle of parallax matching, and the spatial position relationship is adjusted by comparing the depth information of virtual data and real scene data. The spatial fusion process takes into account factors such as lighting conditions and occlusion relationships, and uses a physically based rendering method to achieve seamless integration of virtual information and real scenes to generate augmented reality data.

[0035] For example, in liver tumor surgery, the three-dimensional tumor model reconstructed by preoperative CT is spatially registered with the real-time position data of the surgical instruments. The registration process optimizes the transformation matrix to control the registration error within the millimeter range. Then ray tracing is performed to emit virtual rays from different angles of the surgical field of view and intersect with the reconstructed model to obtain the light field distribution. The density calculation stage evaluates the spatial distribution characteristics of the light field and supplements the data of the sparsely sampled area by interpolation. When processing soft tissue deformation, a deformation field containing hundreds of control points is established, and the deformation characteristics of liver tissue under respiratory movement and surgical operation are simulated by iterative calculation. The surface fitting stage uses high-order spline functions to describe the tissue surface and accurately captures the local detail changes during the deformation process. Finally, through deep alignment and spatial fusion, the virtual navigation information is superimposed on the real-time surgical field of view to achieve accurate surgical navigation.

[0036] In a specific embodiment, the process of executing step S104 may specifically include the following steps: (1) Segment the augmented reality data into time series, track the feature points of the segmented data, and obtain the spatiotemporal state data; (2) Perform multi-level trajectory decomposition on the spatiotemporal state data, extract trajectory points from the decomposition results, and form a trajectory point sequence; (3) Fit the trajectory point sequence with a polynomial curve, perform tangent analysis on the fitting results, and obtain analysis data; (4) Perform collision detection calculations on the analyzed data, filter the detection results according to the safety distance, and form candidate path data; (5) Calculate the path length of the path candidate data, solve the energy function of the calculation results, and obtain the path score data; (6) Calculate multi-objective weights on the path scoring data, and iterate and compare the path weights multiple times to form the optimal path data; (7) Perform speed planning on the optimal path data, calculate the acceleration constraints on the planning results, and obtain motion control data; (8) Perform trajectory smoothing on the motion control data, perform gradient optimization on the smoothing result, and update the optimal path data.

[0037] Specifically, the time series segmentation of augmented reality data is processed. Augmented reality data contains a fusion picture of virtual information and real scenes, and needs to be segmented in time order. Time series segmentation uses a fixed time window method to divide the continuous data stream into several time segments. In each time segment, key points are extracted by feature point detection algorithm. These feature points mainly include the marking points of surgical instruments, the feature points of anatomical structures, and the position information of virtual markers. The movement of these feature points in adjacent time segments is tracked by tracking algorithms, and the changes in their positions and speeds are recorded to form spatiotemporal state data. The spatiotemporal state data contains kinematic information such as the three-dimensional coordinates, movement speed, and acceleration of the feature points. The multi-level trajectory decomposition of the spatiotemporal state data is to analyze the hierarchical structure of surgical operations. The decomposition process uses the wavelet transform method to decompose the complex surgical operation trajectory into motion components of different scales. The main operation trajectory of the operation is reflected at a large scale, the detailed movements are captured at a medium scale, and the small scale corresponds to the fine-tuning of the hand. Feature points are extracted from the decomposition results of each scale. The extraction process selects key points in the motion trajectory based on features such as curvature and speed changes. These key points are connected in time order to form a trajectory point sequence, which reflects the complete action process of the surgical operation.

[0038] When fitting the trajectory point sequence to a curve, the polynomial approximation method is used. The fitting process selects a polynomial function of appropriate order, and the polynomial coefficients are determined by the least squares method. The fitted curve needs to meet the requirements of smoothness and continuity. Tangent analysis is performed on the fitting curve to calculate the tangent direction and curvature at each point. These parameters reflect the direction change and smoothness of the movement to form analysis data. Collision detection calculation is an important part of ensuring surgical safety. For each trajectory point in the analysis data, it is necessary to check its spatial relationship with the surrounding anatomical structure. The detection process uses a hierarchical bounding box method, first performing a rough detection, and then accurately calculating the area where the collision may occur. The detection results are screened according to the preset safety distance, and the path points that are too close to the important structure are eliminated, and the path points that meet the safety requirements are retained to form the path candidate data.

[0039] The path length calculation is based on the accumulation of distances between adjacent points in the path candidate data. The calculation process takes into account the smoothness and continuity of the path to avoid sharp turns. The energy function is applied to the calculation results for evaluation. The energy function includes multiple evaluation indicators such as path length, curvature change, and operation difficulty. The optimal value is solved by numerical optimization methods to obtain the path scoring data. Multi-objective weight calculation is a process of comprehensive evaluation of path scoring data. Multiple objectives need to be balanced during calculation, including the shortest path length, the maximum safety distance, and the minimum operation difficulty. The weight coefficients of each objective are adjusted through an iterative method. Each iteration compares the comprehensive scores of different paths, selects the path with the best comprehensive performance, and forms the optimal path data.

[0040] Speed ​​planning is the process of dynamically optimizing the optimal path data. The planning process takes into account the motion characteristics of surgical instruments, including constraints such as maximum speed and acceleration limit. By calculating the acceleration constraints on the planning results, the motion process is ensured to be smooth and controllable, and motion control data that meets the dynamic requirements is obtained.

[0041] The trajectory smoothing process uses the cubic spline interpolation method to make the discrete points in the motion control data continuous. The interpolation process needs to ensure the continuity of the position, velocity and acceleration at the nodes to avoid sudden changes during the motion process. The sliding window method is used in the smoothing process to optimize the local trajectory segment, and the window size is adjusted according to the accuracy requirements of the surgical operation. After obtaining the smooth trajectory, the trajectory is further refined by the gradient optimization method. The gradient optimization process considers multiple aspects such as the spatial distribution, motion characteristics and energy consumption of the trajectory. The optimization objectives include minimizing the curvature change rate of the trajectory, maintaining the smoothness of the motion, and reducing mechanical vibration. The optimization process uses an iterative method. Each iteration calculates the gradient information of the current trajectory and makes fine adjustments along the gradient direction until the convergence condition is reached. The optimization results are used to update the optimal path data. The update process ensures that the new path data meets the requirements of smoothness and dynamic constraints at the same time, providing a more accurate navigation reference for surgical operations.

[0042] For example, processing augmented reality scenes. The scene includes a three-dimensional reconstructed model of the lesion area, the real-time position of the surgical instrument, and the planned approach markers. These data are time-series segmented, and data is collected every 33 milliseconds to form a continuous time series. During the feature point tracking process, the motion trajectory of the top of the surgical instrument and important anatomical landmarks are tracked at the same time. In the multi-level trajectory decomposition link, different levels of operation actions including instrument introduction, fine operation, and instrument withdrawal are identified. The curve fitting process generates a smooth operation trajectory, and each trajectory point contains position and direction information. During collision detection, a safety distance of 5 mm from important blood vessels and a safety distance of 3 mm from brain tissue are set as constraints. Path planning takes into account multiple factors such as tissue damage risk and ease of operation, and the surgical path is obtained through multiple iterative optimization. In the speed planning stage, the maximum moving speed and acceleration of the instrument are limited to ensure the smoothness and safety of the surgical operation.

[0043] In a specific embodiment, the process of executing step S105 may specifically include the following steps: (1) Sample the optimal path data according to the time window, calculate the position deviation of the sampling points, and form real-time deviation data; (2) Perform cumulative error statistics on real-time deviation data, perform trend prediction on the statistical results, and obtain monitoring data; (3) Decompose the monitoring data into multi-dimensional features, perform correlation analysis on the decomposition results, and obtain the risk feature sequence; (4) Score the risk level of the risk feature sequence and add up the score results to form the analysis result; (5) Compare the analysis results with the preset threshold sequence, classify the risk level of the exceeded data, and obtain early warning data; (6) Conduct spatiotemporal correlation analysis on the early warning data, perform comprehensive probability calculation on the correlation results, and form risk data.

[0044] Specifically, the optimal path data is sampled and processed. The optimal path data contains the ideal motion trajectory of the surgical instrument. The sampling process discretizes these data according to a fixed time window. The size of the time window is set according to the accuracy requirements of the surgical operation, usually in milliseconds. At each sampling point, two sets of data, the actual position and the ideal position, are recorded at the same time, and the position deviation is obtained by calculating the Euclidean distance between the two sets of data. The calculation of the position deviation takes into account the two aspects of translation and rotation in three-dimensional space to form real-time deviation data. Cumulative error statistics of real-time deviation data is an important means of monitoring long-term trends. The statistical process uses a sliding window method to calculate the statistical characteristics of the mean, variance, and maximum value of the deviation within a certain time range. These statistics reflect the stability and accuracy of the surgical operation. Trend prediction uses a time series analysis method to predict the deviation change trend in the future by establishing an autoregressive model. The prediction results are combined with historical data to form monitoring data.

[0045] The decomposition process uses the principal component analysis method to project the original data into different feature spaces. The feature space includes multiple dimensions such as position deviation, speed fluctuation, and acceleration change. The results of the decomposition are subjected to correlation analysis, the correlation coefficients between different dimensions are calculated, and the main risk factors are identified. The results of the correlation analysis are sorted according to the importance, and the most representative feature combinations are extracted to form a risk feature sequence. The multi-index evaluation method is used to score the risk feature sequence. The scoring process considers multiple aspects such as the severity, duration, and rate of change of the feature. A weight coefficient is set for each feature, and a comprehensive score is obtained by weighted summation. The scoring results are normalized to facilitate comparison under different circumstances. The numerical accumulation process takes into account the time decay factor, and the more recent the risk event, the greater the weight, to form the analysis result.

[0046] When comparing the analysis results with the preset threshold sequence, a multi-level threshold judgment method is used. The threshold sequence contains critical values ​​corresponding to multiple risk levels, which are determined based on surgical safety specifications and expert experience. The data exceeding the threshold is divided into risk levels. The division process considers the urgency and degree of risk, divides the risk into different levels, and obtains early warning data. When performing spatiotemporal correlation analysis on early warning data, it is necessary to consider the distribution characteristics of risk events in time and space at the same time. Temporal correlation analysis focuses on the occurrence sequence and time interval of risk events, while spatial correlation analysis examines the distribution law of risk events in the surgical area. Correlation analysis uses graph theory to construct a correlation network of risk events. The propagation characteristics of risks are evaluated by calculating the importance index and connection strength of nodes in the network. Finally, through comprehensive probability calculation, the influence of multiple risk factors is integrated to form risk data.

[0047] For example, in a spinal surgery, the motion trajectory of the surgical instrument is first monitored in real time. The deviation between the instrument position and the preset path is recorded by sampling the position every 10 milliseconds. In the cumulative error statistics stage, a 60-second sliding window is used to calculate the statistical features, and the deviation trend in the next 10 seconds is predicted. The feature decomposition process identifies key risk factors including position deviation, posture error, and speed fluctuation. In the risk scoring stage, different weights are set for different types of deviations, with a position deviation weight of 0.4, a posture error weight of 0.3, and a speed fluctuation weight of 0.3. The warning threshold is set at three levels: when the comprehensive score exceeds 0.7, it is high risk, between 0.4 and 0.7, it is medium risk, and below 0.4, it is low risk. The spatiotemporal correlation analysis found that the position deviation of the instrument when it approaches the spine is significantly correlated with the speed fluctuation, which directly affects the determination of the risk level.

[0048] In a specific embodiment, the process of executing step S106 may specifically include the following steps: (1) Risk data is divided into time series according to the surgical stage, and the division results are subjected to multi-scale time window sliding. Variance analysis and peak detection are performed based on the data volatility within the sliding window. The risk change law is extracted through the data distribution characteristics, and the change trend is quantified to obtain risk characteristic data; (2) Perform multi-level decomposition processing on risk characteristic data, conduct periodic analysis on the decomposed characteristic data through autocorrelation function, divide the data into groups according to the periodic results, perform pattern matching based on the similarity between data groups, perform numerical statistics and normalization processing on the matching results, and form regular data; (3) Construct a multidimensional feature space for the regular data, perform density distribution analysis on the data points in the feature space, cluster the data according to the density distribution results, calculate the dimension correlation based on the distribution characteristics of the cluster centers, sort the calculation results by feature importance, and obtain dimension correlation data; (4) Perform multi-layer cross-validation on dimension-related data, perform uncertainty analysis on the validation results through information entropy calculation, evaluate the reliability of the data according to the uncertainty level, quantitatively describe the data association strength through the evaluation results, normalize the description results numerically, and obtain the association strength data; (5) Construct a relational network based on the correlation strength data, calculate the weight of the connection strength between network nodes, extract the key path based on the weight distribution, characterize the data transmission characteristics through path analysis, perform mathematical modeling on the characterization results, and form mapping relational data; (6) Perform semantic structuring on the mapping relationship data, organize the structured results through hierarchical analysis, extract knowledge units according to the organizational structure, construct rules based on the logical relationship between knowledge units, verify the validity of the construction results, and form the target knowledge data.

[0049] Specifically, the time series division and feature extraction of risk data are carried out. The division of surgical stages is based on the different properties of surgical operations, including key stages such as incision, separation, and suturing. In each stage, a multi-scale time window is used for sliding analysis, and the window size varies from seconds to minutes to capture the risk change characteristics at different time scales. In each sliding window, the variance of the data is calculated to measure the degree of fluctuation of the risk level, and peak detection is performed to identify risk emergencies. The data distribution characteristic analysis includes the calculation of statistics such as mean, standard deviation, skewness, and kurtosis, which together describe the regular characteristics of risk changes. The numerical fitting method is used to quantitatively process the change trend to obtain risk feature data. The risk feature data is decomposed at multiple levels. The decomposition process uses the wavelet transform method to decompose the data into different frequency components. The decomposed feature data is subjected to autocorrelation analysis, and the autocorrelation coefficients under different time delays are calculated to discover the periodic patterns in the data. According to the results of the periodic analysis, the data is divided into groups, and data with similar periods are classified into the same group. Within each data group, pattern matching is performed by calculating the similarity between data sequences. The similarity calculation uses a dynamic time warping algorithm. The matching results are statistically analyzed and normalized by maximum and minimum values ​​to form regular data.

[0050] Construct a multidimensional feature space. In the feature space, each dimension represents a risk feature, and the distribution of data points reflects the relationship between different features. Perform density distribution analysis on the data points, and use the kernel density estimation method to calculate the data density in the space. Cluster the data according to the density distribution, and use the density peak clustering algorithm to determine the cluster center. By analyzing the distribution characteristics of the cluster center, calculate the correlation between different dimensions, and the correlation calculation is based on the mutual information criterion. Finally, sort the feature importance to obtain the dimension correlation data. Divide the data set into multiple subsets, and use the K-fold cross-validation method to evaluate the stability of the correlation relationship. Calculate the information entropy of the verification results to evaluate the uncertainty level of the data. According to the results of the uncertainty analysis, quantitatively evaluate the reliability of the data, and quantitatively describe the data association strength. The description results are normalized to ensure the comparability between different data and form association strength data.

[0051] The nodes in the network represent different risk factors, and the edges represent the associations between the factors. The connection strengths between network nodes are weighted, and the weight values ​​reflect the importance of the associations. By analyzing the weight distribution, the key paths in the network are extracted, and these paths reflect the main channels for risk transmission. The data transmission characteristics are mathematically modeled. The modeling process considers the direct and indirect effects between nodes to form mapping relationship data. Semantic structured processing is performed. The mapping relationship data is converted into understandable knowledge representation, and the ontology model is used to organize the knowledge. The subordinate relationships between knowledge units are determined through hierarchical analysis to establish a knowledge system. According to the logical relationship between knowledge units, reasoning rules are constructed. The rules include causal relationships, constraints, etc. The validity of the constructed rules is verified. The verification process includes logical consistency checking and instance testing to form target knowledge data.

[0052] For example, in liver tumor surgery, the entire surgical process is divided into preoperative planning, navigation positioning, instrument operation, suture and finishing stages. In the instrument operation stage, three sliding windows of 30 seconds, 1 minute and 5 minutes are used for risk monitoring. Through variance analysis, it is found that the risk fluctuates greatly during the vascular separation operation, and peak detection identifies several high-risk events when approaching important blood vessels. After periodic analysis of these risk data, periodic risk change patterns related to respiratory movement are found. In the feature space analysis, the operation position, instrument posture, movement speed and other features are correlated and analyzed, and it is found that the instrument posture has a strong correlation with the operation risk. Through the relationship network analysis, the risk propagation chain of "improper posture-position deviation-tissue damage" is identified. These findings are organized into structured knowledge to form a risk prevention and control guide for liver surgery, including specific rules such as instrument posture control requirements and safe operation boundaries.

[0053] In a specific embodiment, the process of performing the step of multi-dimensional feature decomposition of monitoring data may specifically include the following steps: (1) Perform spatial decomposition of the monitoring data through orthogonal transformation, calculate the main direction component based on the decomposed data, and obtain the characteristic component data; (2) Reconstruct the characteristic component data into a sequence using the time window, and extract the fluctuation characteristic data by the fluctuation law; (3) Calculate the spectrum distribution based on the fluctuation characteristic data, classify the frequency bands according to the energy size, and form frequency domain characteristic data; (4) Perform correlation analysis on the frequency domain feature data through the cross-correlation function, select relevant parameters based on the analysis results, and obtain correlation data; (5) Establish feature significance index based on correlation data, extract key features through statistical test, and obtain feature index data; (6) Organize the feature index data based on time sequence and form a risk feature sequence after reordering the data.

[0054] Specifically, spatial decomposition processing is performed. Orthogonal transformation is the process of projecting high-dimensional data onto an orthogonal basis. Common methods include principal component analysis (PCA) and singular value decomposition (SVD). Monitoring data usually contains information in multiple dimensions such as position, speed, and posture. The main change direction in the data can be identified through orthogonal transformation. In the decomposition process, the data is first centered and standardized, and then the covariance matrix is ​​calculated, and the main direction is obtained through eigenvalue decomposition. The main direction is sorted according to the size of its corresponding eigenvalue. The eigenvector corresponding to the larger eigenvalue represents the main change direction of the data, and the characteristic component data is obtained accordingly. The time window reconstruction process adopts the sliding window method. The characteristic component data is segmented according to a fixed time length. There is overlap between adjacent windows, and the degree of overlap affects the continuity of the data. In each time window, the change trend of the data is analyzed and the fluctuation law is extracted. The extraction of fluctuation law includes multiple aspects such as mean drift, variance change, and mutation point detection. Through statistical analysis of these features, fluctuation characteristic data is formed.

[0055] The Fourier transform method is used to calculate the spectrum distribution. The fluctuation feature data is converted from the time domain to the frequency domain to obtain the energy distribution of different frequency components. Spectral analysis represents the periodic characteristics and main frequency components in the data. The frequency bands are classified according to the energy size, and the spectrum is divided into high energy area, medium energy area and low energy area, which correspond to the fluctuation features of different scales, respectively, to form frequency domain feature data. Cross-correlation analysis is used to evaluate the correlation between different features. The cross-correlation function is calculated for the frequency domain feature data in pairs to obtain parameters such as correlation coefficient and time delay. The correlation analysis results reflect the dependency and influence strength between the features. Parameters are screened according to the size of the correlation coefficient, and the significantly correlated feature pairs are retained to obtain correlation data. The feature significance index is established based on the statistical test method. Hypothesis tests are performed on the correlation data to evaluate the statistical significance of the feature correlation. The significance test uses the t-test or F-test to determine the importance of the feature based on the test results. By setting the significance level, key features are screened out to form feature index data.

[0056] Finally, the temporal organization of data is performed. The feature index data is arranged in chronological order to establish the temporal relationship between features. The reordering process considers the causal relationship and logical order of the features to ensure the coherence of the data. After the reordering process, a risk feature sequence is formed.

[0057] For example, in a neurosurgery operation, the monitoring data includes information such as the three-dimensional position coordinates, posture angle, and operating force of the surgical instrument. Through orthogonal transformation, it was found that the movement of the instrument was mainly concentrated in a certain plane, which was related to the anatomical structure of the surgical target area. Time window analysis showed that there were obvious periodic fluctuations in the movement of the instrument, which was related to the doctor's operating habits and the patient's physiological rhythm. Spectral analysis further confirmed this, and a significant energy peak was found in the 0.2-0.5Hz frequency band, corresponding to the main rhythm of the surgical operation. Cross-correlation analysis found that there was a significant positive correlation between the instrument posture and the operating force, with a time delay of about 200 milliseconds, which reflects the relationship between force control and posture adjustment. Through statistical tests, posture angle, movement speed, and operating force were determined as key monitoring indicators. The indicators are organized in chronological order to form a risk monitoring sequence.

[0058] In a specific embodiment, the process of performing correlation analysis on the frequency domain feature data through the cross-correlation function may specifically include the following steps: (1) Establish a cross-correlation matrix based on the frequency domain feature data, normalize the matrix elements, and form correlation distribution data; (2) Extracting the eigenvalue sequence from the associated distribution data, sorting the eigenvalues ​​according to their numerical values, and obtaining the feature sorting data; (3) Perform threshold segmentation on the feature sorting data, calculate statistics through the segmentation interval, and obtain statistical feature data; (4) Project the statistical feature data into the feature space, perform cluster analysis through distance measurement, and form feature cluster data; (5) Construct a correlation coefficient matrix based on the feature clustering data, extract key parameters based on the matrix elements, and obtain parameter correlation data; (6) Conduct correlation evaluation based on parameter correlation data, and comprehensively calculate the evaluation indicators to form correlation data.

[0059] Specifically, the frequency domain feature data contains the energy distribution information of different frequency bands. The correlation coefficients of these data are calculated pairwise to form a symmetric matrix. Each element of the matrix represents the correlation strength between two frequency bands. When normalizing the matrix elements, the maximum and minimum value normalization method is used to unify the value range to the [0,1] interval. The normalized matrix reflects the relative correlation strength between the frequency bands and forms the correlation distribution data.

[0060] Eigenvalue extraction is the process of eigendecomposing the associated distribution data. By solving the characteristic equation, the eigenvalues ​​and eigenvectors of the matrix are obtained. The eigenvalues ​​reflect the variance of the data in each main direction, and the eigenvectors represent these main directions. The eigenvalues ​​are arranged in order from large to small, and the eigenvectors corresponding to the larger eigenvalues ​​represent the main change patterns. The sorted eigenvalue sequence constitutes the feature sorting data. The threshold segmentation process is to perform stratified processing on the feature sorting data. According to the distribution characteristics of the eigenvalues, appropriate thresholds are set to divide the data into different levels. In each segmentation interval, statistics such as mean, standard deviation, and kurtosis are calculated. These statistics describe the distribution characteristics of the data in the interval. By combining multiple statistics, statistical feature data is formed.

[0061] Feature space projection is the process of mapping statistical feature data to a multidimensional feature space. In the feature space, each dimension represents a statistical feature, and the position of the data point reflects its feature combination. Cluster analysis is performed by calculating the Euclidean distance or Mahalanobis distance between data points. The clustering process uses K-means or hierarchical clustering algorithms to classify similar data points into the same category to form feature clustered data.

[0062] The construction of the correlation coefficient matrix is ​​based on the feature clustering results. For each cluster, the correlation coefficients between the features are calculated to form a local correlation matrix. At the same time, the correlation relationship between different clusters is considered to construct a global correlation matrix. Key parameters are extracted from the matrix, including the size, sign and significance level of the correlation coefficient, to obtain parameter association data.

[0063] The parameter correlation data is used as input and comprehensively evaluated through multiple evaluation indicators. The evaluation indicators include correlation strength, stability and interpretability. These indicators are weighted and the weight value reflects the importance of each indicator. The result of the comprehensive calculation is the correlation data.

[0064] For example, in liver tumor surgery, the frequency domain feature data contains the spectral characteristics of surgical instrument movement. When establishing the cross-correlation matrix, it was found that there was a significant correlation between the 0.2Hz-0.5Hz frequency band and the 1Hz-2Hz frequency band, which reflects the coordination between slow movement and rapid adjustment in surgical operations. Eigenvalue analysis shows that the first three principal components can explain most of the data variation, indicating that surgical operations have obvious pattern characteristics. Through cluster analysis, three typical operation modes were identified: fine dissection, tissue separation, and suturing operations. In each operation mode, the movement characteristics of the instrument present different correlation structures.

[0065] The above describes the surgical minimally invasive surgery data processing method based on AR technology in the embodiment of the present application. The following describes the surgical minimally invasive surgery data processing system based on AR technology in the embodiment of the present application. Figure 2 In the embodiment of the present application, an embodiment of a surgical minimally invasive surgery data processing system based on AR technology includes: The reconstruction module 201 is used to perform three-dimensional reconstruction processing on the medical image data, perform lesion segmentation on the reconstructed data, process the segmented data by feature extraction, and obtain three-dimensional feature point data; The acquisition module 202 is used to acquire and process the position of the surgical instrument using the three-dimensional feature point data, spatially map the acquired position data, and register and fuse the mapped data to form spatial positioning data; A processing module 203 is used to reconstruct light field data according to the three-dimensional feature point data and the spatial positioning data, perform deformation processing on the reconstructed data, and combine the processed data with the real scene data to obtain augmented reality data; An analysis module 204 is used to analyze the operation trajectory according to the augmented reality data, perform path planning on the analyzed data, and obtain optimal path data through optimization processing; The monitoring module 205 is used to monitor the optimal path data in real time, perform risk analysis on the monitoring data, perform early warning processing on the analysis results through threshold judgment, and obtain risk data; The module 206 is established to extract surgical rules using the risk data, perform multi-dimensional analysis on the rule data, establish a mapping relationship between the data through association processing, and form target knowledge data.

[0066] Through the collaboration of the above components, significant technical effects have been achieved through data processing in multiple links: by performing 3D reconstruction and lesion segmentation on medical image data, and combining feature extraction technology to obtain 3D feature point data, accurate positioning and feature expression of the surgical target area are achieved; the 3D feature point data is used to collect and spatially map the position of surgical instruments in real time, and spatial positioning data is formed through registration and fusion, ensuring the accuracy and real-time positioning of surgical instruments; light field data is reconstructed based on 3D feature point data and spatial positioning data, and deformation processing and scene fusion technology are combined to achieve the natural presentation of augmented reality scenes; operation trajectory analysis and path planning are performed based on augmented reality data, and optimal path data is obtained through optimization processing, which improves the accuracy and safety of surgical operations; real-time monitoring and risk analysis of optimal path data are performed, and early warning processing is achieved through threshold judgment to form risk data, which enhances the safety of the surgical process; finally, risk data is used to extract surgical rules, multi-dimensional analysis is performed, and data mapping relationships are established to form target knowledge data, realizing the effective accumulation of surgical experience and knowledge transformation. A complete technical chain from data collection, processing, analysis to knowledge extraction has been achieved, significantly improving the accuracy, safety and intelligence level of minimally invasive surgery.

[0067] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for processing surgical minimally invasive surgery data based on AR technology, characterized in that: The method for processing surgical minimally invasive surgery data based on AR technology includes: Perform three-dimensional reconstruction on medical image data, perform lesion segmentation on the reconstructed data, process the segmented data through feature extraction, and obtain three-dimensional feature point data; The three-dimensional feature point data is used to collect and process the position of the surgical instrument, the collected position data is spatially mapped, and the mapped data is registered and fused to form spatial positioning data; Reconstructing light field data according to the three-dimensional feature point data and the spatial positioning data, performing deformation processing on the reconstructed data, and combining the processed data with the real scene data to obtain augmented reality data; Performing operation trajectory analysis based on the augmented reality data, performing path planning on the analyzed data, and obtaining optimal path data through optimization processing; The optimal path data is monitored in real time, risk analysis is performed on the monitored data, and early warning processing is performed on the analysis results through threshold judgment to obtain risk data; The risk data is used to extract surgical rules, and multi-dimensional analysis is performed on the rule data. A mapping relationship between the data is established through association processing to form target knowledge data.

2. The method for processing surgical minimally invasive surgery data based on AR technology according to claim 1, characterized in that: The three-dimensional reconstruction of the medical image data, segmentation of the reconstructed data, and processing of the segmented data by feature extraction to obtain three-dimensional feature point data include: Projecting the grayscale values ​​in the medical image data into a three-dimensional space, and obtaining initial voxel data by pixel density threshold processing; Performing region growing processing on the initial voxel data, separating the target region according to the grayscale gradient value, and obtaining separated and reconstructed data; Performing connected domain analysis on the separated and reconstructed data through region marking, and performing boundary extraction processing on the connected domain to form reconstructed data; Extracting the lesion boundary from the reconstructed data using a regional segmentation method, performing feature enhancement processing according to the boundary curvature, and obtaining lesion segmentation data; Performing edge detection on the lesion segmentation data, extracting key point coordinates based on the detection results, and obtaining segmentation data; The feature point density is calculated for the segmented data, and the feature points are screened by using spatial distance constraints to obtain the three-dimensional feature point data.

3. The method for processing surgical minimally invasive surgery data based on AR technology according to claim 1, characterized in that: The method of collecting and processing the position of the surgical instrument using the three-dimensional feature point data, spatially mapping the collected position data, and registering and fusing the mapped data to form spatial positioning data includes: Projecting the three-dimensional feature point data into the surgical space, and performing coordinate calibration on the projection result to obtain calibration coordinate data; Optically tracking the position of the surgical instrument, performing spatial fixed-point processing on the tracking data, and obtaining fixed-point data; Converting the position parameters in the fixed-point data into three-dimensional space coordinates, and performing noise filtering on the coordinate data to form the collected position data; Performing coordinate system transformation on the collected position data, and projecting the transformed data into a calibration space to obtain the mapping data; Calculate the overlap between the mapping data and the calibration coordinate data, and perform error compensation processing on the calculation result to form registration data; The registration data is spatially registered, and the registration parameters are multi-dimensionally fused to obtain the spatial positioning data.

4. The method for processing surgical minimally invasive surgery data based on AR technology according to claim 1, characterized in that: The light field data is reconstructed according to the three-dimensional feature point data and the spatial positioning data, the reconstructed data is deformed, and the processed data is combined with the real scene data to obtain augmented reality data, including: Performing spatial registration on the three-dimensional feature point data and the spatial positioning data, and uniformly processing the registration results through coordinate transformation to obtain unified coordinate data; Performing ray tracing on the unified coordinate data, converting the tracing result into a brightness value distribution, and obtaining initial light field data; Performing density calculation on the initial light field data, and performing interpolation processing on the calculation results to form interpolated light field data; Performing tissue deformation calculation on the interpolated light field data, performing numerical iteration on the deformation parameters, and obtaining the reconstructed data; Performing surface fitting on the reconstructed data, and processing the fitting result by shape variable calculation to form the processed data; The processed data is depth aligned, and the alignment result is spatially fused with the real scene data to obtain the augmented reality data.

5. The method for processing surgical minimally invasive surgery data based on AR technology according to claim 1, characterized in that: The performing of operation trajectory analysis based on the augmented reality data, performing path planning on the analyzed data, and obtaining optimal path data through optimization processing includes: The augmented reality data is segmented in time series, and feature points are tracked on the segmented data to obtain spatiotemporal state data; Performing multi-level trajectory decomposition on the spatiotemporal state data, extracting trajectory points from the decomposition results, and forming a trajectory point sequence; Performing polynomial curve fitting on the trajectory point sequence, and performing tangent analysis on the fitting result to obtain the analysis data; Performing collision detection calculation on the analysis data, screening the detection results according to the safety distance, and forming path candidate data; Calculate the path length of the candidate path data, solve the calculation result with an energy function, and obtain path scoring data; Performing multi-objective weight calculation on the path scoring data, and performing multiple iterations of comparison on the path weights to form the optimal path data; Performing speed planning on the optimal path data, performing acceleration constraint calculation on the planning result, and obtaining motion control data; The motion control data is subjected to trajectory smoothing, the smoothing result is subjected to gradient optimization, and the optimal path data is updated.

6. The method for processing minimally invasive surgical data based on AR technology according to claim 1, characterized in that: The method of monitoring the optimal path data in real time, performing risk analysis on the monitoring data, and performing early warning processing on the analysis results through threshold judgment to obtain risk data includes: The optimal path data is sampled according to the time window, and the position deviation of the sampling points is calculated to form real-time deviation data; Performing cumulative error statistics on the real-time deviation data, and performing trend prediction on the statistical results to obtain the monitoring data; Decomposing the monitoring data into multi-dimensional features, performing correlation analysis on the decomposition results, and obtaining a risk feature sequence; Performing risk scoring on the risk feature sequence, and numerically accumulating the scoring results to form the analysis result; Compare the analysis results with the preset threshold sequence, classify the over-limit data into risk levels, and obtain early warning data; Performing a spatiotemporal correlation analysis on the warning data, performing a comprehensive probability calculation on the correlation results, and forming the risk data.

7. The method for processing minimally invasive surgical data based on AR technology according to claim 1, characterized in that: The method of extracting surgical rules by using the risk data, performing multi-dimensional analysis on the rule data, establishing a mapping relationship between the data through association processing, and forming target knowledge data includes: The risk data is divided into time series according to the surgical stage, a multi-scale time window is sliding on the division result, variance analysis and peak detection are performed according to the data volatility in the sliding window, the risk change law is extracted through the data distribution characteristics, the change trend is quantified, and the risk characteristic data is obtained; Perform multi-level decomposition processing on the risk characteristic data, perform periodic analysis on the decomposed characteristic data through an autocorrelation function, divide the data into groups according to the periodic results, perform pattern matching through the similarity between data groups, perform numerical statistics and normalization processing on the matching results, and form the regular data; Constructing a multidimensional feature space for the regular data, performing density distribution analysis on the data points in the feature space, clustering the data according to the density distribution results, calculating dimensional relevance through the distribution characteristics of the cluster centers, sorting the calculation results by feature importance, and obtaining dimension-related data; Perform multi-layer cross-validation on the dimension-related data, perform uncertainty analysis on the validation results through information entropy calculation, perform reliability assessment on the data according to the uncertainty level, quantitatively describe the data association strength through the assessment results, perform numerical normalization on the description results, and obtain association strength data; The association strength data is used to construct a relational network, the connection strength between network nodes is weighted, key paths are extracted according to the weight distribution, data transmission characteristics are characterized by path analysis, mathematical modeling is performed on the characterization results, and mapping relationship data is formed; The mapping relationship data is semantically structured, the structured results are organized through hierarchical analysis, knowledge units are extracted according to the organizational structure, rules are constructed through the logical relationship between knowledge units, the construction results are verified for validity, and the target knowledge data is formed.

8. The method for processing minimally invasive surgical data based on AR technology according to claim 6, characterized in that: The monitoring data is subjected to multi-dimensional feature decomposition, and correlation analysis is performed on the decomposition results to obtain a risk feature sequence, including: Decomposing the monitoring data spatially by orthogonal transformation, calculating the main direction component according to the decomposed data, and obtaining characteristic component data; Reconstructing the characteristic component data into a sequence using a time window, and extracting the fluctuation characteristic data by fluctuation law; Calculating the frequency spectrum distribution according to the fluctuation characteristic data, classifying the frequency bands according to the energy size, and forming frequency domain characteristic data; Perform correlation analysis on the frequency domain feature data through a cross-correlation function, and select relevant parameters according to the analysis results to obtain correlation data; Establishing a feature significance index based on the correlation data, extracting key features through statistical testing, and obtaining feature index data; The feature index data is organized based on the time sequence, and the risk feature sequence is formed after the data is reordered.

9. The method for processing minimally invasive surgical data based on AR technology according to claim 8, characterized in that: The method of performing correlation analysis on the frequency domain feature data by using a cross-correlation function, and screening relevant parameters according to the analysis results to obtain correlation data includes: Establish a cross-correlation matrix based on the frequency domain feature data, normalize the matrix elements, and form correlation distribution data; Extracting a characteristic value sequence from the associated distribution data, sorting the characteristic values ​​according to their numerical values, and obtaining characteristic sorting data; Performing threshold segmentation on the feature sorting data, calculating statistics through the segmentation intervals, and obtaining statistical feature data; Projecting the statistical feature data into a feature space, performing cluster analysis through distance measurement, and forming feature cluster data; Constructing a correlation coefficient matrix according to the characteristic clustering data, extracting key parameters according to matrix elements, and obtaining parameter correlation data; A correlation evaluation is performed based on the parameter association data, and the evaluation indicators are comprehensively calculated to form the correlation data.

10. A minimally invasive surgery data processing system based on AR technology, used to implement the minimally invasive surgery data processing method based on AR technology as described in any one of claims 1 to 9, characterized in that: The minimally invasive surgery data processing system based on AR technology includes: The reconstruction module is used to perform three-dimensional reconstruction on the medical image data, segment the lesions on the reconstructed data, process the segmented data by feature extraction, and obtain three-dimensional feature point data; An acquisition module is used to acquire and process the position of the surgical instrument using the three-dimensional feature point data, spatially map the acquired position data, and register and fuse the mapped data to form spatial positioning data; A processing module, used to reconstruct light field data according to the three-dimensional feature point data and the spatial positioning data, perform deformation processing on the reconstructed data, and combine the processed data with the real scene data to obtain augmented reality data; An analysis module is used to analyze the operation trajectory according to the augmented reality data, perform path planning on the analyzed data, and obtain the optimal path data through optimization processing; A monitoring module is used to monitor the optimal path data in real time, perform risk analysis on the monitoring data, perform early warning processing on the analysis results through threshold judgment, and obtain risk data; A module is established to extract surgical rules using the risk data, perform multi-dimensional analysis on the rule data, establish a mapping relationship between the data through association processing, and form target knowledge data.

Images