Intraoperative multi-parameter analysis decision and auxiliary early warning system based on artificial intelligence

Abstract

The invention relates to the technical field of computer-aided surgical operations, and discloses an intraoperative multi-parameter analysis decision and auxiliary early warning system based on artificial intelligence, and the system employs a visual perception module to extract semantic masks and feature point displacement of an operation video stream; and a dynamic stiffness tensor field is inverted based on contact micro-deformation through an in-situ parameter calibration module. Wherein the neural network processor performs hardware-level optimization on tensor calculation, and is configured to convert a partial differential equation for controlling soft tissue deformation into a static calculation graph, and directly solve a global displacement vector meeting physical constraints by randomly sampling collocation points in a calculation domain. And the early warning decision module updates the space coordinates of the hidden structure, calculates the dynamic risk distance between the surgical instrument and the hidden structure and generates an early warning signal. According to the method, real-time physical solution of soft tissue deformation is realized through a heterogeneous computing architecture, and the real-time performance and accuracy of hidden structure navigation in minimally invasive surgery are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer-aided surgery, in particular to an intraoperative multi-parameter analysis decision and auxiliary warning system based on artificial intelligence. BACKGROUND

[0002] Minimally invasive surgery is operated through visual information provided by an endoscope, which reduces surgical trauma but also limits the field of vision and tactile perception of the surgeon. In complex abdominal or thoracic surgery, key anatomical structures such as blood vessels, nerves or tumors are often covered by surface soft tissue, belonging to hidden structures. In order to assist the surgeon in positioning, the existing surgical navigation system usually constructs a three-dimensional model using the preoperative CT or MRI image of the patient, and superimposes it on the intraoperative video stream.

[0003] However, biological soft tissue has a high degree of nonlinear deformation characteristics. During the operation, the soft tissue morphology will change significantly due to the influence of respiratory motion, instrument traction and gravity. This deformation causes the preoperative static anatomical model to deviate from the actual anatomical position during the operation, and even a small positional drift can lead to navigation failure, and even mislead the surgeon to cause serious complications such as blood vessel rupture. Therefore, real-time and accurate calculation of the soft tissue deformation field is the key to realizing reliable augmented reality navigation.

[0004] Current soft tissue deformation simulation mainly relies on the finite element method (FEM) or the mass point spring model (MSM). The mass point spring model has a faster calculation speed, but its physical accuracy is lower, and it is difficult to truly reflect the mechanical behavior of continuous media. Although the finite element method has high physical accuracy, its calculation process involves the assembly of large-scale global stiffness matrix and the solution of linear equations, and the computational complexity increases exponentially with the number of grids, making it difficult to achieve real-time solution that meets the operation frame rate on conventional computing hardware. In addition, existing methods usually use a general biomechanical parameter library, and do not fully consider the differences in tissue hardness between individual patients, and lack a mechanism for dynamically correcting physical parameters using observed data during the operation, further limiting the accuracy of deformation prediction. Existing deep learning methods mostly focus on semantic segmentation or optical flow tracking of surface images, and are difficult to directly infer the physical state of internal invisible areas, and usually lack the constraint of physical laws, resulting in unreliable prediction results in mechanics. SUMMARY

[0005] In view of the deficiencies of the prior art, the present application provides an intraoperative multi-parameter analysis decision and auxiliary warning system based on artificial intelligence, which aims to solve the technical problems of hidden anatomical structure position drift caused by soft tissue deformation in minimally invasive surgery, and the difficulty of traditional physical simulation to meet real-time interaction.

[0006] The system comprises an image acquisition device, a heterogeneous computing processing device and a human-computer interaction terminal device connected in communication. The heterogeneous computing processing device integrates a central processing unit, a graphics processing unit and a neural network processing unit at a hardware level, and runs a visual perception and semantic analysis module, an in-situ parameter calibration module, a neural physical computing module and a pre-warning decision module that work cooperatively.

[0007] The visual perception and semantic analysis module is configured on the graphics processing unit and is used for processing real-time video streams of a surgical area. The module generates a pixel-level semantic mask through convolutional neural network inference to distinguish different biological tissues and surgical instruments. Meanwhile, the module extracts sparse feature point displacement of a tissue surface using an optical flow method or a simultaneous localization and mapping technology, and projects two-dimensional feature points combined with depth information to a three-dimensional space to establish a surface observation displacement field, thereby providing a dynamic boundary condition for subsequent physical computing.

[0008] The in-situ parameter calibration module is used for performing parameter inversion based on a micro-deformation gradient. The module monitors contact events between a surgical instrument and soft tissue, and determines effective interaction by calculating a motion correlation coefficient of a speed vector of an instrument tip and a speed vector of a tissue surface. After locking a local region of interest, a target function including a data assimilation loss and a physical equation residual is constructed, a gradient optimization algorithm is used to solve a local stiffness correction coefficient, and the correction coefficient is broadcast to all grid elements with the same semantic label based on a semantic mask. This process uses deformation data generated by actual contact during surgery to real-time correct a baseline stiffness tensor field constructed based on general statistical data, and generates a dynamic stiffness tensor field conforming to individual characteristics of a patient.

[0009] The neural physical computing module is configured on the neural network processing unit and is used for performing full-field deformation calculation. The neural network processing unit is hardware-optimized for tensor calculation, and is configured to convert a partial differential equation controlling soft tissue deformation into a static computation graph, and directly solve the full-field displacement vector satisfying physical constraints by randomly sampling collocation points in the calculation domain.

[0010] Specifically, the system calculates a strain tensor and a stress divergence of a network output using automatic differentiation technology, constructs a physical residual term conforming to a static equilibrium equation, combines a boundary condition loss term of surface observation data, and trains the network by minimizing a composite loss function, thereby outputting a continuous displacement field covering internal hidden areas without grid reconstruction. In addition, the module adopts a heterogeneous pipeline parallel strategy, merges matrix operations and activation functions into a single hardware kernel using operator fusion technology, and realizes asynchronous execution of data preparation and tensor operation through a double buffering mechanism.

[0011] The pre-warning decision module is used for applying a global displacement vector to a preoperatively constructed hidden structure grid model to perform Lagrange position updating on the vertexes of the hidden structure grid model.

[0012] In addition, the heterogeneous computing processing device is further configured to perform preoperative anatomical modeling and reference physical field initialization.

[0013] The application provides an intraoperative multi-parameter analysis decision and auxiliary pre-warning system based on artificial intelligence. 1. The application converts partial differential equations for controlling soft tissue deformation into a static calculation graph through hardware-level optimization of a neural network processor for tensor calculation, and directly solves displacement vectors by using a physical information neural network.

[0014] 2. The application extracts micro-deformation gradients generated by the contact between surgical instruments and tissues by using a visual perception module, identifies the tissue categories of the contact areas by combining semantic masks, and automatically inverts and updates local stiffness correction coefficients by minimizing physical equation residuals and observation data errors.

[0015] 3、The application uses the global displacement vector to update the grid vertex coordinates of internal structures such as blood vessels or tumors in real time, and uses depth-based transparent rendering to superimpose them on the endoscope video stream, solving the problem that simple image superposition cannot follow tissue deformation. At the same time, by dynamically updating the hierarchical bounding volume tree structure and calculating the real-time Euclidean distance between the tip of the surgical instrument and the hidden structure, the system can trigger a hierarchical prompt according to the preset safety threshold, effectively assisting the doctor to avoid key anatomical structures and reducing the risk of damage in minimally invasive surgery. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 It is a schematic diagram of the hardware architecture and module connection of the system of the application. Figure 2 It is a flowchart of the intraoperative multi-parameter analysis decision and auxiliary early warning method based on artificial intelligence of one embodiment of the application.

[0017] Among them, 10, image acquisition device; 20, heterogeneous computing processing device; 21, central processing unit; 22, graphics processing unit; 23, neural network processing unit; 24, storage unit; 30, human-computer interaction terminal device; 201, visual perception and semantic analysis module; 202, in-situ parameter calibration module; 203, neuro-physical computing module; 204, early warning decision module. DETAILED DESCRIPTION

[0018] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the specification of the application. Obviously, the described embodiments are only part of the embodiments of the application, not all. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the application.

[0019] Referring to the drawings Figure 1 The system is based on a heterogeneous computing architecture for real-time analysis of soft tissue deformation and provides hidden structure position warning, mainly including a communication-connected image acquisition device 10, a heterogeneous computing processing device 20 and a human-computer interaction terminal device 30.

[0020] The image acquisition device 10 (for example, a high-resolution endoscope configured with a binocular or structured light sensor) is used to acquire real-time video streams of the surgical area containing depth information and biological tissue texture. The human-computer interaction terminal device 30 is used to output augmented reality images superimposed with hidden structure projections and hierarchical audio alarm signals.

[0021] The heterogeneous computing processing device 20 serves as a core operation unit, and is integrated with a central processor 21, a graphics processor 22, a neural network processor 23 (NPU), and a storage unit 24 which work cooperatively through a high-speed bus at a hardware level. The graphics processor 22 is configured to perform high-parallelism matrix operations such as image preprocessing, semantic segmentation inference, and feature extraction. The neural network processor 23 is hardware-optimized for tensor calculation and is configured to accelerate the forward and backward propagation of a physics information based neural network (PINN) to solve a nonlinear deformation field within a millisecond level. The storage unit 24 is used to store preoperative images, three-dimensional models, and network weight data.

[0022] At a functional logic level, the heterogeneous computing processing device 20 runs a plurality of cooperative modules: a visual perception and semantic analysis module 201 deployed on the graphics processor 22, which is used to analyze video streams to generate semantic masks distinguishing tissue types and track surface feature points; an in-situ parameter calibration module 202, which is used to monitor contact events between instruments and tissues, calculate micro-deformation gradients to correct and generate a dynamic stiffness tensor field; a neural physics computing module 203 deployed on the neural network processor 23, which is used to receive the stiffness tensor field as a constraint and output a global displacement vector containing the location of hidden structures by solving physical equations; and a warning decision module 204, which is used to calculate a dynamic risk distance in combination with the displacement vector and instrument position, and generate a warning control signal accordingly.

[0023] Referring to the accompanying drawings Figure 2 The method is mainly executed by the heterogeneous computing processing device 20 through the cooperative modules, and specifically includes: Step S100: preoperative anatomical modeling and reference physical field initialization. The system constructs a discretized volumetric mesh model containing internal nodes based on the preoperative images (CT or MRI) of the patient. According to histological statistical data, each node is assigned an initial semantic label and reference biomechanical parameters: uniform modulus is assigned to isotropic regions (such as fat); for anisotropic regions (such as blood vessels, muscles), the geometric orientation is combined to construct a transverse isotropic reference stiffness tensor, forming an initial reference stiffness tensor field.

[0024] Step S200: intraoperative visual perception and semantic analysis. The visual perception and semantic analysis module 201 performs CNN inference on real-time surgical video frames to generate pixel-level semantic masks distinguishing between tissues and instruments. At the same time, sparse surface feature point displacements are extracted using an optical flow method or SLAM, and are mapped to the surface of the three-dimensional mesh model to establish surface observation point cloud data.

[0025] Step S300: In-situ parameter dynamic calibration based on micro-deformation gradient. The in-situ parameter calibration module 202 monitors the non-cutting contact between the instrument and the tissue. Within the contact region, the observed strain energy density index based on visual measurement is compared with the simulated strain energy density index based on the benchmark parameters and the NPU forward calculation. The specific correction coefficient is calculated using the ratio of the two, and the specific correction coefficient is broadcast to all similar tissue nodes in the field according to the semantic mask to generate a calibrated dynamic stiffness tensor field to correct the parameter deviation caused by individual differences.

[0026] Step S400: Full-field deformation calculation based on neuro-physical field. The neuro-physical calculation module 203 inputs the dynamic stiffness tensor field as a constraint into the physics information neural network (PINN). On the NPU, the global displacement field covering the internal invisible area is solved and output by minimizing the loss function containing three parts (physical residual term satisfying the stress balance equation, assimilation term matching surface observation data, and bone boundary constraint term).

[0027] Step S500: Dynamic mapping and early warning decision of hidden structure. The early warning decision module 204 uses the global displacement field to update the spatial coordinates of the hidden structure in real time and performs AR projection display. At the same time, the dynamic distance between the instrument tip and the hidden structure is calculated, and the collision risk is judged in combination with the direction of the instrument velocity vector. When the distance is less than the threshold value and there is an approaching trend, a hierarchical visual or audio alarm is triggered.

[0028] In the preoperative preparation stage or the surgical initialization stage, the heterogeneous computing processing device 20 performs an anatomical modeling task, which aims to convert discrete two-dimensional medical images into a continuous medium mechanical model suitable for physical calculation. The process specifically includes the following steps: Step S111: Image data acquisition and region of interest extraction. The heterogeneous computing processing device 20 reads the medical digital imaging and communication format data sequence of the patient's abdomen. The system uses a preset tissue density threshold interval (for example, based on Hounsfield units) in combination with a region growing algorithm to segment the soft tissue, bone, and tubular structure in the target surgical region from the image data. For noise or artifacts present in the image, the system uses Gaussian filtering for smoothing. The specific image preprocessing algorithm of this step belongs to the conventional technical means in the art, and will not be described here. The system defines the geometric boundary of the surgical region of interest and generates an initial voxel data set.

[0029] Step S112: Iso-surface extraction and surface reconstruction. Based on the voxel data set, the system uses the marching cubes algorithm to extract the iso-surface of each tissue interface and generate a set of triangular patches describing the outer contour of the organ and tissue. In this process, the system performs Laplacian smoothing and mesh extraction operations on the generated triangular patches to reduce data redundancy while preserving anatomical features such as curvature features at blood vessel bifurcations, resulting in a closed surface manifold mesh.

[0030] Step S113: Tetrahedral volume mesh generation. For subsequent internal stress transfer and deformation calculations, the manifold mesh containing only surface information must be transformed into a volume mesh containing internal nodes. The heterogeneous computing processing device 20 employs a constrained Delaunay triangulation algorithm to discretize the closed space surrounded by the surface manifold mesh into a set of tetrahedral elements.

[0031] Specifically, the physical domain under the reference configuration is defined as follows: The system will Discretize into a set of nodes and tetrahedral unit set The constructed computational grid Among them, the node set Include vertices, represented as: ,in Indicates the first The three-dimensional coordinates of each node in the reference coordinate system.

[0032] Tetrahedral unit set Include Each unit is represented as: .

[0033] For any tetrahedral unit It consists of a set of nodes The system consists of four non-coplanar node indices. A list of element connections is established to define the topological adjacency relationships between nodes, ensuring that the mesh is physically continuous and non-overlapping, thus meeting the geometric continuity requirements for finite element calculations or neural network collocation.

[0034] Step S114: Mesh Quality Optimization and Density Control. To balance computational accuracy and real-time performance, the system implements a mesh density control strategy. The system calculates the local curvature of the surface mesh. For areas with high curvature (such as blood vessel bends or tissue edges), a smaller element size limit is set to generate a high-density mesh; for flat areas (such as large fat layers), a larger element size limit is set to generate a sparse mesh. The system detects and repairs distorted elements, ensuring that the Jacobian determinant of all tetrahedral elements is positive and within a preset quality threshold range to prevent divergence in subsequent physical calculations.

[0035] Finally, the heterogeneous computing processing unit 20 will generate a computing grid. The corresponding node coordinates are stored in the storage unit, serving as the geometric basis for subsequent construction of the reference physical field and assignment of semantic parameters.

[0036] After generating the tetrahedral volume mesh, the heterogeneous computing processing device 20 performs attribute assignment from geometric model to physical model using the in-situ parameter calibration module 202 and the pre-stored histology database in the storage unit. This process aims to define the physical response characteristics of each discrete node in the mesh in the constitutive relation, specifically including the following steps: Step S121: Node-level semantic attribute mapping. The heterogeneous computing processing device 20 reads the spatial coordinates of each node in the three-dimensional volume mesh and maps them back to the original medical image data space in step S111 to obtain the image gray value (such as CT value) and anatomical region index corresponding to the coordinate position. The system defines a semantic mapping function : , which divides the nodes into different tissue categories according to their anatomical positions. For example, set indicates that the node belongs to fat or connective tissue (isotropic matrix), indicates that the node belongs to blood vessel wall or ureter (transversely anisotropic fiber structure), indicates that the node belongs to muscle tissue. For nodes located at the junction of tissues, the system uses a neighborhood weighted voting method to determine the dominant semantic label, ensuring the continuity of the semantic field in space.

[0037] Step S122: Initialization of reference isotropic parameters. For all nodes in the mesh, whether they eventually exhibit anisotropy or not, the system first assigns them basic isotropic elastic parameters. The system queries the pre-stored biomechanical parameter library to obtain the corresponding Young's modulus and Poisson's ratio according to the semantic label of the node . In order to facilitate subsequent tensor operations and physical equation solving in the neural network processor 23, the system converts the engineering constants to Lame constants and . The conversion relationship is defined by the following formula: ; Based on the Lame constants, the system constructs the basic isotropic stiffness tensor . The isotropic stiffness tensor describes the same mechanical response characteristics of the material in all directions.

[0038] Step S123: Geometric extraction of fiber direction vector. For nodes with semantic labels indicating anisotropic structures (such as or ), the system needs to further determine the spatial orientation of the reinforcing fibers. The heterogeneous computing processing device 20 performs skeletonization processing on the mesh regions identified as tubular or beam-like structures to extract the axis paths therein. For any node in this region​ The system searches for its nearest projection point on the skeleton line and calculates the tangent vector at this projection point. This tangent vector is normalized to a unit vector which represents the local fiber direction at this point (e.g. the axial direction of a blood vessel or the muscle fiber direction of a muscle). For isotropic regions, this vector is set to a zero vector.

[0039] Step S124: Synthesis of the transversely isotropic stiffness tensor. The system constructs the final fourth-order stiffness tensor field based on the semantic labels and the extracted fiber directions. For tissues with fiber-reinforced properties, the present application employs a transversely isotropic constitutive model which superimposes a directional reinforcement term on the basis isotropic tensor.

[0040] Specifically, the construction of the fourth-order stiffness tensor follows the following tensor addition rules: ; which is expanded into component form or tensor product form, i.e. ; where is the second-order identity tensor; is the fourth-order symmetric identity tensor which ensures the symmetry of the stress tensor; denotes the tensor product operation; is the local fiber direction unit vector extracted in step S123; is the fiber reinforcement coefficient whose value is provided by a pre- configured database according to the semantic labels, representing the additional stiffness contribution along the fiber direction.

[0041] Through the above steps, the system generates an initial fourth-order stiffness tensor field covering the entire surgical region. This tensor field not only contains the softness and hardness information of the material, but also embeds the anatomical texture direction information of the tissue through the structure of the tensor, providing a direction-sensitive reference model for the subsequent step S300 of physical calibration using the micro-deformation gradient. When the surgical instrument touches the tissue along different directions (such as perpendicular to the axial direction of the blood vessel or parallel to the axial direction of the blood vessel), the tensor field can produce differentiated simulated deformation responses, thereby supporting high-precision parameter inversion.

[0042] The visual perception and semantic analysis module 201 is configured in the graphics processor 22 of the heterogeneous computing processing device 20, and is used for frame-by-frame analysis of the endoscope video stream input by the image acquisition device 10, converting the semantic label matrix with anatomical significance from the RGB pixel matrix without semantics. The process specifically includes the following execution steps: Step S211: Image preprocessing and tensorization. The visual perception and semantic parsing module 201 receives real-time image frames and performs preprocessing such as denoising, histogram equalization, etc. on the images using the graphics processor 22, and normalizes the pixel values. Subsequently, the processed image data is converted into a four-dimensional tensor (batch, channel, height, width) that meets the input requirements of the deep fully convolutional neural network (FCN).

[0043] Steps S212 and S213: Feature extraction and fusion based on encoder-decoder architecture.

[0044] The preprocessed tensor is input into the FCN based on the encoder-decoder architecture. The encoder path gradually reduces the spatial resolution of the feature map while increasing the number of channels through multiple convolutional layers and pooling layers in series, in order to extract high-level abstract semantic features of the image, such as the structure or texture features of the tissue. The decoder path recovers the feature map size layer by layer through upsampling operations. In this process, the system performs a skip connection to splice and fuse the shallow high-resolution feature map at the corresponding level in the encoder path with the deep semantic feature map of the decoder path in the channel dimension. This operation preserves spatial detail information to ensure the accuracy of the segmentation boundary.

[0045] Step S214: Pixel-level classification and probability calculation. The end of the decoder is connected to a 1x1 convolutional layer, whose output channel number is equal to the total number of predefined semantic categories (for example: fat, blood vessels, nerves, surgical instruments, etc.). The graphics processor 22 performs a Softmax normalization exponential operation on the vector of each two-dimensional image coordinate of the output feature map, calculating the posterior probability that the pixel belongs to the th semantic category. The probability calculation formula is as follows: , where is the value (Logits value) of the th semantic category channel in the two-dimensional image coordinate of the output feature map of the last layer of the neural network; is the summation index variable used for denominator normalization calculation; is the predicted class label variable of the pixel at the two-dimensional image coordinate ; is the exponential function operation, which is used to map the Logits value in the real number domain to a positive number.

[0046] Step S215: Semantic mask generation and physical property association. The visual perception and semantic parsing module 201 performs a maximum value index operation on the probability map, assigning each pixel the class index with the maximum probability, thereby generating a two-dimensional semantic mask The values in the mask are directly used as the index of the physical parameter library, providing the spatial distribution information of the tissue properties for the subsequent in-situ parameter calibration module 202 and the neural physical computing module 203. For example, when a certain region in the mask is marked as a category index , the system will automatically call the corresponding constitutive model and benchmark parameters of the index when constructing the physical field equation, thereby realizing the real-time association of visual semantic information and physical computing.

[0047] In order to provide real external observation constraints for the subsequent physical computing module, the visual perception and semantic analysis module 201 performs a feature tracking task based on the optical flow method in the graphics processor 22 in parallel. This process aims to recover the three-dimensional motion trajectory of the tissue surface from the two-dimensional video stream, which specifically includes the following steps: Step S221: Semantic mask guided feature point extraction. The visual perception and semantic analysis module 201 receives the current image frame and the two-dimensional semantic mask generated in step S215 . The system performs binary processing on the mask to establish a region of interest containing only biological soft tissue, while excluding instruments and background interference. Under this regional constraint, an angle point detection algorithm (such as the Shi-Tomasi angle point detection operator) is used to extract pixel points with significant gray scale gradients, ensuring that all tracking points are physically attached to the soft tissue surface.

[0048] Step S222: Sparse optical flow tracking. For the selected feature points, the system calculates their two-dimensional motion vectors using the pyramid Lucas-Kanade optical flow algorithm. Based on the brightness constancy assumption, the optical flow constraint equation is constructed as follows: where represents the two-dimensional coordinate vector of the feature point on the image plane ; represents the gray intensity function of the image at coordinates ; represents the gradient vector of the image gray in the spatial domain ; represents the partial derivative of the image gray in the time domain, i.e., the gray difference between the adjacent two image frames; represents the two-dimensional optical flow velocity vector to be solved, representing the instantaneous motion velocity of the feature point in the image plane. The system obtains the two-dimensional pixel coordinates of the feature point in the current frame by solving the least squares solution of the equation in the local neighborhood window centered on the feature point. In order to eliminate false matches, the system performs forward-backward error checking to remove abnormal points whose tracking trajectories are not closed.

[0049] Step S223: 3D spatial coordinates reconstruction. The heterogeneous computing processing device 20 back-projects the 2D feature points to 3D camera coordinate system using the depth information (derived from binocular disparity or pre-trained depth estimation network) provided by the image acquisition device 10. If the image acquisition device 10 is a binocular endoscope, the system calculates the depth using the disparity map of left and right views; if it is a monocular endoscope, the system obtains the depth map using the pre-trained depth estimation network.

[0050] For any feature point at 2D image coordinate , its 3D spatial coordinate is calculated following pinhole camera model: , where represents the 3D coordinate vector of the feature point in the camera coordinate system at current time ; represents the depth value corresponding to the feature point; represents the inverse matrix of the camera intrinsic matrix , where contains focal length and principal point coordinate parameters; represents the homogeneous pixel coordinate vector of the feature point .

[0051] Step S224: Reference configuration registration and displacement field calculation. To calculate the physical deformation, the system establishes the mapping relationship between the current observed point cloud and the reference configuration mesh generated in step S113. At the initial time of surgery or in a static state, the system registers the initial observed point cloud to the anatomical mesh surface using the Iterative Closest Point (ICP) algorithm, thereby determining the corresponding mesh surface particle for each tracked feature point .

[0052] In each subsequent frame, the system calculates the observed displacement vector of the feature point relative to the reference position according to its real-time 3D coordinate : , where and respectively represent the global rigid rotation matrix and translation vector of the reference coordinate system to the current camera coordinate system, which is used to eliminate the overall field of view changes caused by the endoscope itself motion, thereby separating out the pure tissue deformation displacement.

[0053] Through the above steps, the visual perception and semantic analysis module 201 outputs a sparse surface observed displacement field set , where Total number of valid feature points that are successfully tracked in the current image frame and pass the forward-backward error check. The sparse set of surface-observed displacement fields will be transmitted to the neural network processor 23 as boundary condition data for constraining the training process of the physics-informed neural network.

[0054] To achieve dynamic inversion of tissue physical parameters, the in-situ parameter calibration module 202 precisely captures the moment of effective interaction between the surgical instrument and the tissue based on specific triggering mechanisms. The specific execution steps are as follows: Step S311: Semantic boundary-based contact area preliminary screening. The in-situ parameter calibration module 202 uses the real-time two-dimensional semantic mask generated in step S215 to perform geometric contact determination. The system extracts the pixel region set of the “surgical instrument” and the pixel region set of the “biological soft tissue” . The morphological dilation operation is performed to generate the instrument mask region , and the intersection region of the system is calculated and . If the number of pixels in the intersection region exceeds the preset non-zero threshold , it is determined that there is potential contact, and a candidate event is generated.

[0055] Step S312: Motion causality check. To distinguish between suspension and actual contact, the system introduces kinematic data for causality check. The system obtains the instrument tip velocity vector and the average velocity vector of the tissue in the potential contact area (based on the time differentiation of the sparse set of surface-observed displacement fields in step S224 ). The system calculates the motion correlation index : ; wherein, represents the motion correlation coefficient, usually in the range of [−1, 1]; represents the three-dimensional velocity vector of the surgical instrument tip at the current time; represents the average three-dimensional velocity vector of the tissue surface in the potential contact area; represents the dot product operation of vectors; represents the Euclidean norm (modulus) of vectors; represents a small positive number (smoothing term) to prevent the denominator from being zero. When is greater than the preset positive correlation threshold (for example, 0.8), the system determines that an effective physical interaction driven by the instrument is currently occurring, i.e., it is confirmed that the motion of the tissue is caused by the instrument pushing, rather than the autonomous peristalsis or respiratory motion of the tissue.

[0056] Step S313: Non-destructive operation filtering. To ensure that the parameter calibration is within the elastic deformation range, the system checks the two-dimensional semantic mask . If the pixel categories such as "smoke", "arc" or "bleeding" are detected on the periphery of the contact area, or the discontinuity of the tissue surface topology (abrupt change of the distance between adjacent feature points) is detected, the system will shield the interaction event and not trigger the parameter inversion.

[0057] Step S314: Local region of interest (ROI) locking and triggering. Once the above verification is passed, the system confirms that the current time is an effective interaction time. The system takes the geometric center of the contact area as the center of the circle, and a preset physical influence radius to lock the local region of interest in the three-dimensional volume grid. Its definition is: wherein, represents the node set of the locked local region of interest; represents the spatial coordinates of the th node in the volume grid; represents the total set of all grid nodes. The system then generates a trigger signal, packs the sparse surface observation displacement field set within the local region of interest and the corresponding initial stiffness tensor data, and sends them to the neural physics computing module 203 and the in-situ parameter calibration module 202 to start the subsequent local stiffness correction coefficient calculation process. Through this event-driven mechanism, the system automatically captures effective touch operations and continuously corrects physical parameters during the operation without manual intervention by medical personnel.

[0058] To solve the high computing power consumption problem of real-time physical calculation of the whole field of biological tissues, the in-situ parameter calibration module 202 dynamically constructs an independent calculation sub-domain based on the triggering event of step S314. The specific execution steps are as follows: Step S321: Contact center mapping and sub-domain initialization. The in-situ parameter calibration module 202 receives the geometric center of the contact area. Through the nearest neighbor search, the node closest to in the total set of all grid nodes is determined, which is defined as the topological center node of the local sub-domain.

[0059] Step S322: Volume element screening and sub-grid extraction. The system takes as the center of the sphere, and the physical influence radius to establish a spherical bounding box. From the total set of tetrahedral elements , all elements whose vertices are located within the bounding box are screened out to form a local sub-grid set​ The physical domain covered by this subgrid is defined as: where, represents the set of four vertices constituting a tetrahedral element . represents the spatial coordinates of the vertices, whose values are set to be large enough to ensure that the stress perturbation at the boundary of the subdomain decays to a negligible level.

[0060] Meanwhile, the in-situ parameter calibration module 202 maps the fourth-order stiffness tensor field generated in step S124 to the subdomain, so that each element within the subdomain inherits the anisotropic or isotropic material properties corresponding to it in the full-field model as the initial guess for the subsequent inversion.

[0061] Step S323: Boundary condition classification and definition. To make the extracted physical domain an independent mechanical boundary value problem, the system automatically divides its surface boundary into two categories:

[0062] The first category: free surface boundary corresponds to the real outer surface of the biological tissue. In this area, the system applies the sparse set of surface observation displacement fields obtained in step S224 as Dirichlet boundary conditions or data assimilation constraint points.

[0063] The second category: truncated internal boundary corresponds to the artificial interface that “cuts” the subdomain from the complete organ. Based on the micro-deformation assumption, the system assumes that the displacement is zero on the truncated surface far from the contact center, i.e., a zero displacement constraint is applied: .

[0064] Through this process, the in-situ parameter calibration module 202 simplifies the complex full-field problem into a local boundary value problem and assembles the subgrid topology, initial material properties, and the above two types of boundary conditions into a standard data package, which is transmitted to the neural physics computation module 203 for constructing the loss function of the physical information neural network.

[0065] After constructing the independent computational subdomain of the local region of interest, the neural physics computation module 203 utilizes the high-dimensional tensor operation capability of the neural network processor 23 to perform real-time comparison of the observed physical field and the simulated physical field. This process is achieved by constructing and solving an optimization problem constrained by a physical partial differential equation (PDE), which specifically includes the following steps: Step S331: Construction and forward propagation of the physical information neural network. The neural physics computing module 203 constructs a fully connected deep neural network, which is used as a surrogate model to approximate the displacement field function within the local sub-domain. The input of the network is the arbitrary sampling point , and the output is the predicted simulated displacement vector . In order to capture the tiny deformation gradient, the network adopts an activation function with high-order derivative smoothing characteristics (such as Tanh or Swish).

[0066] For any sampling point within the local physical domain defined in step S322 , the output of the network is represented as: , where represents the simulated displacement vector output by the neural network at the arbitrary sampling point ; represents the nonlinear mapping function of the neural network; represents the set of weight and bias parameters of the neural network, which determines the shape of the displacement field; represents the local stiffness correction coefficient to be calibrated, which is a scalar or tensor field, directly acting on the reference stiffness tensor , used to adjust the softness or hardness of the material to match the observed data.

[0067] Step S332: Assimilation of sparse observation data and data loss calculation. The neural physics computing module 203 introduces the sparse surface observation displacement field set obtained in step S224 into the calculation process. Since the observation data only exists on the discrete feature points on the surface of the object, the system needs to calculate the Euclidean distance error between the network prediction value and the actual observation value.

[0068] The system defines the data assimilation loss function as: , where represents the total number of valid feature points in the current frame, consistent with the definition in step S224; represents the spatial coordinates of the th feature point in the reference configuration, which is input into the neural network to obtain the corresponding predicted displacement; represents the simulated displacement vector predicted by the neural network at the feature point position; represents the real observed displacement vector obtained by the optical flow method and depth reconstruction; represents the square of the L2 norm.

[0069] This step realizes the "anchoring" of sparse visual observation data into continuous physical field simulation, forcing the neural network to generate displacement fields that comply with actual observations at surface keypoints.

[0070] Step S333: Calculation of deformation gradient and constitutive relation mapping. To ensure that the generated displacement field complies with the laws of continuum mechanics, rather than just fitting surface data, the system uses automatic differentiation techniques to calculate the gradient of the network output with respect to the input arbitrary sampling points .

[0071] Linear strain tensor is calculated as: ; Next, the system combines the stiffness correction coefficient to be calibrated and the reference stiffness tensor to calculate the simulated stress tensor according to Hooke's law: where represents the second-order tensor of displacement gradient; represents the transpose operation of the tensor; represents the double-point product contraction operation, which is used to map the fourth-order stiffness tensor and the second-order strain tensor into the second-order stress tensor.

[0072] Step S334: Construction of physical residual gradient. According to the static equilibrium equation (ignoring body force and inertial force, as the micro-deformation process is usually considered quasi-static), the stress divergence inside the object should be zero. The neural physics calculation module 203 again uses automatic differentiation to calculate the spatial derivative of the stress tensor , calculating the physical equation residual .

[0073] The physical residual loss function is defined as: where represents the number of random sampling pairs inside the physical domain ; represents the spatial coordinates of the th node in the volume grid; represents the divergence operator, which is used to calculate the spatial rate of change of the stress tensor.

[0074] Through the above steps, the system constructs a comprehensive evaluation system that includes observation gradients (reflected through ) and simulated physical gradients (reflected through ). The core logic of this system is: if the current stiffness correction coefficient If the above is accurate, the stress field generated by the neural network should also satisfy the mechanical equilibrium equation while meeting the surface observation displacement. If both cannot be met simultaneously, it indicates that there is a deviation in the stiffness parameters or displacement field form, and the total gradient signal generated will guide the subsequent parameter inversion update.

[0075] In the neural physics computation module 203, the system enters the inversion solving stage, aiming to find the optimal stiffness correction coefficient, so that the simulated physical field not only meets the visual observation, but also satisfies the mechanical equilibrium. The specific execution steps are as follows: Step S341: Construct the total loss function and optimization objective. The neural physics computation module 203 performs weighted fusion on the data assimilation loss function calculated in step S332 and the physical equation residual calculated in step S334 to construct the total loss function for parameter inversion : wherein, represents the total objective function with respect to the set of weight and bias parameters of the neural network and the local stiffness correction coefficient to be calibrated ; represents the weight coefficient of the data assimilation term, used to adjust the constraint strength of the observation data on the final result. Specifically, is inversely proportional to the visual observation noise variance, to prevent overfitting noise.

[0076] Step S342: Gradient-based joint optimization solving. The neural physics computation module 203 uses a gradient-based optimization algorithm (such as Adam or L-BFGS) to minimize while inverting the displacement field distribution and material parameters. In each iteration step , the system calculates the gradient of the total loss with respect to the set of weight and bias parameters of the neural network and the local stiffness correction coefficient to be calibrated using the backpropagation algorithm, and updates as follows: ; ; wherein, and represent the learning rate of the network parameters and the physical parameters, respectively, represents the set of all trainable parameters in the physics-informed neural network (PINN) at the th iteration step, including the connection weights (Weights) and bias terms (Biases) between the neurons of each layer; represents the set of neural network parameters at the th iteration step after one gradient descent update. Indicates the first The current estimate of the stiffness correction coefficient to be calibrated at the next iteration step; This indicates that after one gradient descent update, at the i-th... New estimates of the stiffness correction coefficients at the next iteration step; Indicates the state based on the current parameters. The calculated total loss function value includes the data assimilation error term and the physical equation residual term; This represents the set of weights and bias parameters for a neural network. Gradient operator for partial derivatives This indicates the direction of network parameter adjustments that will cause the total loss function to decrease the fastest; This represents the local stiffness correction factor to be calibrated. Gradient operator for partial derivatives It indicates the direction for adjusting material parameters to minimize physical residuals and data errors.

[0077] To ensure the physical meaning (non-negativity) of the stiffness coefficient, the system employs an exponential mapping. For unconstrained auxiliary variables Optimize. When When the system converges or reaches the maximum number of iterations, it outputs the optimal stiffness correction coefficient. .

[0078] Step S343: Semantic association retrieval. Obtain... Subsequently, the in-situ parameter calibration module 202 retrieves the semantic category index of the center of the contact area that triggered this interaction. (Determined by step S215, such as "liver parenchyma").

[0079] Step S344: Full-field parameter broadcasting and update. In-situ parameter calibration module 202 is based on semantic category index. Traversing the entire field of anatomical mesh Utilizing the parallel processing capabilities of the graphics processor 22, all semantic tags were filtered out. The mesh elements are optimized, and the optimal stiffness correction coefficient is obtained. The properties of these units are written in parallel to memory. The update rules are as follows: ,in, This represents the updated real-time stiffness tensor field; This represents a fourth-order stiffness tensor field; Indicates spatial location The semantic category labeling function is used at this location. Through this semantic broadcasting mechanism, the system automatically calibrates the biomechanical model parameters of similar tissues throughout the entire organ using local touch operations, and updates them to the physical parameter database in real time.

[0080] To fully exploit the hardware acceleration feature of the neural network processor 23 for tensor operations, the neural physics computing module 203 does not adopt the traditional finite element mesh discretization method, but directly encodes the partial differential equation (PDE) that controls the deformation of soft tissue into the loss function of the neural network. The process converts the physical law into a differentiable computational graph node, and the specific execution steps are as follows: Step S411: Full connection neural network architecture construction. The neural physics computing module 203 first configures a multi-layer perceptron (MLP) model in the computing memory as a proxy function of the physical field. The input layer of the network receives a three-dimensional spatial coordinate vector , and the output layer generates a three-dimensional displacement prediction vector .

[0081] To ensure that the network has the ability to calculate the second-order physical derivative, the system selects an activation function with high-order derivative smoothing characteristics (such as the hyperbolic tangent function Tanh or the Swish function) to construct the hidden layer, instead of the ReLU function which is not derivable at zero. The forward propagation process of the network is represented as: wherein represents the predicted displacement vector at any sampling point ; represents the depth of the number of layers of the network; and represent the weight matrix and the bias vector of the l-th layer, respectively , and these parameters constitute the solution space to be solved; represents a nonlinear activation function, which is used to fit a complex nonlinear deformation field. Step S412: Spatial derivative calculation based on automatic differentiation. In physics, strain is defined as the symmetric part of the spatial gradient of displacement. The neural physics computing module 203 uses the automatic differentiation (AD) engine of the deep learning framework to construct a gradient computation graph for any sampling point

[0082] . The system first calculates the displacement gradient tensor

[0083] , and then calculates the linear strain tensor : wherein represents the second-order Cauchy strain tensor; represents the predicted displacement vector .The first-order derivative with respect to any sampling point , i.e., the displacement gradient tensor or Jacobian matrix. This matrix describes the rate of change of the displacement field in space; Transposition operator symbol for matrices or tensors.

[0084] Step S413: Constitutive mapping and physical residual construction. The system inputs the strain tensor generated in step S412 into the constitutive model module. Based on the calibrated stiffness tensor in the aforementioned embodiments, the system calculates the stress tensor according to the generalized Hooke's law. Subsequently, the system invokes the automatic differentiation engine again to calculate the divergence of the stress tensor to construct the residual of the static force balance equation.

[0085] Physical equation residual is defined as: where, denotes the divergence operator, which describes the net outflow of the stress field in space; denotes the second-order Cauchy stress tensor; denotes the known volume force vector (such as gravity) acting inside the tissue, which is usually negligible or set as a constant vector in the minimally invasive surgery environment.

[0086] Step S414: NPU computation graph compilation and operator fusion. In order to efficiently execute the above calculations on the neural network processor 23, the neural physics computation module 203 converts the complete logic including forward propagation, automatic differentiation, and residual calculation into a static computation graph. The system uses an in-domain random sampling strategy to generate collocation points, i.e., within the bounding box of the anatomical structure defined by the semantic mask, a set of spatial coordinate points is generated using the Sobol sequence or uniform distribution generator to ensure that the physical constraints cover the entire continuous calculation domain.

[0087] Subsequently, the system applies operator fusion technology to merge the continuous matrix multiplication, addition, and activation function operations into a single hardware instruction kernel using the NPU dedicated graph compiler, reducing the memory read-write overhead.

[0088] Finally, the system defines the physical loss function for NPU training: where, denotes the total number of collocation points sampled in the calculation domain; denotes the spatial coordinates of the th node in the volume mesh, which is generated by the random sampling strategy described above; denotes the L2 norm square of the vector.

[0089] Through the above steps, the system converts the complex continuous medium mechanics solving problem into a standard unsupervised neural network training problem, enabling the NPU to directly solve the physical field using its large-scale parallel floating-point operation unit without assembling a global high-dimensional stiffness matrix.

[0090] The neurophysical computation module 203 constructs a composite loss function that couples observational data constraints with physical law constraints into the same optimization objective to guide network convergence. The specific execution steps are as follows: Step S421: Construction of boundary condition constraints. The system defines the boundary condition loss function. This constraint is used to constrain the network output to conform to the externally observed state. This constraint applies to two types of boundaries: tissue surface displacement boundaries tracked visually. and the anatomically fixed constraint boundaries determined by semantic segmentation. (e.g., at ligament junctions). The loss function is defined as: ,in, This represents the total number of boundary constraint points, which consists of a set of visual feature points and a set of anatomical fixed points. Indicates the first The three-dimensional spatial coordinates of the boundary constraint points; This indicates that the neural network is in operation when the input is... The predicted displacement vector output at that time; Indicates the first The true value of the target displacement at each boundary point (for anatomically fixed points, this value is the zero vector; for surface visual points, this value is the displacement vector calculated by the optical flow algorithm).

[0091] Step S422: Integration of residual terms in the physical governing equations. The system incorporates the residuals from the physical equations. As a regularization term, it ensures that the network maintains mechanical equilibrium throughout the computational domain. This residual term is computed using the NPU's automatic differentiation capability, requiring no external label data. The existence of this constraint means that the learning results of the neural network conform to the principles of continuum mechanics, thus enabling the correct inference of internal stress and deformation distribution based on boundary deformation.

[0092] Step S423: Calculation of adaptive gradient balancing weights. Because... and Due to significant differences in physical dimensions, the system introduces a dynamic weighting mechanism based on gradient statistics to prevent the optimizer from favoring a single factor. The total loss function is defined as: Among them, the weighting coefficient and In each Dynamic updates are performed after 10 training iterations (Epochs). This is a preset hyperparameter, for example, with a value of 100. In this embodiment, it is set... =1 is used as the baseline, and adjustments are made dynamically only. . The update strategy is based on balancing the statistical properties of the backpropagation gradients of each loss term, and its suggested value is... Calculations such as: where, denotes the gradient of the loss function with respect to the last hidden layer weight matrix of the neural network. Subsequently, the actual weights are updated by applying a moving average: where, and denote the gradient tensor of the boundary loss term and the physical residual term with respect to the last hidden layer weight matrix of the neural network, respectively; denotes the maximum value of the absolute values of the elements in the gradient tensor, which measures the maximum driving force of the loss term in the direction of parameter update; denotes the arithmetic mean of the absolute values of all elements in the gradient tensor; denotes a numerical stability constant to prevent the denominator from being zero; denotes a moving average coefficient (e.g., taking the value of 0.9) to smooth the changes of the weights and prevent training oscillation. This adaptive weighting mechanism achieves a dynamic balance between data-driven fitting and physics-driven solving.

[0093] To balance the real-time requirement and the high computational complexity of physical field solving, the neural-physical computing module 203 adopts a heterogeneous pipeline parallel strategy to asymmetrically distribute the computational load to the central processor 22 and the neural network processor 23. This strategy decouples the physical solving task into logic control-intensive and computing-intensive tasks, with the specific execution steps as follows: Step S431: Static division and mapping of heterogeneous computing tasks. The system establishes a heterogeneous task mapping mechanism between the host (Host) and the device. The CPU as the host is responsible for logic control-intensive tasks, including boundary condition data extraction, weight update strategy calculation, and training data batch assembly. The NPU as the device focuses on computing-intensive tasks, i.e., parallel floating-point operations in forward propagation, automatic differentiation, and backpropagation. To eliminate data copy delay, the system opens a page-locked memory pool, and the CPU writes data to this area, which is asynchronously read by the NPU through the direct memory access (DMA) controller, realizing the overlapping execution of computation and data transmission.

[0094] Step S432: Construction of double-buffering asynchronous pipeline. To mask the time-consuming of NPU computation, the neural-physical computing module 203 constructs a double-buffering mechanism. The system defines a computation stream and a data stream. When the NPU's computation stream processes the data of the th iteration, the CPU's data stream prepares the data of the th iteration in parallel and fills it into the standby buffer. After the NPU completes the th iteration, it immediately switches to the standby buffer through a synchronization event, and the original buffer is released to the CPU. This ping-pong operation mode ensures the high-load operation of the NPU operation unit and avoids I / O waiting.

[0095] Step S433: Implementation of mixed precision physical calculation strategy. To take advantage of the hardware optimization of NPU for half-precision floating point (FP16) while ensuring the numerical stability of physical gradient accumulation, the system adopts an automatic mixed precision (AMP) strategy. The system maintains two sets of network parameter copies in the video memory: an FP32 main parameter copy for parameter update , and an FP16 calculation copy for NPU calculation . To prevent numerical underflow of the gradient under FP16, the system introduces a dynamic loss scaling factor . The parameter update logic is as follows: Calculate the scaled gradient under FP16 : ; Convert the gradient back to FP32 and update the main parameter: ; Convert the updated main parameter back to FP16 for the next iteration: , where represents the network weight and bias set stored in high-precision format (FP32) as the reference of the physical field state; represents the network weight and bias set stored in low-precision format (FP16 / BF16) that actually participates in the matrix multiplication operation of NPU; represents the dynamic loss scaling factor. The system monitors whether there is a non-number (NaN) or infinity (Inf). If numerical overflow is detected, then is halved ( ) and the current step update is skipped; if no overflow occurs for consecutive iteration steps (e.g. =2000), then is doubled ( ) to fully utilize the dynamic range of the half-precision format; and represent data type conversion operators, which are used to forcibly convert the input tensor to single-precision floating point format (FP32) and half-precision floating point format (FP16), respectively; represents the scaled gradient tensor.

[0096] Step S434: Operator fusion and kernel generation of the graph. For specific operator sequences frequently occurring in physical computation (e.g., matrix multiplication, addition of bias, activation function), the system utilizes the just-in-time compiler to perform graph-level optimization. Through a pattern matching algorithm, the system identifies and fuses consecutive point-to-point operation subgraphs into a single hardware execution kernel. This technique enables the reuse of intermediate results directly in the NPU's registers or on-chip cache, without the need to write back to the global memory, thereby significantly reducing memory bandwidth occupancy and supporting real-time solution of high-dimensional physical fields on embedded hardware.

[0097] To assist the physician in intuitively perceiving the invisible lesions and key blood vessel positions inside the tissue, the hidden structure mapping module 204 applies the deformation information obtained by the aforementioned neural physical field solution to the preoperative three-dimensional anatomical model in real time, and accurately projects the updated three-dimensional geometric information onto the two-dimensional image plane of the endoscope. The specific execution steps are as follows: Step S511: Physical deformation update of the hidden structure mesh. The system reads the preoperative anatomical model stored in the video memory, which is composed of a series of discrete vertices interconnected, containing the geometric information of internal hidden structures such as tumors, blood vessels, and nerves. The system defines a set of these mesh vertices. The hidden structure mapping module 204 uses the physical field prediction results output by the aforementioned step S433, i.e., the displacement field , to update the position of each static reference vertex , thereby obtaining the real-time vertex coordinates after deformation at the current time .

[0098] The update calculation follows the Lagrangian description method: , where is the unique sequence index of the discrete vertex in the hidden structure mesh model , and is the total number of vertices). This index is used throughout the physical deformation calculation and subsequent projection transformation process to identify the same geometric point.

[0099] Through this step, the system converts the static anatomical model into a dynamic model that interacts with the soft tissue in real time, ensuring that the displayed blood vessel positions remain consistent with the actual physical compression positions.

[0100] Step S512: Perspective projection transformation from three-dimensional space to two-dimensional imaging plane. To superimpose the deformed three-dimensional hidden structure on the two-dimensional endoscopic video stream, the system performs a series of coordinate system transformations. The system obtains the camera extrinsic matrix calculated in real time by the visual tracking module (step S213) and the camera intrinsic matrix previously calibrated.

[0101] The hidden structure mapping module 204 utilizes a homogeneous coordinate transformation to map the morphed vertices in the world coordinate system to the projected points in the image pixel coordinate system , with the transformation formula as: wherein, represents the pixel coordinate of the mesh vertex with index projected on the two-dimensional image plane; represents the scale factor corresponding to the depth value of the point in the camera coordinate system ; represents the intrinsic matrix of the camera, which contains the focal length and the principal point coordinates , describing the optical imaging characteristics of the camera; represents the rotation matrix (3x3) from the world coordinate system to the camera coordinate system, describing the current pose of the endoscope lens; represents the translation vector (3x1) from the reference coordinate system to the current camera coordinate system, describing the current position of the endoscope lens; represents the three-dimensional coordinate column vector of the morphed vertex with index .

[0102] Step S513: Depth-based virtual transparency rendering. In order to provide visual coding of depth position information in the augmented reality image while preserving the texture features of the surface for the doctor to locate, the system adopts a depth-based Alpha blending technique. The hidden structure mapping module 204 calculates the transparency channel value of each projected pixel point in the fragment shader stage of the graphics rendering pipeline.

[0103] The calculation of transparency depends on the depth of the hidden structure relative to the tissue surface. In order to meet the real-time calculation requirement, the system utilizes the depth buffer technique to calculate the relative depth in the line-of-sight direction by differentiating the camera space depth value of the hidden structure fragment and the tissue surface depth value corresponding to the current viewport. The calculation formula of the Alpha value is: wherein, wherein, represents the Alpha channel value (value range 0 to 1) when rendering the hidden structure pixel point; represents the preset maximum opacity (e.g. 0.8), used to prevent the virtual image from completely blocking the real surgical view; represents the relative depth distance of the point on the hidden structure to the current soft tissue surface in the direction of the camera optical axis; represents the depth attenuation feature length, used to control the rate of transparency decay with increasing depth, which is preset according to the type of surgery.​

[0104] Step S514: Multi-layer graphic overlay and video stream synthesis. The system utilizes the graphic processing unit to perform the final image synthesis. The system constructs a multi-pass rendering pipeline: the first pass inputs the real-time RGB video frames captured by the endoscope as the background texture; the second pass inputs the rendered layer of the hidden structure with semantic color coding (e.g., red for arteries, blue for veins, and green for tumor regions) and depth transparency calculated in the above steps. The system processes the superimposition relationship between virtual objects and real backgrounds through a blending function, and finally outputs the synthesized image in a streaming format to the surgical display, providing the surgeon with a virtual-real integrated navigation guide.

[0105] To prevent accidental injury to critical blood vessels or nerves located in the deep tissue during the operation of surgical instruments, the early warning decision module 205 constructs a dynamic safety constraint boundary based on the aforementioned real-time updated three-dimensional morphology of the hidden structure, and calculates the spatial relationship between the surgical instrument and the high-risk anatomical structure in real time. The specific execution steps are as follows: Step S521: Dynamic update of the hidden structure level bounding volume hierarchy (BVH). Given that the hidden structure contains a large number of mesh vertices, direct full-point-to-point distance calculation cannot meet the real-time requirements. The early warning decision module 205 uses the bounding volume hierarchy technique to divide the deformed mesh in space. The system constructs an axis-aligned bounding box (AABB) tree structure based on the reference configuration in the initialization phase. During the operation, the system uses a bottom-up update strategy to maintain the tree structure as the physical solver outputs the vertex displacement of each time step .

[0106] The system re-calculates the minimum coordinate point and the maximum coordinate point of the leaf node bounding box using the deformed real-time vertex coordinates calculated in step S511 . Subsequently, the system backtracks along the tree structure to the root node, updates the parent node bounding box by taking the union of the child node bounding box coordinates, and updates the root node. The computational complexity of this update process is , and the topology of the tree does not need to be changed, which can be completed within the same time step of the physical field solution, ensuring that the spatial structure for collision detection is strictly synchronized with the current physical deformation state.

[0107] Step S522: Calculation of the minimum distance between the surgical instrument and the hidden structure. The system obtains the real-time position of the tip of the surgical instrument in the world coordinate system through the visual tracking module. The early warning decision module 205 traverses the updated AABB tree, uses the bounding box intersection test algorithm to quickly exclude anatomical structure regions far from the instrument tip, and screens out a set of potential collision candidate patches .

[0108] For each triangle patch (composed of vertices ) in the candidate set, the system calculates the geometric shortest distance from the instrument tip to the patch. The calculation process employs the orthogonal projection method: first project onto the plane where the triangle lies to get the projection point , then calculate the barycentric coordinates of the projection point; if the barycentric coordinates show that the point is inside the triangle, then the distance is the perpendicular distance; if it is outside, clamp to the nearest edge or vertex, and calculate the distance to the clamped point.

[0109] The system traverses all the patches in and takes the minimum value: where represents the shortest Euclidean distance from the surgical instrument tip to the surface of the hidden structure; represents the three-dimensional coordinate vector of the surgical instrument tip in the world coordinate system; represents the nearest point on the candidate triangle patch surface (which can be an in-patch point, an edge point, or a vertex); represents the set of surface areas of the hidden structure adjacent to the instrument position after hierarchical bounding volume screening.

[0110] Step S523: Multi-level risk area division and state determination. In order to realize hierarchical warning, the system defines a multi-layer virtual safety boundary, i.e. a dynamic electronic fence, around the surface of the hidden structure. The system presets a warning distance threshold and a danger distance threshold . The warning decision module 205 divides the current operation state into three levels according to the real-time calculation of , and the determination logic is as follows: ;

[0111] Step S524: Augmented reality visualization warning feedback. The system maps the determined safety state into visual graphical properties and superimposes them on the endoscope image. When in state, the rendering color of the hidden structure remains the standard semantic color set in step S514, and the rendering mode is semi-transparent; when entering state, the system renders the outline of the hidden structure in bright yellow, and displays the distance reading in the peripheral area of the screen; when entering ​In the state, the system executes visual shock prompt. The system inputs a uniform variable that changes periodically with the system clock to the fragment shader, uses a sine function to modulate the saturation or transparency channel of the rendering color, so that the hidden structure presents a pulsating flashing effect of red color. At the same time, the system superimposes a graphic mark of "forbidden operation" in the center of the display, and triggers the audio alarm module to emit high-frequency intermittent alarm sound.

[0112] For the implementation of early warning feedback, the system adjusts the rendering parameters in real time by updating the state machine of the graphic rendering pipeline, without reloading the geometric data, ensuring the synchronous response of the early warning signal and the current rendering frame.

Claims

1. An intraoperative multi-parameter analysis decision and assistance warning system based on artificial intelligence, characterized in that, The image acquisition device, the heterogeneous computing processing device and the human-computer interaction terminal device are connected in communication; The image acquisition device is configured to acquire a real-time video stream of a surgical region containing depth information and biological tissue texture; The human-computer interaction terminal device is configured to output an augmented reality image superimposed with a hidden structure projection and a hierarchical audio alarm signal; The heterogeneous computing processing device integrates a central processing unit, a graphics processing unit, a neural network processing unit and a storage unit, and runs a vision perception and semantic analysis module, an in-situ parameter calibration module, a neuro-physical computing module and a pre-warning decision module in cooperation; The vision perception and semantic analysis module is configured in the graphics processing unit and is configured to generate a semantic mask by performing convolutional neural network inference on the real-time video stream, and extract sparse feature point displacement of a tissue surface; The in-situ parameter calibration module is configured to monitor a contact event between a surgical instrument and a tissue, calculate micro-deformation gradient by using the sparse feature point displacement, and generate a dynamic stiffness tensor field based on the semantic mask inversion; The neuro-physical computing module is configured in the neural network processing unit and is configured to receive the dynamic stiffness tensor field as a constraint, solve a physical equation by a physical information neural network, and output a global displacement vector containing a hidden structure position; The pre-warning decision module is configured to update a spatial coordinate of the hidden structure by using the global displacement vector, calculate a dynamic risk distance between the surgical instrument and the hidden structure, and generate a pre-warning control signal accordingly.

2. The intraoperative multi-parameter analysis decision and assistance warning system based on artificial intelligence according to claim 1, characterized in that, The heterogeneous computing processing device is further configured to perform an anatomical modeling and a reference physical field initialization task before surgery: A discretized volume mesh model containing internal nodes is constructed based on a preoperative image of a patient, and an initial semantic label is assigned to the nodes according to their anatomical region indexes in the image; A reference isotropic parameter is assigned to the nodes in the discretized volume mesh model, and a local fiber direction unit vector is extracted for the nodes with a semantic label for anisotropic structure; Based on the reference isotropic parameter and the local fiber direction unit vector, a transversely isotropic reference stiffness tensor is constructed to generate an initial reference stiffness tensor field.

3. The intraoperative multi-parameter analysis decision and assistance warning system based on artificial intelligence according to claim 1, characterized in that, The vision perception and semantic analysis module performs the following operations: A full convolutional neural network with an encoder-decoder architecture is used to perform pixel-level classification on image frames to generate a two-dimensional semantic mask, and the values of the two-dimensional semantic mask are used as indexes of a physical parameter library; Feature points are extracted in a soft tissue region defined by the two-dimensional semantic mask, a two-dimensional motion vector is calculated by using an optical flow method, and the two-dimensional feature points are back-projected to a three-dimensional space in combination with the depth information; Three-dimensional feature points are registered to a reference configuration mesh surface by an iterative closest point algorithm to generate a sparse surface observation displacement field set containing reference positions and current observation displacement vectors.

4. The intraoperative multi-parameter analysis decision and assistance warning system based on artificial intelligence according to claim 1, characterized in that, The in-situ parameter calibration module further performs contact region preliminary screening and motion causality verification: The intersection of the surgical instrument region and the biological soft tissue region in the semantic mask is calculated, and if the number of intersection pixels exceeds a threshold, it is determined as a potential contact; A motion correlation coefficient between a surgical instrument tip velocity vector and an average velocity vector of the tissue surface within a potential contact area is calculated, and if the motion correlation coefficient is greater than a positive correlation threshold and no destructive operation pixel class is detected in the periphery, a local region of interest is locked and a parameter inversion process is triggered.

5. An intraoperative multi-parameter analysis decision and assistance warning system based on artificial intelligence according to claim 4, characterized in that, The in-situ parameter calibration module also performs parameter inversion and full-field update: a calculation sub-domain is constructed in the local region of interest, and the sparse feature point displacement is used as a boundary condition to construct a total loss function including data assimilation loss and physical equation residual; The total loss function is minimized using a gradient optimization algorithm to solve the optimal local stiffness correction coefficient; The semantic class index of the region center where contact is triggered is retrieved, and the optimal local stiffness correction coefficient is broadcast to the grid cells with the same semantic class index in the full field to generate a calibrated dynamic stiffness tensor field.

6. The intraoperative multi-parameter analysis decision and assistance warning system based on artificial intelligence according to claim 1, characterized in that, The fully connected neural network constructed by the neural physics computing module takes spatial coordinates as input and predicted displacement vectors as output, and the training process uses a composite loss function, which includes: A boundary condition loss term is used to constrain the network output to be consistent with the sparse feature point displacement on the tissue surface and to have a displacement of zero at the anatomical fixed boundary; A physical equation residual term is used to calculate the stress tensor divergence by combining the dynamic stiffness tensor field and the strain tensor calculated by automatic differentiation of the network output displacement field, and then to constrain the automatic differentiation network output to satisfy the statics equilibrium equation; An adaptive weighting mechanism based on gradient statistics is used to balance between the boundary condition loss term and the physical equation residual term.

7. The intraoperative multi-parameter analysis decision and assistance warning system based on artificial intelligence according to claim 1, characterized in that, The neural physics computing module adopts a heterogeneous pipeline parallel strategy: The central processing unit is responsible for boundary condition data extraction, weight update strategy calculation, and training data batch assembly; The neural network processor is responsible for performing forward propagation, automatic differentiation, and backward propagation tensor operations using an automatic mixed precision strategy; The system uses operator fusion technology to combine matrix operations in physical calculations and activation function operations into a single hardware execution kernel, and implements asynchronous execution of central processing unit data preparation and neural network processor calculation through a double buffering mechanism.

8. The intraoperative multi-parameter analysis decision and assistance warning system based on artificial intelligence according to claim 1, characterized in that, The early warning decision module includes a hidden structure mapping unit, which is configured to: Update the vertices of the hidden structure grid model using the global displacement vector to obtain real-time vertex coordinates after deformation; Perspective project the real-time vertex coordinates after deformation to a two-dimensional image plane, calculate the transparency channel value based on the depth of the hidden structure relative to the tissue surface, and use virtual transparent rendering to superimpose the hidden structure in the real-time video stream.

9. The intraoperative multi-parameter analysis decision and assistance warning system based on artificial intelligence according to claim 8, characterized in that, The early warning decision module is also configured to perform dynamic risk calculation: Update the hierarchical bounding volume tree structure of the hidden structure from bottom to top based on the real-time vertex coordinates after deformation; Iterate through the hierarchical bounding volume tree structure to filter a candidate patch set, and calculate the shortest Euclidean distance from the surgical instrument tip to the candidate patch set; The shortest Euclidean distance is compared with a preset alert distance threshold and a dangerous distance threshold, a safety state level of current operation is determined, and a rendering color of the stealth structure or a visual shock prompt is adjusted according to the state level.

10. The intraoperative multi-parameter analysis decision and assistance warning system based on artificial intelligence according to claim 1, characterized in that, The neural network processor is hardware-optimized for tensor calculation, is configured to convert a partial differential equation controlling soft tissue deformation into a static calculation graph, and directly solves the global displacement vector satisfying physical constraints by randomly sampling a matching point in a calculation domain.

Images