Surgical anatomy monitoring method and system based on AR glasses

Abstract

The invention provides a surgical anatomy monitoring method and system based on AR glasses, and relates to the technical field of augmented reality, and the method comprises the steps: obtaining medical knowledge data through augmented reality equipment, and generating a digital data set containing a three-dimensional model, a text and a two-dimensional image; acquiring real-time image data from the surgical site, extracting corresponding anatomical structure data, and generating a three-dimensional model data set; presenting the three-dimensional model data set through augmented reality equipment, and superposing the surgical operation guidance data to generate an interactive display result; and receiving a voice control instruction, adjusting the three-dimensional model data set or the surgical operation guidance data, and generating adjusted display content. According to the method, the shielded organ structure in the image data can be analyzed, a complete anatomical structure is generated by combining database inference, and related information is automatically labeled. According to the system, by integrating medical knowledge, the augmented reality technology and image recognition analysis, visual and accurate visualization assistance is provided for an operation, and the operation precision and the teaching effect are improved.

Description

Technical Field

[0001] The present invention relates to the field of augmented reality technology, and in particular to a surgical anatomy monitoring method and system based on AR glasses. Background Art

[0002] A major technical challenge in medical AR glasses systems is how to accurately restore and dynamically display occluded organ structures in complex surgical scenarios. Specifically, when surgeons perform deep-tissue surgery, the target organ is often obscured by other tissues, preventing direct visualization of the complete structure. This complicates surgical procedures and increases medical risks. While the system incorporates a detailed anatomical database, accurately integrating this static data with the real-time surgical footage and dynamically displaying the three-dimensional structure of the occluded area presents a significant challenge. This requires addressing multiple technical challenges, including real-time image recognition, spatial positioning, data matching, and stereoscopic rendering. Furthermore, given the potential for organ position changes during surgery, achieving real-time tracking and updating of occluded structures presents a significant challenge. Furthermore, ensuring low latency throughout the restoration and display process to meet the real-time demands of surgical operations, given limited computing resources, is a pressing issue. Overcoming these technical challenges will significantly enhance the application value of AR glasses in complex surgical scenarios. Summary of the Invention

[0003] The present invention provides a surgical anatomy monitoring method and system based on AR glasses, which mainly include:

[0004] Acquire medical knowledge data through augmented reality devices and generate a digital data set containing three-dimensional models, text and two-dimensional images; acquire real-time image data from the surgical site, extract the corresponding anatomical structure data, and generate a three-dimensional model data set; present the three-dimensional model data set through the augmented reality device, overlay the surgical operation guidance data, and generate an interactive display result; receive voice control instructions, adjust the three-dimensional model data set or surgical operation guidance data, and generate adjusted display content; update the dynamic stereoscopic presentation of the augmented reality device based on the adjusted display content, and generate an updated interactive display result; acquire the first-person perspective image data of the augmented reality device, transmit it to an external display device, and generate teaching analysis image data; detect the target organ from the teaching analysis image data, and if there is occlusion, infer the complete anatomical structure data by combining the medical knowledge data to generate annotated image data.

[0005] Furthermore, the method of obtaining medical knowledge data through an augmented reality device and generating a digital data set including a three-dimensional model, text and two-dimensional images includes: obtaining an original data set from the fields of anatomy, physiology, pathology and pharmacology; removing redundancy and noise through data cleaning to obtain a cleaned data set; using an image processing algorithm to convert the cleaned data set into a three-dimensional model, text and two-dimensional image to obtain a digital data set; if the resolution of the three-dimensional model is lower than a preset threshold, using bilinear interpolation optimization to obtain an optimized digital data set; storing the optimized digital data set in a local storage module of the augmented reality device through a compression algorithm to obtain a stored data set; obtaining the stored data set through a query interface, using an indexing algorithm to implement real-time query, and obtaining query results.

[0006] Furthermore, the method of obtaining real-time image data from the surgical site, extracting corresponding anatomical structure data, and generating a three-dimensional model data set includes: obtaining real-time image data of the surgical site through the camera of the augmented reality device; processing the real-time image data using a feature extraction algorithm to obtain a feature data set; if the number of key points of the feature data set is lower than a preset threshold, optimizing it using an edge detection algorithm to obtain an optimized feature data set; querying anatomical structure data that matches the optimized feature data set from medical knowledge data through an indexing algorithm to obtain a structured data set; and generating three-dimensional models of organs, tissues, and blood vessels using a three-dimensional reconstruction algorithm to obtain a three-dimensional model data set.

[0007] Furthermore, the method of presenting a three-dimensional model data set through an augmented reality device, superimposing surgical operation guidance data, and generating an interactive display result includes: using a semantic segmentation algorithm to process real-time image data to generate structured anatomical data; converting the structured anatomical data into three-dimensional model data through a three-dimensional point cloud algorithm; using layered rendering technology to fuse the three-dimensional model data with the real-time image data to obtain a fused image; obtaining operation steps and instrument trajectory data from a preset surgical database and aligning them with the three-dimensional model data; using hybrid rendering technology to superimpose the operation guidance data on the fused image to generate an interactive display result; and adjusting rendering parameters according to organ displacement to maintain the refresh rate of the dynamic stereoscopic image.

[0008] Furthermore, the receiving of voice control instructions, adjusting the three-dimensional model data set or surgical operation guidance data, and generating adjusted display content include: processing the voice signal through a noise reduction algorithm to generate a denoised voice signal; converting the denoised voice signal into instruction text using a voice recognition model; extracting adjustment keywords in the instruction text through a natural language processing model to generate adjustment labels; obtaining the grid data or instrument trajectory data of the target organ from a database according to the adjustment labels; adjusting the posture of the grid data or generating a guidance mark image using a three-dimensional processing algorithm; aligning the adjustment data with the environmental data through coordinate transformation and point cloud registration to generate adjusted display content.

[0009] Furthermore, the method updates the dynamic stereoscopic presentation of the augmented reality device according to the adjusted display content to generate an updated interactive display result, including: extracting adjustment instructions from the voice signal using voice recognition and natural language processing models; obtaining mesh data or instrument trajectory data of the target organ from a database according to the adjustment instructions; adjusting the mesh data or generating a guide marker image through a three-dimensional transformation algorithm; aligning the adjustment data with the environmental point cloud using a point cloud registration algorithm to generate a transformation matrix; adjusting rendering parameters through a rendering pipeline to generate rendering frame data; and fusing the rendering frame data with the environmental data through the display channel of the augmented reality device to generate an updated interactive display result.

[0010] Furthermore, the method of obtaining the first-perspective picture data of the augmented reality device and transmitting it to the external display device to generate teaching analysis image data includes: obtaining the first-perspective picture data through the camera of the augmented reality device; extracting the dynamic area data using a dynamic area detection algorithm; compressing the first-perspective picture data through a video encoding algorithm to generate compressed picture data; transmitting the compressed picture data to the external display device through a transmission protocol, and obtaining a real-time display picture after decoding; extracting key frames from the real-time display picture, adjusting the resolution and generating adjusted picture data; adding timestamps and trajectory annotations through a motion trajectory algorithm to generate teaching analysis image data.

[0011] Furthermore, the target organ is detected from the teaching and analysis image data. If there is occlusion, the complete anatomical structure data is inferred by combining the medical knowledge data to generate annotated image data, including: using an edge detection algorithm to extract the contour feature data of the target organ from the teaching and analysis image data; if there are discontinuous areas in the contour feature data, the three-dimensional model data of the target organ is obtained from the medical knowledge data; matching point pair data is generated by a feature matching algorithm; the occluded area is completed by an interpolation algorithm to generate completed contour data; complete anatomical structure data is generated by a surface reconstruction algorithm; three-dimensional visualization data is generated by stereoscopic projection technology, and compared with standard anatomical data to generate annotated image data.

[0012] Furthermore, the surgical anatomy monitoring method based on AR glasses includes a data processing module, a data extraction module, an interactive display generation module, a processing module, an update module, an image generation module and an inference module; the data processing module is used to obtain medical knowledge data through an augmented reality device and generate a digital data set including a three-dimensional model, text and a two-dimensional image; the data extraction module is used to obtain real-time image data from the surgical site, extract the corresponding anatomical structure data, and generate a three-dimensional model data set; the interactive display generation module is used to present the three-dimensional model data set through the augmented reality device, overlay the surgical operation guidance data, and generate an interactive display result; the processing module is used to receive voice control instructions, adjust the three-dimensional model data set or the surgical operation guidance data, and generate adjusted display content; the update module is used to update the dynamic stereoscopic presentation of the augmented reality device according to the adjusted display content, and generate an updated interactive display result; the image generation module is used to obtain the first-person perspective image data of the augmented reality device, transmit it to an external display device, and generate teaching analysis image data; the inference module is used to detect the target organ from the teaching analysis image data, and if there is occlusion, infer the complete anatomical structure data in combination with the medical knowledge data to generate annotated image data.

[0013] A computer device comprises: a memory and a processor; the memory stores a computer program, and the processor implements the steps of the surgical anatomy monitoring method based on AR glasses when executing the computer program.

[0014] The technical solution provided by the embodiment of the present invention may have the following beneficial effects:

[0015] The present invention discloses a surgical anatomy monitoring method based on AR glasses. The system converts medical knowledge data into a digital format and stores it locally. During the operation, the surgical site is scanned through image recognition, and the corresponding anatomical structure data is extracted from the database. The data is presented in a dynamic three-dimensional manner on the augmented reality glasses, and surgical operation instructions are superimposed. The system can receive voice commands to adjust the display content and transmit the first-person perspective image to an external device in real time. In addition, the present invention can analyze the occluded organ structure in the image data, combine the database to infer and generate a complete anatomical structure, and automatically annotate relevant information. By integrating medical knowledge, augmented reality technology and image recognition analysis, the system provides intuitive and accurate visual assistance for surgery, improving surgical accuracy and teaching effectiveness. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a flow chart of a surgical anatomy monitoring method and system based on AR glasses of the present invention. DETAILED DESCRIPTION

[0017] The following will describe the technical solutions in the embodiments of the present invention in detail with reference to the accompanying drawings. The described embodiments are only a part of the embodiments of the present invention.

[0018] like Figure 1 In this embodiment, a surgical anatomy monitoring method and system based on AR glasses may specifically include:

[0019] S101. Medical knowledge data acquired from the fields of anatomy, physiology, pathology, and pharmacology are converted into digital formats of three-dimensional models, text, and two-dimensional images, and stored in a local storage module of the augmented reality glasses system to form a medical knowledge database that can be queried in real time.

[0020] Medical knowledge data was acquired, and structured data was extracted from the fields of anatomy, physiology, pathology, and pharmacology. Scrapy was used to crawl the data to obtain the original dataset. Pandas was used to clean the original dataset to remove redundancy and noise, resulting in a cleaned dataset. The anatomical, physiological, pathological, and pharmacological data in the cleaned dataset were converted into 3D models, text formats, and 2D images using OpenCV and PIL to obtain a digitized dataset. If the 3D model resolution in the digitized dataset was lower than a preset threshold, bilinear interpolation was used to optimize it, resulting in an optimized digitized dataset. The optimized digitized dataset was stored in the local storage module of the augmented reality glasses, and the ZIP compression algorithm was used to reduce storage space, resulting in a stored dataset. The stored dataset was retrieved through the query interface of the augmented reality glasses, and a B-tree index was used for real-time querying to obtain the query results. The query results were then displayed in the augmented reality glasses using OpenGL to obtain interactive display results.

[0021] Specifically, the acquisition and processing of medical knowledge data involves the integration of multiple technologies, encompassing data crawling, cleaning, conversion, storage, querying, and visualization. The following analysis and examples address each technical topic, focusing on medical knowledge data processing with rigorous logic and a focus on beneficial outcomes. Scrapy is an efficient web crawler framework for extracting structured data from medical websites.

[0022] For example, crawling anatomical, physiological, pathological, and pharmacological data from authoritative medical databases such as PubMed or professional forums.

[0023] In one possible implementation, Scrapy defines crawler rules and uses PubMed's open API to extract the titles, abstracts, and links of articles containing anatomical atlases, generating a raw dataset in JSON format. This approach ensures reliable data sources, high crawling efficiency, and reduced manual collection costs. Pandas is used for data cleaning to address redundancy and noise in the raw dataset.

[0024] For example, the raw data may contain duplicate anatomical descriptions or inconsistently formatted pharmacological dosage information. Pandas uses deduplication functions and regular expressions to standardize measurement units, such as "mg" and "milligram" to "mg," and removes records with missing key fields. This cleaned dataset is more organized, facilitating subsequent processing and reducing analytical errors. OpenCV and PIL are used to convert the cleaned dataset into 3D models, text, and 2D images.

[0025] Preferably, anatomical data describing bones is converted into a 3D skeleton model using OpenCV, physiological data such as heart rate information is converted into text format, and pathological data such as cancer cell slice images are converted into 2D images using PIL. Digital datasets can be presented in multiple formats to meet the needs of different medical scenarios.

[0026] For example, doctors can visualize bone structure through 3D models, significantly improving diagnostic intuitiveness. Bilinear interpolation is used to optimize 3D model resolution when it falls below a preset threshold.

[0027] It should be noted that if the resolution of the anatomical model is lower than 512x512, bilinear interpolation can be used to smooth the transition between pixels and improve the clarity of the model.

[0028] In one embodiment, the optimized bone model has richer details, smoother edges, and more realistic augmented reality display, which helps doctors make accurate judgments. The optimized digital data set is stored in the local module of the augmented reality glasses using a ZIP compression algorithm.

[0029] As you can understand, ZIP compression reduces the dataset from 500MB to 200MB, saving storage space and accelerating data loading. The stored dataset maintains integrity, ensuring efficient access in offline environments, making it suitable for scenarios without a network connection, such as operating rooms. B-tree indexes are used for real-time queries on the stored dataset.

[0030] Specifically, the AR glasses' query interface uses a B-tree index to quickly locate pharmacological data, such as querying a drug's dosage and side effects. The B-tree's efficient search reduces query times from seconds to milliseconds, improving doctors' decision-making efficiency in emergency situations. OpenGL renders interactive display results within the AR glasses.

[0031] For example, 3D skeletal models are presented with a dynamic perspective, pathology slice images are displayed as a 2D overlay, and pharmacology text is presented as a floating window. The interactive display results are intuitive and clear, and doctors can zoom in and out of the model or switch text using gestures, significantly enhancing the teaching and diagnostic experience. Through the collaboration of multiple technologies, a highly efficient closed-loop system is formed from crawling to display of medical knowledge data, significantly improving data reliability, processing efficiency, and display intuitiveness, contributing to the precision and convenience of medical scenarios.

[0032] S102. In a medical scenario, the image recognition function of augmented reality glasses is used to scan the surgical site and extract the corresponding anatomical structure data from the medical knowledge database, including three-dimensional models of organs, tissues, and blood vessels.

[0033] Real-time image data of the surgical site is acquired through the camera of the augmented reality glasses. The SIFT algorithm in OpenCV is used to extract features from the image, generating a feature dataset of the surgical site. If the number of key points in the feature dataset falls below a preset threshold, the Canny edge detection algorithm is used to extract edges from the image, generating an optimized feature dataset. A B-tree index is used to retrieve anatomical data matching the optimized feature dataset from the medical knowledge database, generating a structured dataset containing organs, tissues, and blood vessels. A 3D reconstruction algorithm in VTK is used to generate 3D models of the organs, tissues, and blood vessels in the structured dataset, generating a 3D model dataset. The 3D model dataset is rendered in real time using OpenGL in the rendering module of the augmented reality glasses, generating a rendered 3D model. If the resolution of the rendered 3D model falls below a preset threshold, bilinear interpolation is used to optimize the model texture, resulting in an optimized 3D model. In the display interface of the augmented reality glasses, the optimized 3D model is fused with the real-time image data using OpenGL's layered rendering technology, resulting in an interactive display.

[0034] Specifically, the augmented reality glasses use the YOLOv5 algorithm to identify the surgical site in real time. When the camera captures the patient's abdominal area, the system detects the liver contour at a frame rate of 30fps, achieving a positioning accuracy of ±1.2mm. During the recognition process, HSV color space conversion is used to separate red blood vessels (H values ​​range from 0-10) in the surgical field from surrounding tissue, and gaps in the vascular contours are filled using morphological closing operations. The system retrieves a matching 3D liver model from a local medical database and uses a KD tree spatial index to complete the matching of 100,000 triangular facets within 200ms. The model data is compressed using the Draco compression algorithm, reducing the original 150MB liver model to 45MB. To address intraoperative organ movement, the ICP iterative closest point algorithm is used for real-time registration, calculating the transformation matrix every 500ms. A registration error exceeding 2mm triggers a model position update. Vascular branching data is reconstructed from CT images using the MarchingCubes algorithm, using a 0.5mm isosurface spacing to generate smooth surfaces. Anisotropic diffusion filtering is used to enhance continuity for microvessels with diameters less than 1mm. 3D rendering is implemented using the Vulkan API, dynamically adjusting the LOD level. A high-polygon model with 20,000 vertices is displayed at a viewing distance of 20 cm, switching to a medium-polygon model with 5,000 vertices at a viewing distance of 50 cm, while maintaining a rendering frame rate of 45 fps. Critical structure warnings triggered intraoperatively are implemented using a FastText classifier. When the distance between the instrument tip and the portal vein is less than 3 mm, a red, translucent warning layer is immediately superimposed on the field of view. The warning area is enhanced using Sobel edge detection.

[0035] S103. Use augmented reality glasses to present anatomical structure data in a dynamic three-dimensional manner, showing the details of organs, tissues and blood vessels, and at the same time superimpose surgical operation guidance data, including operation steps and key precautions.

[0036] A U-Net network is used to perform semantic segmentation on live images, extracting pixel-level masks of organs, tissues, and blood vessels to generate structured anatomical data. The 2D mask data is input into the VTK library, and a 3D point cloud model is constructed using the marching cubes algorithm. The output 3D model contains topological information. The 3D model is rendered layer by layer using a depth buffer technique, and the rendered result is spatially registered with the original image. If the resolution of the fused image falls below a preset threshold, the texture coordinates are resampled using bilinear interpolation. Procedure coordinates and instrument trajectory data are extracted from a pre-defined surgical database and aligned with the 3D model using the ICP algorithm. The guidance data is overlaid onto the fused image using an alpha blending formula, with a blending coefficient of 0.3. If the brightness of the overlaid area falls below 100 cd / m², the blending coefficient is linearly increased to 0.5. Optical flow calculation is performed on five consecutive frames to obtain the organ displacement vector field. VTK rendering parameters are adjusted based on the displacement amplitude: dynamic subdivision is enabled when the displacement is greater than 5 pixels, and the sampling rate is reduced to 50% when it is less than 2 pixels. The final output is a dynamic stereo image with guidance markers, maintained at a refresh rate of 60Hz.

[0037] Specifically, the augmented reality glasses used the deep learning-based Mask R-CNN network to perform instance segmentation of the surgical area, distinguishing the stomach from surrounding adipose tissue with a confidence threshold of 0.8. The segmentation mask resolution was set to 512×512 pixels, and the segmentation accuracy, as assessed by the Dice coefficient, reached 0.92. The system used a pre-trained ResNet50 feature extractor to extract texture features of the gastric wall muscle layer from DICOM images. Frequency domain features were extracted using a Gabor filter bank in six directions with a bandwidth of 1.5 octaves. The 3D reconstruction phase used the Poisson surface reconstruction algorithm with a depth parameter of 9 and five smoothing iterations. A 50,000-vertex stomach model was generated from a 2mm slice-thick CT sequence. Dynamic guidance data was parsed using a BiLSTM temporal model to analyze the surgical procedure text. The "pyloric sphincterotomy" procedure was decomposed into seven keyframes, each annotated with an instrument angle range of 30°±5° and a depth limit of 2mm in the submucosal layer. The spatial registration uses a RANSAC algorithm based on feature points. 20 SIFT feature points are selected on the lesser curvature of the stomach for matching. Re-registration is triggered when the reprojection error exceeds 1.5 pixels. The OpenXR framework is used for real-time rendering. The tessellation shader is enabled for the vascular network within 10 cm from the viewpoint, and the 1 mm diameter blood vessels are subdivided into 16-sided cross-sections. The instrument tracking module integrates IMU and optical hybrid positioning, and uses Kalman filtering to fuse 9-axis sensor data to achieve a positioning accuracy of 0.3 mm at the instrument tip at a 50 Hz update rate. When the power of the electrosurgical knife is detected to exceed 30 W and lasts for 200 ms, the system automatically calls out the heat diffusion simulation layer, uses the finite element method to calculate the tissue heating range, and the isotherm interval is set to 5°C.

[0038] S104. In a surgical scenario, receiving a voice control instruction, and adjusting the anatomical structure data or surgical operation guidance data presented by the augmented reality glasses according to the instruction to generate adjusted display content.

[0039] The speech signal collected by the microphone array is denoised by WebRTC's NS module and then input into Kaldi's GMM-HMM model for speech recognition. The output is a timestamped keyword sequence as the original instruction text. The original instruction text undergoes dependency analysis using spaCy's en_core_web_lg model. Anatomical structure adjustment tags are generated when the verb phrase contains "rotate" or "transparent," and operation guidance adjustment tags are generated when it contains "highlight" or "move." The anatomical structure adjustment tags trigger 3DSlicer's DICOM parsing module, which loads the STL mesh data of the target organ from a preset database. VTK's vtkPolyDataMapper receives the mesh data. If the command contains a view angle parameter, it calls vtkCamera to adjust the pitch angle. If it contains a transparency parameter, it modifies the alpha value of a vtkProperty and outputs an OBJ file with pose information as the first adjustment data. The operation guidance adjustment tags initiate a Neo4j Cypher query to obtain the instrument trajectory coordinates for the surgical step. OpenCV's cv2.circle function draws a guide marker on a blank canvas based on the coordinates. The color parameters are converted to HSV space using cv2.cvtColor, and the position parameters are corrected using the affine transformation cv2.warpAffine, generating the second adjustment data in PNG format. The first or second adjustment data is converted to the world coordinate system using the ROS tf2_ros package. ICP point cloud registration is performed with the depth map acquired by Intel RealSense, outputting a 4×4 homogeneous transformation matrix. This transformation matrix is ​​input into Unity3D's ShaderGraph, where the exposure is adjusted using the ColorAdjustments node in the HDRP pipeline, generating an RGBA32 video frame. The Unity output video frame is then fed into ARKit's ARWorldMap, where feature points are matched against the environment mesh scanned by the LiDAR. Finally, the HoloLens 2's HPU chip renders the video onto the LCoS microdisplay.

[0040] Specifically, in surgical scenarios, augmented reality glasses receive commands through a voice recognition module. Using a Transformer-based speech recognition model, they convert voice signals into text commands with a recognition accuracy of 95%. Based on the command content, the system uses a pre-trained BERT model to perform semantic analysis on the text and extract key operational intent.

[0041] For example, when receiving the "Zoom in on the liver region" command, the system uses a region growing algorithm to extract the liver contour from the CT image, sets the growth threshold to 120 HU, and generates a 3D model containing 30,000 vertices. For the "Show portal vein branches" command, the system uses a graph-cut-based vascular segmentation algorithm to extract the portal vein network from the enhanced CT sequence, sets a smoothing coefficient of 0.8, and generates a vascular tree model containing 2,000 branches. For the "Hide surgical instruments" command, the system uses a YOLOv5-based object detection model to track the instrument's position in real time, sets a confidence threshold of 0.9, and dynamically hides the AR display when the instrument is detected. When receiving the "Adjust view" command, the system uses a quaternion-based view transformation algorithm, sets a rotation angle of 45°, a translation distance of 10 cm, and a smoothing time of 0.5 seconds. For the "Show tumor boundary" command, the system uses a pre-trained U-Net model to segment the tumor region from the MRI image, sets a Dice loss weight of 0.7, and generates a 3D boundary model containing 5,000 vertices. For the "Toggle Tissue Transparency" command, the system uses a physics-based optical transmission model, setting the tissue absorption coefficient to 0.5 and the scattering coefficient to 0.3 to achieve a translucent effect. When the "Show Blood Flow Direction" command is received, the system simulates the motion of 1,000 particles in the blood vessels through a particle tracking-based fluid dynamics simulation with a time step of 0.01 seconds, generating a dynamic blood flow direction indication.

[0042] S105. Update the dynamic stereoscopic presentation of the augmented reality glasses in real time based on the adjusted display content, and synchronously display the updated anatomical structure data and surgical operation guidance data.

[0043] The raw speech signal collected by the microphone array undergoes RNNoise noise reduction processing to generate a denoised speech signal. The denoised speech signal is then fed into the wav2vec2 model, which outputs a timestamped instruction text. The instruction text is then fed into the BERT model, which extracts keywords related to anatomical structure adjustment or surgical guidance adjustment and outputs a classification label. If the classification label is anatomical structure adjustment, the OBJ format mesh data of the target organ is read from the MySQL database. The mesh data is then adjusted using the Open3D rotate and translate functions to generate a rotated mesh data. The rotated mesh data is then aligned with the ambient point cloud acquired by Intel RealSense using the ICP algorithm: the nearest corresponding points between the mesh point cloud and the ambient point cloud are calculated, and the rotation matrix and translation vector are calculated using SVD decomposition, outputting a 4x4 transformation matrix. The transformation matrix is ​​assigned to the shader via a Unity C# script, and the diffuse reflectance and specular reflection intensity are adjusted using the Standard Shader to generate the rendered frame data. If the classification label is "Surgical Guide Adjustment," execute the Cypher query "MATCH(s:Step)-[r:USE]->(t:Tool)RETURNr.trajectory" in Neo4j to retrieve the instrument trajectory data. This trajectory data is fed into OpenCV's arrowedLine and circle functions, with a line width of 3 pixels and a color of RGB (255, 0, 0) to generate a guide marker image. The guide marker image is converted to a texture using Unity's Texture2D, overlaid with the mesh adjustment data in AR Foundation, and ultimately output through the HoloLens 2's display channel.

[0044] Specifically, when updating the dynamic stereoscopic presentation, the augmented reality glasses use a real-time spatial positioning algorithm based on SLAM, set the keyframe matching threshold to 0.85, and control the processing delay of each frame to within 16 milliseconds to ensure that the anatomical structure is accurately aligned with the real space. For the dynamic rendering of vascular structures, the system applies a volume rendering technology based on ray casting, sets the sampling step size to 0.1 mm, and uses the Phong lighting model to calculate the diffuse reflection coefficient of 0.6 and the specular reflection coefficient of 0.3 to generate the lighting effect of the blood vessel wall in real time. For the visualization of neural tissue, the system calls the pre-trained ResNet-18 model for nerve bundle segmentation, sets the feature extraction layer to conv3_x, outputs a 128-dimensional feature vector, reconstructs the neural path through the MarchingCubes algorithm, and generates a mesh model containing 8,000 triangles. When updating surgical instrument guidance, the system uses a trajectory prediction algorithm based on Kalman filtering, sets the process noise covariance to 0.01 and the measurement noise covariance to 0.05, and predicts the instrument movement trajectory within the next three frames. To compensate for organ displacement, the system implements a non-rigid registration algorithm based on Demons, sets the maximum number of iterations to 100 times, and the deformation field update step size to 0.2, to correct the organ position offset caused by respiratory movement in real time. For multimodal data fusion display, the system applies an image fusion method based on wavelet transform, uses the db4 wavelet basis function for three-layer decomposition, sets the low-frequency coefficient weight to 0.7 and the high-frequency coefficient weight to 0.3, to achieve pixel-level fusion of CT and ultrasound images. In terms of dynamic occlusion processing, the system runs an occlusion detection algorithm based on a depth buffer, sets the depth comparison threshold to 0.5 mm, and automatically adjusts the rendering priority of AR content when it detects that the distance between the instrument and the tissue is less than the threshold.

[0045] S106: Capture a first-person perspective image containing updated presentation content from the augmented reality glasses, transmit it to an external display device in real time, and process the first-person perspective image into image data for teaching or analysis.

[0046] First-person perspective image data is acquired through the camera module of the augmented reality glasses. The inter-frame difference method is used to detect dynamic areas in the image and obtain dynamic area data. If the dynamic area data exceeds a preset threshold, the first-person perspective image data is compressed using the H.264 encoding algorithm to generate compressed image data. The compressed image data is transmitted to an external display device via the WebRTC protocol and decoded to obtain a real-time display image. Key frames are extracted from the real-time display image, and the resolution is adjusted using the resize function of OpenCV to generate adjusted image data. For the adjusted image data, the calcOpticalFlowPyrLK function of OpenCV is used to calculate the motion trajectory of the object in the image to obtain motion trajectory data. Based on the motion trajectory data, timestamps and trajectory annotations are added using the putText function of OpenCV to generate annotated image data. The annotated image data is converted to MP4 format and stored in a local database to obtain teaching analysis image data.

[0047] Specifically, the augmented reality glasses use a built-in H.265 encoder to compress the first-person view in real time, setting a bitrate of 8Mbps and a frame rate of 30fps to ensure smooth transmission to external displays. The system uses real-time transmission technology based on the UDP protocol, setting the packet size to 1400 bytes and the retransmission interval to 50 milliseconds to reduce network latency. During the image processing stage, the system uses the OpenCV library for image enhancement and applies the CLAHE algorithm for histogram equalization, setting a contrast limit of 2.0 and a block size of 8×8 to enhance the clarity of image details. For educational purposes, the system integrates an object detection model based on YOLOv5 with a confidence threshold of 0.75. Key anatomical structures in the image are annotated in real time and JSON data containing structure names and locations is generated. For analytical purposes, the system uses a motion analysis algorithm based on optical flow, setting the optical flow window size to 15×15, calculating the displacement vector of the tissue in each frame, and generating a CSV file containing timestamps and displacement amounts. In terms of data storage, the system applies FFmpeg-based video encoding technology to save the processed images in MP4 format, set the resolution to 1920×1080, and the bit rate to 12Mbps. At the same time, the annotation and analysis data are stored in the MySQL database, and the index field is set to timestamp for subsequent retrieval and analysis.

[0048] S107. Analyze the target organ in the image data using the image recognition function. If part of the target organ structure is blocked, infer and generate complete anatomical structure data of the target organ by combining the three-dimensional model data in the medical knowledge database.

[0049] Image recognition technology is used to detect the contour features of the target organ from the image data. The Canny edge detection algorithm is used to extract the contour feature data, obtaining contour feature data. If there are discontinuous areas in the contour feature data, it is determined to be structural occlusion. The three-dimensional model data of the target organ from the medical knowledge database is obtained to obtain three-dimensional model data. The SIFT feature matching algorithm is used to calculate matching point pairs for the contour feature data and the three-dimensional model data, generating matching point pair data. Based on the matching point pair data, the bilinear interpolation algorithm is used to complete the occluded areas in the contour feature data, generating completed contour data. The surface geometric features of the target organ are extracted from the completed contour data, and the Poisson reconstruction algorithm is used to generate the complete anatomical structure data of the target organ, obtaining anatomical structure data. Based on the anatomical structure data, stereomicroscope projection technology is used to generate three-dimensional visualization data of the target organ, obtaining three-dimensional visualization data. The three-dimensional visualization data is compared with the standard anatomical data in the medical knowledge database, and the Euclidean distance calculation method is used to determine the deviation areas, generating deviation annotation data.

[0050] Specifically, when the system detects that part of the target organ structure is occluded, it first uses an instance segmentation algorithm based on Mask R-CNN to accurately segment the visible part, sets the confidence threshold to 0.8, and the IoU threshold to 0.7, to generate a binary mask containing the organ contour. The system then accesses a pre-built medical knowledge database, which stores standardized organ three-dimensional model data. The model uses the OBJ format, has a vertex accuracy of 0.1mm, and contains approximately 500,000 triangular facets. The system aligns the segmented two-dimensional contour with the three-dimensional model in the database, uses the ICP iterative closest point algorithm, sets the maximum number of iterations to 100 times, and the convergence threshold to 0.01mm. The spatial transformation parameters are optimized by the least squares method. After the alignment is completed, the system uses a surface interpolation algorithm based on the radial basis function according to the boundary vertex coordinates of the occluded area, sets the kernel function to the Gaussian kernel, and the bandwidth parameter to 5.0 to perform three-dimensional reconstruction of the missing part. During the reconstruction process, the system calls the medical constraint rule library, which contains more than 2,000 anatomical constraints, such as the vascular branch angle range is set to 45-60 degrees, the standard deviation of organ wall thickness does not exceed 0.3mm, etc., to ensure that the reconstruction results meet anatomical specifications. The final complete organ model is output in PLY format, including vertex normal vector information, and automatically generates the corresponding STL file for 3D printing. The mesh simplification rate is set to 30% to retain key anatomical features. The system also updates the medical knowledge database and stores the newly reconstructed case data in an incremental manner. It uses a MongoDB sharding cluster, sets the sharding key to the organ type, and stores approximately 10TB of data per shard, supporting fast retrieval and comparison.

[0051] S108. Based on the complete anatomical structure data, the image data is photographed or recorded to generate annotated image content including the anatomical structure and operation instructions, the name and function information of the target part are automatically annotated, and the content is transmitted to an external display device.

[0052] From the complete anatomical data in DICOM format, the ITK-SNAP tool was used to segment the target organ voxel data. The MarchingCubes algorithm was used to extract isosurfaces from the segmented voxel data, generating surface feature point cloud data. When the surface feature data contained holes, the Elastix tool was used to non-rigidly register the pre-set STL-format organ template with the point cloud to complete the missing regions. The completed feature data was imported into Volume Graphics VG Studio MAX software, and 3D organ visualization data was generated using the volume rendering pipeline. From the 3D visualization data, the VTK library was used to extract the spatial coordinate matrix of the target part. This coordinate matrix was spatially aligned with anatomical landmarks in the OpenGauss medical knowledge database to generate JSON-formatted annotation data containing part names and functional information. The VTK annotation renderer was used to overlay the JSON annotation data onto the 3D visualization data, generating annotated BMP-formatted image data. The annotated image data was transmitted to a BARCOM DCC-6230 medical display using the DICOM network transmission protocol. When the EDID information of the display shows a resolution lower than 3840×2160, the EDSR algorithm is called to perform 4x super-resolution reconstruction on the image data, and finally generate DICOM format data that meets the display requirements.

[0053] Specifically, the system uses the OpenCV library to capture or record images based on complete anatomical data, with a resolution of 1920×1080 and a frame rate of 30 fps to ensure image clarity sufficient for medical analysis. During image generation, the system uses a pretrained YOLOv5 model with a confidence threshold of 0.85 to detect and locate target areas in real time. An attention-based annotation algorithm automatically annotates the images with their names and functional information, using a 12-point font size and red color to ensure clarity. The system encodes the annotated images using FFmpeg, using the H.264 compression algorithm at a bitrate of 5 Mbps, generating MP4 video files. The annotated images are then transmitted in real time to an external display device via the RTSP protocol, with a transmission delay of no more than 200 ms to ensure smooth and real-time viewing. During transmission, the system uses a TCP / IP-based streaming protocol with a 1 MB buffer size to prevent data loss. Simultaneously, the system dynamically updates the annotations using a medical knowledge database to ensure accuracy and timeliness. Finally, the system stores the generated annotated images in a distributed file system, adopting the HDFS architecture and setting the number of replicas to 3 to ensure data security and accessibility.

[0054] The above embodiments are intended to illustrate the technical solutions of the present invention and are not intended to limit the present invention. The present invention is described in detail with reference to the preferred embodiments only. It should be understood by those skilled in the art that the technical solutions of the present invention may be modified or replaced with equivalents without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or equivalents should be included within the scope of the claims of the present invention.

Claims

1. A surgical anatomy monitoring method and system based on AR glasses, characterized in that: include: Acquire medical knowledge data through augmented reality devices to generate digital data sets containing three-dimensional models, text, and two-dimensional images; Acquire real-time image data from the surgical site, extract corresponding anatomical structure data, and generate a three-dimensional model dataset; Use augmented reality devices to present 3D model data sets, overlay surgical operation guidance data, and generate interactive display results; Receive voice control instructions, adjust the three-dimensional model data set or surgical operation guidance data, and generate adjusted display content; updating the dynamic stereoscopic presentation of the augmented reality device based on the adjusted display content to generate an updated interactive display result; Acquire the first-person perspective image data of the augmented reality device, transmit it to an external display device, and generate teaching analysis image data; Detect target organs from teaching analysis image data. If there is occlusion, combine medical knowledge data to infer complete anatomical structure data and generate labeled image data.

2. The surgical anatomy monitoring method based on AR glasses according to claim 1, characterized in that: The method of acquiring medical knowledge data through an augmented reality device and generating a digital data set containing a three-dimensional model, text, and two-dimensional images includes: Acquire original datasets from the fields of anatomy, physiology, pathology, and pharmacology; Remove redundancy and noise through data cleaning to obtain a clean data set; Image processing algorithms are used to convert the cleaned dataset into three-dimensional models, texts, and two-dimensional images to obtain a digitized dataset; If the resolution of the three-dimensional model is lower than the preset threshold, bilinear interpolation optimization is used to obtain an optimized digital data set; storing the optimized digital data set in a local storage module of the augmented reality device through a compression algorithm to obtain a stored data set; The stored data set is obtained through the query interface, and the index algorithm is used to implement real-time query to obtain the query results.

3. The surgical anatomy monitoring method based on AR glasses according to claim 1, characterized in that: The method of acquiring real-time image data from the surgical site, extracting corresponding anatomical structure data, and generating a three-dimensional model data set includes: Obtain real-time image data of the surgical site through the camera of the augmented reality device; Use feature extraction algorithm to process real-time image data to obtain feature data set; If the number of key points in the feature data set is lower than the preset threshold, the edge detection algorithm is used for optimization to obtain the optimized feature data set; The anatomical structure data matching the optimized feature dataset is searched from the medical knowledge data through an indexing algorithm to obtain a structured dataset. A three-dimensional reconstruction algorithm is used to generate three-dimensional models of organs, tissues and blood vessels to obtain a three-dimensional model data set.

4. The surgical anatomy monitoring method based on AR glasses according to claim 1, characterized in that: The method of presenting a three-dimensional model data set through an augmented reality device, superimposing surgical operation guidance data, and generating an interactive display result includes: Use semantic segmentation algorithms to process real-time image data and generate structured anatomical data; Convert structured anatomical data into 3D model data through 3D point cloud algorithm; Adopt layered rendering technology to fuse 3D model data with real-time image data to obtain a fused image; Obtain operation steps and instrument trajectory data from the preset surgery database and align them with the 3D model data; Hybrid rendering technology is used to overlay operation guidance data onto the fused image to generate interactive display results; Rendering parameters are adjusted according to organ displacement to maintain the refresh rate of dynamic stereo images.

5. The surgical anatomy monitoring method based on AR glasses according to claim 1, characterized in that: The receiving of the voice control instruction, adjusting the three-dimensional model data set or the surgical operation guide data, and generating the adjusted display content includes: Process the speech signal through a noise reduction algorithm to generate a denoised speech signal; Use a speech recognition model to convert the denoised speech signal into command text; Extract adjustment keywords from instruction texts through natural language processing models and generate adjustment labels; Obtaining grid data or instrument trajectory data of the target organ from a database according to the adjustment label; Use 3D processing algorithms to adjust the position of grid data or generate guidance mark images; The adjusted data is aligned with the environment data through coordinate transformation and point cloud registration to generate the adjusted display content.

6. The surgical anatomy monitoring method based on AR glasses according to claim 1, characterized in that: The updating of the dynamic stereoscopic presentation of the augmented reality device according to the adjusted display content to generate an updated interactive display result includes: Use speech recognition and natural language processing models to extract adjustment instructions from speech signals; Obtaining grid data of the target organ or instrument trajectory data from a database according to the adjustment instruction; Adjusting grid data or generating guide mark images through three-dimensional transformation algorithms; Use point cloud registration algorithm to align the adjustment data with the environment point cloud and generate a transformation matrix; Adjust rendering parameters through the rendering pipeline to generate rendering frame data; The rendered frame data is fused with the environment data through the display channel of the augmented reality device to generate an updated interactive display result.

7. The surgical anatomy monitoring method based on AR glasses according to claim 1, characterized in that: The method of obtaining the first-view image data of the augmented reality device, transmitting the data to the external display device, and generating teaching analysis image data includes: Acquire first-person perspective image data through the camera of the augmented reality device; Dynamic area detection algorithm is used to extract dynamic area data; Compressing the first-view picture data by a video encoding algorithm to generate compressed picture data; The compressed image data is transmitted to the external display device through the transmission protocol, and the real-time display image is obtained after decoding; Extract key frames from the real-time display screen, adjust the resolution and generate adjusted screen data; add timestamps and trajectory annotations through the motion trajectory algorithm to generate teaching analysis image data.

8. The surgical anatomy monitoring method based on AR glasses according to claim 1, characterized in that: The target organ is detected from the teaching analysis image data. If there is occlusion, the complete anatomical structure data is inferred by combining the medical knowledge data to generate annotated image data, including: The edge detection algorithm is used to extract the contour feature data of the target organ from the teaching analysis image data; if there are discontinuous areas in the contour feature data, the three-dimensional model data of the target organ is obtained from the medical knowledge data; Generate matching point pair data through feature matching algorithm; Use interpolation algorithm to complete the occluded area and generate completed contour data; Generate complete anatomical structure data through surface reconstruction algorithm; Three-dimensional visualization data is generated through stereo projection technology and compared with standard anatomical data to generate annotated image data.

9. A surgical anatomy monitoring system based on AR glasses, based on the surgical anatomy monitoring method based on AR glasses according to any one of claims 1 to 8, characterized in that: including, a data processing module, a data extraction module, an interactive display generation module, a processing module, an updating module, an image generation module and an inference module; a data processing module, configured to acquire medical knowledge data through an augmented reality device and generate a digital data set comprising a three-dimensional model, text, and two-dimensional images; A data extraction module is used to obtain real-time image data from the surgical site, extract the corresponding anatomical structure data, and generate a three-dimensional model data set; An interactive display generation module is used to present a three-dimensional model dataset through an augmented reality device, overlay surgical operation guidance data, and generate an interactive display result; a processing module, configured to receive voice control instructions, adjust the three-dimensional model data set or surgical operation guide data, and generate adjusted display content; An updating module, configured to update the dynamic stereoscopic presentation of the augmented reality device according to the adjusted display content, and generate an updated interactive display result; An image generation module is used to obtain the first-person perspective image data of the augmented reality device, transmit it to an external display device, and generate teaching analysis image data; The inference module is used to detect target organs from teaching and analysis image data. If there is occlusion, the complete anatomical structure data is inferred by combining medical knowledge data to generate labeled image data.

10. A computer device comprising: memory and processor; The memory stores a computer program, characterized in that when the processor executes the computer program, the steps of the surgical anatomy monitoring method based on AR glasses described in any one of claims 1 to 8 are implemented.

Images