VR-assisted tumor pathology three-dimensional diagnosis system and pathology section 3D reconstruction method

Abstract

The application discloses a VR-assisted tumor pathology three-dimensional diagnosis system and a pathological section 3D reconstruction method, and the system comprises a data layer, an algorithm layer, a VR interaction layer and a cooperation layer. The algorithm layer realizes 3D reconstruction of pathological sections through two-stage alignment, and completes sub-pixel level fine alignment through a mixed model of a Transformer-UNet; the tumor area is segmented and the core area is located based on a 3D U-Net++ network; and an improved A* algorithm is used to generate an intelligent observation path combined with lesion probability. The VR interaction layer highlights and marks key areas in the three-dimensional model according to the analysis result and realizes multi-user cooperative operation. The cooperation layer supports remote VR consultation through encrypted transmission and real-time synchronization. The application realizes the whole-process integration of pathological section reconstruction, intelligent analysis and immersive diagnosis, and effectively improves the accuracy and cooperation efficiency of tumor diagnosis.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of medical image processing, virtual reality interaction and telemedicine technology, and specifically relates to a VR-assisted three-dimensional diagnosis system for tumor pathology and a method for 3D reconstruction of pathological slides based on the system. Background Technology

[0002] Pathological diagnosis is the gold standard for confirming tumor diagnosis, and its accuracy directly determines the formulation of tumor treatment plans. Traditional tumor pathological diagnosis relies on pathologists manually interpreting two-dimensional paraffin sections and piecing together the three-dimensional structure of the tumor based on their personal experience, which has many technical limitations: The limitations of three-dimensional structural cognition lead to a high rate of misdiagnosis and missed diagnosis: Two-dimensional slides can only present local tissue morphology, making it difficult for doctors to accurately grasp the location of the tumor core area, invasion path, and edge infiltration range. Traditional two-dimensional slide reading has a high rate of misdiagnosis for early-stage and complex tumors. Low efficiency and poor accuracy of 3D reconstruction of pathological slides: Existing 3D reconstruction technologies mostly use pure convolutional neural networks to achieve slide alignment, which is insufficient in global feature extraction capabilities. As a result, the reconstruction of 500 consecutive slides takes more than 10 hours, and the alignment accuracy can only reach the 2-pixel level, which cannot meet the needs of rapid clinical diagnosis.

[0003] Incomplete information transmission during remote consultations: Current remote pathology consultations mainly rely on two-dimensional image transmission, lacking three-dimensional spatial information of tumors. Multiple experts cannot simultaneously observe the three-dimensional structure of lesions, resulting in longer diagnosis cycles for difficult cases and delaying the best treatment opportunity.

[0004] Reading images relies on doctors' experience, resulting in a steep learning curve: novice doctors need to accumulate years of experience to independently complete complex tumor diagnoses, and their diagnostic accuracy is lower than that of senior doctors. The lack of pathological diagnostic resources in primary hospitals is a prominent issue.

[0005] While existing technologies have attempted to apply VR technology to medical visualization and the Transformer architecture to image segmentation—for example, patent CN116246019A discloses a method for 3D reconstruction of pathological slides—they do not involve VR intelligent navigation. Current technologies have not yet achieved a complete integration of high-precision alignment of pathological images, 3D reconstruction, VR intelligent navigation, and medical-grade remote collaboration. Furthermore, existing telemedicine technologies have not designed adaptation schemes for the large capacity and high real-time requirements of pathological 3D volumetric data, resulting in problems such as slow data transmission, poor security, and high latency in multi-user synchronization.

[0006] Therefore, there is an urgent need for a tumor pathology diagnostic system that can achieve efficient and accurate 3D reconstruction, intelligent guided image reading, and safe and low-latency remote collaboration, in order to overcome the above-mentioned shortcomings of existing technologies. Summary of the Invention

[0007] To address the aforementioned issues, this invention provides a VR-assisted three-dimensional tumor pathology diagnostic system, comprising a data layer, an algorithm layer, a VR interaction layer, and a collaboration layer that are sequentially connected in communication. The data layer uses a distributed file system to store pathological slide images, preprocessed data, and 3D volume data, and supports DICOM and TIFF formats. The algorithm layer, deployed on a GPU server, is used to realize 3D reconstruction of pathological slides, tumor region segmentation, and observation path planning. It includes: a 3D reconstruction submodule: employing a two-stage strategy of coarse and fine alignment to achieve spatial alignment of continuous pathological slides; the fine alignment is based on a Transformer-UNet hybrid model, which contains a 6-layer Transformer encoder and an 8-layer UNet decoder; a tumor region segmentation submodule: achieving accurate tumor region segmentation based on a 3D U-Net+ network, locating the tumor core area using density peak clustering, and extracting the tumor invasion path through 3D median transformation; and an observation path planning submodule: generating personalized observation paths based on an improved A* algorithm, where the heuristic function of the improved A* algorithm includes a weighted sum of distance cost and lesion probability cost. The VR interaction layer is used for real-time rendering of 3D volume data and human-computer interaction. Based on the tumor core area location, invasion path and personalized observation path output by the algorithm layer, corresponding visualization marking and navigation guidance are performed in the rendering scene. The collaboration layer is used for remote task scheduling, secure data transmission, and multi-user collaborative consultation. It uses the AES-256 encryption algorithm to encrypt data throughout the process and processes multi-terminal remote reconstruction requests through a priority queue scheduling algorithm.

[0008] Preferably, the execution process of the 3D reconstruction submodule in the algorithm layer includes: First, coarse alignment is performed. Feature points of continuous pathological slices are extracted based on the improved SIFT algorithm, and feature point matching is completed through the FLANN matcher. Then, the RANSAC algorithm is combined to remove mismatched points, achieving preliminary spatial alignment between slices. The RANSAC (Random Sample Consensus) algorithm is an iterative method used to estimate the parameters of a mathematical model from a set of observational data containing outliers. Unlike least squares, RANSAC effectively handles data with a large number of outliers by iteratively selecting a random subset for fitting. The basic idea of ​​the RANSAC algorithm is to randomly select a minimum number of data points to estimate model parameters, and then determine how many data points fit the model. This process is repeated multiple times, and each time a different model may be generated. Finally, the best model is selected, that is, the model supported by the most suitable data points (inliers). The RANSAC algorithm has wide applications in image processing, autonomous driving, machine learning, and other fields. For example, in computer vision, RANSAC can be used to robustly estimate geometric transformations or eliminate mismatches from image features.

[0009] Then, fine alignment is performed based on the Transformer-UNet hybrid model. The Transformer-UNet hybrid model first extracts the global features of the slice image through a 6-layer Transformer encoder, and then fuses the global features and local detail features through an 8-layer UNet decoder to output a pixel-level displacement field. Based on this displacement field, the sub-pixel-level fine alignment of the slice is completed. The Transformer-UNet hybrid model has a Transformer encoder with 12 self-attention mechanisms per layer, a hidden layer dimension of 768, and uses the GELU activation function. The GELU (Gaussian Error Linear Unit) activation function is a non-linear activation function proposed by Dan Hendrycks and Kevin Gimpel in 2016. It is widely used in Transformer architectures (such as BERT and GPT series) and has become an important component in deep learning due to its smoothness and performance advantages. Because of its smoothness and performance enhancements, GELU is widely adopted in Transformer, computer vision, generative models, and other fields, and is one of the preferred activation functions for modern deep learning models.

[0010] The UNet decoder uses a 3×3 convolutional kernel with padding of 1 and integrates a batch normalization module; the loss function of the Transformer-UNet hybrid model is the chamfer distance loss. With cross-entropy loss The weighted sum, with weights of 0.6 and 0.4 respectively, and the chamfer distance loss. The weights are used to focus on the spatial deviation between the predicted and actual displacement fields; cross-entropy loss The weights are used to constrain the accuracy of pixel-level classification / regression of the displacement field, and the specific formula is as follows: ; ; ; Where N is the total number of pixels in a single pathological slide; i is the pixel index, ranging from 1 to N, representing the i-th pixel in the slide; This represents the predicted displacement value of the i-th pixel in the x-direction; Let be the predicted displacement value of the i-th pixel in the y-direction; This represents the actual displacement value of the i-th pixel in the x-direction; is the true displacement value of the i-th pixel in the y-direction; C represents the number of categories; c is the category index, with a value ranging from 1 to C; This represents the true label of the i-th pixel in category c; This represents the predicted probability of the i-th pixel in category c; Preferably, the execution process of the tumor region segmentation submodule in the algorithm layer includes: First, tumor region segmentation is performed based on the 3D U-Net++ network. A 3D voxel block of a preset size is input, and multi-scale tissue features are extracted by the network encoder and high and low layer features are fused by the decoder. The result is a binarized segmentation of the tumor region and normal tissue. Then, the density peak clustering algorithm is used to locate the tumor core area, calculate the cell density of each voxel in the segmented tumor area, select the local density maxima as the cluster center, and determine the area that meets the preset density threshold and distance threshold as the tumor core area. Finally, the central skeleton of the tumor region is extracted by 3D central axis transformation, and the invasion and expansion vector is determined by combining the gradient ascent direction of the tumor edge voxels. Based on the skeleton orientation and expansion vector, the main invasion path and branch invasion paths of the tumor are generated. The density peak clustering algorithm sets the local density threshold ρ to 1.2 × 10⁻⁶. 3 The criteria for determining the tumor core area are: a cell density not less than 3 times that of normal tissue and a nuclear morphological irregularity not less than 0.7, with a local density threshold ρ = 1.2 × 10⁻⁶ m³ / m³ and a distance threshold δ of 3 mm. 3 pcs / m 3The reason for setting this threshold is as follows: This threshold is precisely set based on the difference in cell density between normal pathological tissue and tumor tissue. Referring to clinical pathological statistics, the cell density of normal human pathological tissue (such as lung, breast, and other common lesion tissues) is approximately 3.0 × 10⁻⁶. 2 ~4.0×10 2 pcs / m 3 Tumor tissue, due to abnormal cell proliferation, generally has a density 3 to 5 times that of normal tissue; simultaneously, considering the clustering characteristics of the density peak clustering algorithm, ρ is set to 1.2 × 10⁻⁶. 3 pcs / m 3 It can accurately distinguish between tumor areas (cell density above the threshold) and normal tissue areas (cell density below the threshold), and avoid missing tumor edge areas due to excessively high thresholds and introducing redundant clustering of normal tissues due to excessively low thresholds, thus providing accurate density screening basis for locating the tumor core area.

[0011] The reason for setting the distance threshold δ to 3mm is as follows: The core of setting δ is to define the spatial range of the tumor core area. Combining the voxel size after 3D reconstruction of pathological sections (adapting to the volume data characteristics of 512×512 sections and Z-axis stacked reconstruction) and common sense of clinical tumor diagnosis, the tumor core area is mostly a spherical or near-spherical region with dense cells and active proliferation, with a diameter of 2~4mm. Setting δ to 3mm can ensure that during the clustering process, adjacent cells with a density higher than ρ can be classified into the same cluster. It will not cause the tumor core area to be split into multiple scattered clusters due to δ being too small, nor will it cause multiple independent tumor core areas to be merged due to δ being too large. At the same time, it adapts to the spatial accuracy requirements of subsequent 3D central axis transformation and invasion path extraction.

[0012] The criteria for determining the tumor core area are established for the following reasons: This criterion is based on density clustering and further screened by core indicators of tumor pathological diagnosis to ensure that the located core area is the true core of the tumor lesion, rather than normal tissue or the edge area of ​​the tumor. 1. Cell density not less than 3 times that of normal tissue: Referring to the clinical pathological diagnostic criteria, due to abnormal proliferation, the cell density in the core area of ​​tumor cells is usually 3 times or more than that of the corresponding normal tissue. This is a key indicator for distinguishing the tumor core area from the tumor edge area and normal tissue. This standard echoes the setting of ρ mentioned above, further verifying the tumor attributes of the cluster and avoiding misjudging normal tissue with high density (such as inflammatory hyperplasia areas) as the tumor core area.

[0013] 2. Nuclear morphology irregularity not less than 0.7: Nuclear morphology irregularity is a key parameter for measuring tumor cell atypia (a core indicator for determining the malignancy of tumors in pathological diagnosis). Normal tissue cells have regular nuclear morphology, and the irregularity is usually less than 0.3. However, due to gene mutations, cells in the tumor core area have distorted and irregular nuclear morphology, and the irregularity is generally higher than 0.7. Setting this threshold can accurately screen out cell clusters with typical tumor characteristics, exclude benign proliferative areas with normal nuclear morphology but adequate density, ensure the accuracy of tumor core area localization, and provide reliable core targets for subsequent tumor invasion path extraction.

[0014] In summary, the above parameters and judgment criteria take into account the clinical pathological reality, the adaptability of the density peak clustering algorithm, and the accuracy requirements of subsequent tumor region segmentation and invasion path extraction. This forms a dual guarantee of density screening and pathological feature verification, ensuring accurate and reliable localization of the tumor core area and supporting the high-quality development of subsequent VR interaction and remote collaborative consultation.

[0015] Preferably, the execution process of the observation path planning submodule in the algorithm layer includes: First, obtain the lesion area information such as the tumor core area, invasion path, and marginal infiltration area output by the tumor region segmentation submodule, and determine the starting point (normal tissue transition area) and ending point (tumor core area) of the path, as well as key nodes along the path (key points of the invasion path). Then construct the heuristic function for the improved A* algorithm. in, , ; As a cost of distance; The lesion probability cost is the distance cost, which is the Euclidean distance from the current node to the target node. The lesion probability cost is the tumor probability of the region where the current node is located (output by the tumor segmentation model). The initial path is planned based on the heuristic function, and the constrained path covers 100% of the tumor core area and more than 90% of the invasion path; Finally, an adaptive path adjustment mechanism is provided, which supports automatic adaptation of path emphasis according to tumor type, or manual dragging of path nodes through VR interactive devices to customize the observation path. The adjusted path is synchronized to the VR interactive layer in real time.

[0016] Preferably, the VR interaction layer is based on the Unity 3D rendering engine and supports three interaction methods: gesture, voice, and controller buttons; the VR interaction layer integrates a voice navigation module, which triggers voice prompts in key lesion areas through a voice synthesis engine.

[0017] Preferably, the collaboration layer constructs a real-time transmission channel based on the WebRTC protocol, adopts a combined UDP and SRTP transmission mode, and integrates a 30ms adjustable Jitter Buffer dynamic buffering mechanism. The 3D volume data uses VP9 video encoding, supporting simultaneous online VR scene synchronization for up to 5 people; It provides 3D drawing tools, precise measurement and text / voice annotation tools, and the annotation information is synchronized to all online user terminals in real time, retaining the annotation history of the most recent 100 times.

[0018] Preferably, a 3D reconstruction method for pathological slides, applied to the VR-assisted three-dimensional diagnosis system for tumor pathology, includes the following steps: Step 1, Preprocessing: The digital image of the pathological slide is subjected to staining normalization, wrinkle repair and cropping scaling in sequence. The staining normalization uses an improved Macenko algorithm to convert the RGB image to optical density space. The improved Macenko algorithm includes: a. Adaptive background purification technology before optical density space conversion, which accurately removes blank areas of the slide and scans pixels of impurities and necrotic tissue through Otsu threshold segmentation; b. Introduce a multi-reference image fusion strategy, and combine HE staining pathological prior knowledge to constrain the range of hematoxylin and eosin core staining vectors. This breaks through the limitation of poor generalization of the original algorithm with a single reference template, and adapts to the staining differences of pathological slides from different hospitals and different batches. c. Adaptive staining normalization technology dynamically adjusts the normalization coefficient and incorporates a mild Gaussian noise suppression mechanism. While achieving a unified global staining style, it fully preserves the details of tissue texture and avoids the impact of local staining deviations and scanning noise on the accuracy of subsequent coarse and fine alignment of sections. Among them: The Macenko algorithm is a commonly used staining normalization method in pathological image processing. Its core idea is to extract the main staining vectors through optical density (OD) space transformation and singular value decomposition (SVD), thereby mapping the colors of slides from different sources to a unified reference standard. This method can effectively reduce color deviation caused by differences in scanning equipment and experimental conditions in digital pathology analysis. Step 2, two-stage alignment: Based on the improved SIFT algorithm, feature points of the slides are extracted and coarse alignment is achieved. Then, sub-pixel-level fine alignment is achieved through the Transformer-UNet hybrid model. Step 3, 3D volume data generation: The aligned slides are stacked along the Z-axis to generate initial 3D volume data. A multi-scale Gaussian pyramid fusion strategy is used to eliminate gray-level jumps between slides. Wavelet transform is used to achieve lossless compression of 3D volume data. Step 4, remote adaptation and transmission: A priority queue scheduling algorithm is used to allocate computing resources. Slice transmission and MD5 verification are used to achieve breakpoint resumption of slide data transmission. 3D volume data is transmitted in a reconstruction-while-transmitting mode.

[0019] Preferably, in step 2, the improved SIFT algorithm sets the number of Gaussian difference pyramid layers to 6, the feature point detection threshold to 0.03, and extracts at least 2000 feature points per slice; feature point matching is performed using the FLANN matcher, and mismatched points are removed using the RANSAC algorithm.

[0020] Preferably, in step 3, the multi-scale Gaussian pyramid fusion adopts a 4-layer Gaussian pyramid with σ values ​​of 0.5, 1.0, 2.0, and 4.0 respectively; the wavelet transform is based on the db4 wavelet basis to achieve three-level decomposition, with a compression ratio of 10:1.

[0021] Preferably, in step 4, the priority queue scheduling algorithm divides the remote reconstruction task into three priority levels, namely emergency cases, consultation cases, and general cases. Emergency cases are processed within 5 minutes of receiving the request. The fragmented transmission adopts a fragmentation strategy of 512MB / fragment.

[0022] Beneficial effects: Compared with the prior art, the present invention has the following significant beneficial effects: 1. Breakthrough in both efficiency and accuracy of 3D reconstruction: By adopting the Transformer-UNet hybrid model and GPU parallel processing, efficiency is greatly improved and alignment accuracy is high, solving the core pain points of low efficiency and poor accuracy of existing reconstruction technologies.

[0023] 2. Significantly improved diagnostic accuracy: VR immersive observation combined with intelligent navigation allows doctors to intuitively grasp the three-dimensional structure of tumors, improving diagnostic accuracy, reducing reliance on doctors' experience, and helping primary hospitals improve their diagnostic capabilities.

[0024] 3. Optimized remote collaboration efficiency: Multiple experts can access the VR pathology model simultaneously, shortening the diagnosis time for difficult cases and solving the problems of incomplete information and inefficient communication in cross-regional consultations.

[0025] 4. Medical data security is controllable: The combination of AES-256 encryption, digital signature and breakpoint resume solution ensures no data leakage after 1000 consecutive transmissions, meeting the requirements of the "Medical Data Security Guidelines".

[0026] 5. Strong clinical adaptability: Supports diagnosis of multiple types of tumors and adaptation to multiple VR devices, and provides automatic / manual dual-mode path adjustment to meet the clinical needs of hospitals at different levels. Attached Figure Description

[0027] Figure 1 This is a diagram of the overall system architecture of the present invention; Figure 2 This is a flowchart of the 3D reconstruction submodule of the present invention; Figure 3This is a schematic diagram of the Transformer-UNet hybrid model structure of the present invention; Figure 4 This is a diagram showing the results of tumor region segmentation and path extraction in this invention. Figure 5 This is a data flow diagram for remote collaborative consultation according to the present invention; Figure 6 This is a schematic diagram of the VR interactive scene of the present invention. Specific Implementation like Figure 1 As shown, a VR-assisted three-dimensional diagnostic system for tumor pathology includes a data layer, an algorithm layer, a VR interaction layer, and a collaboration layer that are sequentially connected in communication.

[0029] The data layer uses a distributed file system to store pathological slide images, preprocessed data, and 3D volume data, supporting DICOM and TIFF formats.

[0030] The algorithm layer, deployed on a GPU server, is used to implement 3D reconstruction of pathological slides, tumor region segmentation, and observation path planning, including: (1) 3D reconstruction submodule: adopts a two-stage strategy of coarse alignment and fine alignment to realize the spatial alignment of continuous pathological slices. The fine alignment is based on the Transformer-UNet hybrid model, which includes 6 layers of Transformer encoder and 8 layers of UNet decoder.

[0031] (2) Tumor region segmentation submodule: Based on 3D U-Net+ network, the tumor region is accurately segmented. The tumor core area is located by combining density peak clustering algorithm and the tumor invasion path is extracted by 3D central axis transformation.

[0032] (3) Observation path planning submodule, which generates personalized observation paths based on the improved A* algorithm. The heuristic function of the improved A* algorithm includes a weighted sum of distance cost and lesion probability cost.

[0033] The VR interaction layer is used for real-time rendering of 3D volume data and human-computer interaction.

[0034] The collaboration layer is used for remote task scheduling, secure data transmission, and multi-user collaborative consultation. It uses the AES-256 encryption algorithm to achieve end-to-end data encryption and processes multi-terminal remote reconstruction requests through a priority queue scheduling algorithm.

[0035] like Figure 2 The execution process of the 3D reconstruction submodule in step 1 of the algorithm layer includes: First, coarse alignment is performed. Feature points of continuous pathological slices are extracted based on the improved SIFT algorithm. Feature point matching is completed through the FLANN matcher. Then, the RANSAC algorithm is combined to remove mismatched points, thus achieving preliminary spatial alignment between slices. Then, fine alignment is performed based on the Transformer-UNet hybrid model, which is as follows: Figure 3 As shown, the global features of the sliced ​​image are first extracted by a 6-layer Transformer encoder, and then the global features and local detail features are fused by an 8-layer UNet decoder to output a pixel-level displacement field. Based on this displacement field, the sub-pixel level fine alignment of the slice is completed. The Transformer-UNet hybrid model features a Transformer encoder with 12 self-attention heads per layer, a hidden layer dimension of 768, and uses the GELU activation function. The UNet decoder employs 3×3 convolutional kernels with padding of 1 and integrates a batch normalization module. The loss function of the Transformer-UNet hybrid model is the chamfer distance loss. With cross-entropy loss The weighted sum, with weights of 0.6 and 0.4 respectively, and the chamfer distance loss. The weights are used to focus on the spatial deviation between the predicted and actual displacement fields; cross-entropy loss The weights are used to constrain the accuracy of pixel-level classification / regression of the displacement field, and the specific formula is as follows: ; ; ; Where N is the total number of pixels in a single pathological slide; i is the pixel index, ranging from 1 to N, representing the i-th pixel in the slide; This represents the predicted displacement value of the i-th pixel in the x-direction; Let be the predicted displacement value of the i-th pixel in the y-direction; This represents the actual displacement value of the i-th pixel in the x-direction; is the true displacement value of the i-th pixel in the y-direction; C represents the number of categories; c is the category index, with a value ranging from 1 to C; This represents the true label of the i-th pixel in category c; This represents the predicted probability of the i-th pixel in category c.

[0036] like Figure 4 As shown, the execution process of the tumor region segmentation submodule in step 2 of the algorithm layer includes: First, tumor region segmentation is performed based on the 3D U-Net++ network. A 3D voxel block of preset size is input, and multi-scale tissue features are extracted by the network encoder and high and low layer features are fused by the decoder. The binarized segmentation results of the tumor region and normal tissue are output. Then, the density peak clustering algorithm is used to locate the tumor core area, calculate the cell density of each voxel in the segmented tumor area, select the local density maxima as the cluster center, and determine the area that meets the preset density threshold and distance threshold as the tumor core area. Finally, the central skeleton of the tumor region is extracted by 3D central axis transformation, and the invasion and expansion vector is determined by combining the gradient ascent direction of the tumor edge voxels. Based on the skeleton orientation and expansion vector, the main invasion path and branch invasion paths of the tumor are generated. The density peak clustering algorithm sets the local density threshold ρ to 1.2 × 10⁻⁶. 3 The criteria for determining the tumor core area are: cell density not less than 3 times that of normal tissue and nuclear morphological irregularity not less than 0.7, with a density of 1 cubic meter per cubic meter and a distance threshold δ of 3 mm.

[0037] In this embodiment, the execution process of the observation path planning submodule in step 3 of the algorithm layer includes: First, obtain the lesion area information such as the tumor core area, invasion path, and marginal infiltration area output by the tumor region segmentation submodule, and determine the starting point (normal tissue transition area) and ending point (tumor core area) of the path, as well as key nodes along the path (key points of the invasion path). Then construct the heuristic function for the improved A* algorithm. in, , ; As a cost of distance; The lesion probability cost is the cost of distance; the distance cost is the Euclidean distance from the current node to the target node, and the lesion probability cost is the tumor probability of the region where the current node is located (output by the tumor segmentation model).

[0038] The initial path is planned based on the heuristic function, and the constrained path covers 100% of the tumor core area and more than 90% of the invasion path; Finally, an adaptive path adjustment mechanism is provided, which supports automatic adaptation of path emphasis according to tumor type, or manual dragging of path nodes through VR interactive devices to customize the observation path. The adjusted path is synchronized to the VR interactive layer in real time.

[0039] like Figure 6 As shown, the VR interaction layer is based on the Unity 3D rendering engine and supports three interaction methods: gestures, voice, and controller buttons. The VR interaction layer integrates a voice navigation module, which triggers voice prompts in key lesion areas through a voice synthesis engine.

[0040] like Figure 5 As shown, the collaboration layer builds a real-time transmission channel based on the WebRTC protocol, adopts a combination of UDP and SRTP transmission modes, and integrates a 30ms adjustable Jitter Buffer dynamic buffering mechanism.

[0041] The 3D volume data uses VP9 video encoding, supporting VR scene synchronization for up to 5 people online simultaneously.

[0042] It provides 3D drawing tools, precise measurement and text / voice annotation tools, and the annotation information is synchronized to all online user terminals in real time, retaining the annotation history of the most recent 100 times.

[0043] Applied to Figure 1-6 A method for 3D reconstruction of pathological sections includes the following steps: Preprocessing: The digital images of pathological sections were subjected to staining normalization, wrinkle repair and cropping scaling in sequence. Staining normalization used an improved Macenko algorithm to convert the RGB image to optical density space.

[0044] The improved Macenko algorithm includes: a. Adaptive background purification technology before optical density space conversion, which accurately removes blank areas of the slide and scans pixels of impurities and necrotic tissue through Otsu threshold segmentation; b. Introduce a multi-reference image fusion strategy, and combine HE staining pathological prior knowledge to constrain the range of hematoxylin and eosin core staining vectors. This breaks through the limitation of poor generalization of the original algorithm with a single reference template, and adapts to the staining differences of pathological slides from different hospitals and different batches. c. Adaptive staining normalization technology dynamically adjusts the normalization coefficient and incorporates a mild Gaussian noise suppression mechanism. While achieving a unified global staining style, it fully preserves tissue texture details and avoids local staining deviations and scanning noise affecting the accuracy of subsequent coarse and fine alignment of sections.

[0045] (2) Two-stage alignment: Based on the improved SIFT algorithm, feature points of the slices are extracted and coarse alignment is achieved. Then, the Transformer-UNet hybrid model is used to achieve sub-pixel level fine alignment. Among them: the improved SIFT algorithm sets the number of Gaussian difference pyramid layers to 6 and the feature point detection threshold to 0.03, and extracts at least 2000 feature points for each slice; feature point matching is performed by FLANN matcher and RANSAC algorithm is used to remove mismatched points.

[0046] (3) 3D volume data generation: The aligned slices are stacked along the Z-axis to generate initial 3D volume data. A multi-scale Gaussian pyramid fusion strategy is used to eliminate gray-level jumps between slices, and lossless compression of the 3D volume data is achieved through wavelet transform. Among them: the multi-scale Gaussian pyramid fusion uses a 4-layer Gaussian pyramid with σ values ​​of 0.5, 1.0, 2.0 and 4.0 respectively; the wavelet transform is based on the db4 wavelet basis to achieve three-level decomposition, and the compression ratio reaches 10:1.

[0047] (4) Remote Adaptive Transmission: A priority queue scheduling algorithm is used to allocate computing resources. The data slices are resumed from where they broke through by using slice transmission and MD5 checksum. The 3D volume data is transmitted in a reconstruction-while-retransmitting mode. Specifically, the priority queue scheduling algorithm divides the remote reconstruction task into three priority levels, namely emergency cases, consultation cases, and general cases. Emergency cases are processed within 5 minutes of receiving the request. The slice transmission adopts a slice strategy of 512MB / slice.

Claims

1. A VR-assisted three-dimensional diagnostic system for tumor pathology, characterized in that: This includes a data layer, an algorithm layer, a VR interaction layer, and a collaboration layer that are connected in sequence. The data layer employs a distributed file system for storing pathological slide images, preprocessed data, and 3D volume data. The algorithm layer, deployed on a GPU server, is used to implement 3D reconstruction of pathological slides, tumor region segmentation, and observation path planning, including: 3D Reconstruction Submodule: A two-stage strategy of coarse alignment and fine alignment is adopted to achieve spatial alignment of continuous pathological slices. The fine alignment is based on the Transformer-UNet hybrid model, which includes a 6-layer Transformer encoder and an 8-layer UNet decoder. Tumor region segmentation submodule: Accurate tumor region segmentation is achieved based on 3D U-Net+ network, the tumor core area is located by density peak clustering algorithm, and the tumor invasion path is extracted by 3D midline transformation; The observation path planning submodule generates personalized observation paths based on the improved A* algorithm. The heuristic function of the improved A* algorithm includes a weighted sum of distance cost and lesion probability cost. The VR interaction layer is used for real-time rendering of 3D volume data and human-computer interaction. Based on the tumor core area location, invasion path and personalized observation path output by the algorithm layer, corresponding visualization marking and navigation guidance are performed in the rendering scene. The aforementioned collaboration layer is used for remote task scheduling, secure data transmission, and multi-user collaborative consultation. It employs the AES-256 encryption algorithm to achieve end-to-end data encryption and uses a priority queue scheduling algorithm to process remote reconstruction requests from multiple terminals.

2. The VR-assisted three-dimensional tumor pathology diagnostic system according to claim 1, characterized in that: The execution process of the 3D reconstruction submodule in the algorithm layer includes: First, coarse alignment is performed. Feature points of continuous pathological slices are extracted based on the improved SIFT algorithm. Feature point matching is completed through the FLANN matcher. Then, the RANSAC algorithm is combined to remove mismatched points, thus achieving preliminary spatial alignment between slices. Then, fine alignment is performed based on the Transformer-UNet hybrid model. The Transformer-UNet hybrid model first extracts the global features of the slice image through a 6-layer Transformer encoder, and then fuses the global features and local detail features through an 8-layer UNet decoder to output a pixel-level displacement field. Subpixel-level fine alignment of the slice is then completed based on this displacement field. The Transformer-UNet hybrid model has a Transformer encoder with 12 self-attention mechanisms per layer, a hidden layer dimension of 768, and uses the GELU activation function. The UNet decoder uses a 3×3 convolutional kernel with padding of 1 and integrates a batch normalization module; The loss function of the Transformer-UNet hybrid model is the chamfer distance loss. With cross-entropy loss The weighted sum, with weights of 0.6 and 0.4 respectively, and the chamfer distance loss. The weights are used to focus on the spatial deviation between the predicted and actual displacement fields; cross-entropy loss The weights are used to constrain the accuracy of pixel-level classification / regression of the displacement field, and the specific formula is as follows: ； ； ； Where N is the total number of pixels in a single pathological slide; i is the pixel index, ranging from 1 to N, representing the i-th pixel in the slide; This represents the predicted displacement value of the i-th pixel in the x-direction; Let be the predicted displacement value of the i-th pixel in the y-direction; This represents the actual displacement value of the i-th pixel in the x-direction; is the true displacement value of the i-th pixel in the y-direction; C represents the number of categories; c is the category index, with a value ranging from 1 to C; This represents the true label of the i-th pixel in category c; This represents the predicted probability of the i-th pixel in category c.

3. The VR-assisted three-dimensional tumor pathology diagnostic system according to claim 1, characterized in that: The execution process of the tumor region segmentation submodule in the algorithm layer includes: First, tumor region segmentation is performed based on the 3D U-Net++ network. A 3D voxel block of a preset size is input, and multi-scale tissue features are extracted by the network encoder and high and low layer features are fused by the decoder. The result is a binarized segmentation of the tumor region and normal tissue. Then, the density peak clustering algorithm is used to locate the tumor core area, calculate the cell density of each voxel in the segmented tumor area, select the local density maxima as the cluster center, and determine the area that meets the preset density threshold and distance threshold as the tumor core area. Finally, the central skeleton of the tumor region is extracted by 3D central axis transformation, and the invasion and expansion vector is determined by combining the gradient ascent direction of the tumor edge voxels. Based on the skeleton orientation and expansion vector, the main invasion path and branch invasion paths of the tumor are generated. The density peak clustering algorithm sets the local density threshold ρ to 1.2 × 10⁻⁶. 3 The criteria for determining the tumor core area are: a cell density of not less than 3 times that of normal tissue and a nuclear morphological irregularity of not less than 0.7, with a density of 3 mm per cubic meter and a distance threshold δ of 3 mm.

4. The VR-assisted three-dimensional tumor pathology diagnostic system according to claim 1, characterized in that: The execution process of the observation path planning submodule in the algorithm layer includes: First, obtain the lesion area information of the tumor core area, invasion path, and marginal infiltration area output by the tumor region segmentation submodule, and determine the starting point (normal tissue transition zone), ending point (tumor core area), and key nodes along the path (key points of the invasion path). Reconstruct the heuristic function for the improved A* algorithm: in, , ; As a cost of distance; The distance cost is the Euclidean distance from the current node to the target node, and the lesion probability cost is the tumor probability in the region where the current node is located, output by the tumor segmentation model. The initial path is planned based on the heuristic function described above, and the constrained path covers 100% of the tumor core area and more than 90% of the invasion path. Finally, an adaptive path adjustment mechanism is provided, which supports automatic adaptation of path emphasis according to tumor type, or manual dragging of path nodes through VR interactive devices to customize the observation path. The adjusted path is synchronized to the VR interactive layer in real time.

5. The VR-assisted three-dimensional tumor pathology diagnostic system according to claim 1, characterized in that: The VR interaction layer is based on the Unity 3D rendering engine and supports three interaction methods: gestures, voice, and controller buttons. The VR interaction layer integrates a voice navigation module, which triggers voice prompts in key lesion areas through a voice synthesis engine.

6. The VR-assisted three-dimensional tumor pathology diagnostic system according to claim 1, characterized in that: The aforementioned collaboration layer is based on the WebRTC protocol to build a real-time transmission channel, adopts a combined UDP and SRTP transmission mode, and integrates a 30ms adjustable JitterBuffer dynamic buffering mechanism. The 3D volume data uses VP9 video encoding, supporting simultaneous online VR scene synchronization for up to 5 people; It provides 3D drawing tools, precise measurement and text / voice annotation tools, and the annotation information is synchronized to all online user terminals in real time, retaining the annotation history of the most recent 100 times.

7. A method for 3D reconstruction of pathological sections, applied to the VR-assisted three-dimensional diagnostic system for tumor pathology as described in any one of claims 1-6, characterized in that: Includes the following steps: Step 1, Preprocessing: The digital images of pathological sections are subjected to staining normalization, wrinkle repair and cropping and scaling in sequence. The staining normalization uses the improved Macenko algorithm to convert the RGB image to optical density space. The improved Macenko algorithm includes: a. Adaptive background purification technology before optical density space conversion, which accurately removes blank areas of the slide and scans pixels of impurities and necrotic tissue through Otsu threshold segmentation; b. Introduce a multi-reference image fusion strategy, and combine HE staining pathological prior knowledge to constrain the range of hematoxylin and eosin core staining vectors. This breaks through the limitation of poor generalization of the original algorithm with a single reference template, and adapts to the staining differences of pathological slides from different hospitals and different batches. c. Adaptive staining normalization technology dynamically adjusts the normalization coefficient and incorporates a mild Gaussian noise suppression mechanism. While achieving a unified global staining style, it fully preserves tissue texture details and avoids local staining deviations and scanning noise affecting the accuracy of subsequent coarse and fine alignment of slices. Step 2, Two-stage alignment: Based on the improved SIFT algorithm, feature points of slices are extracted and coarse alignment is achieved. Then, sub-pixel-level fine alignment is achieved through the Transformer-UNet hybrid model. Step 3, 3D volume data generation: The aligned slices are stacked along the Z-axis to generate initial 3D volume data. A multi-scale Gaussian pyramid fusion strategy is used to eliminate grayscale jumps between slices. Wavelet transform is used to achieve lossless compression of 3D volume data. Step 4, Remote adaptation and transmission: A priority queue scheduling algorithm is used to allocate computing resources. Slice transmission and MD5 verification are used to achieve breakpoint resumption of slice data transmission. 3D volume data is transmitted in a reconstruction-while-transmitting mode.

8. The method for 3D reconstruction of pathological sections according to claim 7, characterized in that, In step 2, the improved SIFT algorithm sets the number of Gaussian difference pyramid layers to 6 and the feature point detection threshold to 0.03, extracting at least 2000 feature points from each slice; feature point matching is performed using the FLANN matcher, and mismatched points are removed using the RANSAC algorithm.

9. The method for 3D reconstruction of pathological sections according to claim 7, characterized in that, In step 3, the multi-scale Gaussian pyramid fusion uses a 4-layer Gaussian pyramid with σ values ​​of 0.5, 1.0, 2.0, and 4.0 respectively; the wavelet transform is based on the db4 wavelet basis to achieve three-level decomposition, with a compression ratio of 10:

1.

10. The method for 3D reconstruction of pathological sections according to claim 7, characterized in that, In step 4, the priority queue scheduling algorithm divides the remote reconstruction task into three priority levels: emergency cases, consultation cases, and general cases. Emergency cases are processed within 5 minutes of receiving the request. The fragmented transmission adopts a fragmentation strategy of 512MB / fragment.

Images