High-voltage steep pulse tumor ablation visualization system and method based on augmented reality technology

Abstract

The present invention provides a high-voltage steep pulse tumor ablation visualization system and method based on augmented reality technology, wherein the system comprises an intraoperative image acquisition device, a computing processing unit, an AR projection device, a display device, and a high-voltage steep pulse generating device; the method comprises the following steps: S1) intraoperative image acquisition, S2) data preprocessing, S3) ablation area segmentation, S4) three-dimensional reconstruction, S5) multimodal image fusion, S6) AR projection, and S7) tumor ablation surgical operation. The advantages of the present invention are: high surgical accuracy, enhanced intraoperative intuitive perception, reduced intraoperative risks, and improved surgical safety.

Description

Technical Field

[0001] The present invention relates to the technical field of medical imaging and tumor treatment, and in particular to a high-voltage steep pulse tumor ablation visualization system and method based on augmented reality technology. Background Art

[0002] High-voltage steep pulse technology has been widely used in tumor ablation therapy. It achieves the therapeutic effect by generating high-voltage pulses in a very short time to destroy tumor cells. During the actual operation, doctors need to understand the situation of the ablation area and surrounding tissues in real time to ensure the accuracy and safety of ablation.

[0003] In the existing technology, traditional imaging equipment is difficult to provide real-time and intuitive visual feedback of the intraoperative ablation process, which increases the difficulty and risk of surgical operations; the lack of real-time feedback of the intraoperative ablation process may lead to incomplete ablation or damage to surrounding healthy tissues. Summary of the invention

[0004] The purpose of the present invention is to provide a high-voltage steep pulse tumor ablation visualization system and method based on augmented reality technology, which has high surgical accuracy, enhances intraoperative intuitive perception, reduces intraoperative risks, and improves surgical safety.

[0005] In order to achieve the above object, the present invention is implemented by the following technical solutions:

[0006] A high-voltage steep pulse tumor ablation visualization system based on augmented reality technology, including an intraoperative image acquisition device, a computing processing unit, an AR projection device, a display device, and a high-voltage steep pulse generating device;

[0007] The intraoperative image acquisition device includes an intraoperative ultrasound device, an intraoperative CT device, and an intraoperative MRI device. The intraoperative imaging device collects data in real time and performs data integration, time synchronization, and spatial calibration.

[0008] The computing processing unit performs data preprocessing, ablation area segmentation, three-dimensional reconstruction, and multi-modal image fusion processing on the data integrated by the intraoperative image acquisition device; wherein the data preprocessing includes image enhancement and noise removal, the ablation area segmentation is achieved by a convolutional neural network image segmentation algorithm, and the three-dimensional reconstruction obtains a three-dimensional reconstructed image R of the intraoperative ablation area and surrounding tissues. 3D Multimodal image fusion combines the image data of different modalities obtained from intraoperative ultrasound equipment, intraoperative CT equipment, and intraoperative MRI equipment to generate a comprehensive image I fusion ;

[0009] The AR projection device reconstructs the three-dimensional image R 3D and integrated image I fusionCalculation is performed through projection to generate AR projection images and project them into the surgical field of view;

[0010] The display device is an AR glasses, through which the surgeon can view the augmented reality images of the ablation area and surrounding tissues in real time;

[0011] The high-voltage steep pulse generating device is used to generate high-voltage steep pulses, and the surgeon performs surgery on the patient with the assistance of AR glasses through the high-voltage steep pulse generating device.

[0012] The high-voltage steep pulse tumor ablation visualization method based on augmented reality technology includes the following steps:

[0013] S1) Intraoperative Image Acquisition

[0014] Acquisition and integration of intraoperative ultrasound equipment (I US )、Intraoperative CT equipment (I CT ), intraoperative MRI equipment (I MRI ) image data: I(t)={I US (t), I CT (t), I MRI (t)}; where I(t) is the set of all image data acquired at time t;

[0015] and perform temporal synchronization and spatial calibration on the integrated imaging data;

[0016] S2) Data preprocessing

[0017] S2-1) Image Enhancement

[0018] Perform image enhancement processing on the collected original image data to improve the contrast and clarity of the image. The formula is I enh (t) = f enh (I(t)); where, I enh (t) is the enhanced image data, f enh is the image enhancement function;

[0019] S2-2) Noise Removal

[0020] Eliminate noise in the image and improve image quality. The formula is I pre (t) = f pre (I enh (t)); where I pre (t) is the denoised image data, f pre is the noise removal function;

[0021] S3) Ablation area segmentation

[0022] S3-1) Data input

[0023] Input preprocessed image data I pre (t), where t is the time variable, I pre (t) represents the image data collected at different time points;

[0024] S3-2) Convolutional neural network structure design

[0025] The network structure includes multiple convolutional layers, pooling layers and fully connected layers. The specific structure is as follows:

[0026] Convolutional layer: extracts local features of the image. The formula is Conv(I pre (t))=∑ i，j I pre (x+i,y+j,t)·W ij +b; where Conv(I pre (t)) is the output of the convolutional layer, I pre (x, y, t) is the pixel value of the preprocessed image data at position (x, y) and time t, i, j is the index of the convolution kernel, W ij is the convolution kernel weight, b is the bias;

[0027] Pooling layer: Reduce the dimension of feature map and reduce the amount of calculation. The formula is Pool(I conv )=max i，j I conv (x+i, y+j); where Pool(I conv ) is the output of the pooling layer, I conv (x, y) is the pixel value at position (x, y) output by the convolutional layer, and max is the maximum pooling operation;

[0028] Fully connected layer: The features extracted by the convolutional layer and the pooling layer are integrated for classification. The formula is FC(I pool )=σ(W fc I pool +b fc );wherein, FC(I pool ) is the output of the fully connected layer, I pool is the output of the pooling layer, W fe is the weight matrix of the fully connected layer, b fc is the bias vector of the fully connected layer, σ is the activation function; S3-3) Image segmentation

[0029] The preprocessed image data I pre (t) is input into the trained convolutional neural network, which will automatically perform feature extraction and region segmentation and output the ablation region R(t), with the formula R(t) = CNN(I pre (t)); where R(t) is the segmentation result at time t;

[0030] S3-4) Post-processing of segmentation results

[0031] Perform morphological processing and edge correction on the segmentation result R(t) to improve the accuracy of the segmentation result;

[0032] S4) 3D reconstruction

[0033] The 3D reconstruction calculation is based on volume integral rendering, and the formula is R 3D =∫ V R edge (t)dV; where R 3D is the reconstructed three-dimensional image, V is the reconstructed volume, dV is the volume element, R edge (t) is the ablation area after segmentation and edge correction at time t;

[0034] S5) Multimodal image fusion

[0035] The image data of different modalities are fused to generate a comprehensive image, the formula is I fusion =αI US (t)+βI CT (t)+γI MRI (t); where I fusion For comprehensive images, I US (t) is the ultrasound image at time t, I CT (t) is the CT image at time t, I MRI (t) is the MRI image at time t, α, β, γ are the reconstruction volume weight coefficients, satisfying α+β+γ=1;

[0036] S6) AR projection

[0037] The three-dimensional reconstructed image R 3D and integrated image I fusion The input AR projection device performs calculation through projection to generate an AR projection image;

[0038] The generated AR projection image is projected into the surgical field, and the surgeon uses AR glasses to view the augmented reality image of the ablation area and surrounding tissue structures in real time;

[0039] S7) Tumor ablation surgery

[0040] With the help of AR glasses, surgeons can view the augmented reality images of the ablation area and surrounding tissue structures in real time while performing the surgery using a high-voltage steep pulse generator.

[0041] During the surgical operation, if the real-time image changes, steps S1) to S6) are repeated to obtain updated real-time augmented reality images until the operation is completed.

[0042] Furthermore, in step S1), the time synchronization formula is: US =t CT =t MRI =t; where t US is the timestamp of ultrasound image data, t CT is the timestamp of the CT image data, t MRI It is the timestamp of MRI image data;

[0043] The formula for spatial calibration is: I cal (x, y, z) = f cal (I US (x, y, z), I CT (x, y, z), I MRI (x, y, z)); where I cal (x, y, z) is the calibrated image data, f cal is the spatial calibration function, (x, y, z) is the spatial coordinate;

[0044] Furthermore, in step S3-4), the morphological processing formula is R morph (t) = Morph(R(t)); where R morph (t) is the segmentation result after morphological processing, Morph is the morphological processing function, including erosion and dilation;

[0045] The edge correction formula is R edge (t)=Edge(R morph (t)); where R edge (t) is the segmentation result after edge correction, and Edge is the edge detection and correction function.

[0046] Compared with the prior art, the present invention has the following advantages:

[0047] The high-voltage steep pulse tumor ablation visualization system of the present invention is based on augmented reality technology, which has high surgical accuracy, enhances intuitive perception during surgery, reduces intraoperative risks, and improves surgical safety.

[0048] (1) High surgical precision: Providing real-time visual feedback of the ablation area during surgery to help doctors perform precise operations and reduce the risk of misoperation.

[0049] (2) Enhanced intuitive perception during surgery: AR technology is used to enhance the doctor's intuitive perception of the ablation area and surrounding tissue structures, simplify the surgical process, and improve operational efficiency.

[0050] (3) Reduce intraoperative risks: Provide accurate real-time information on the ablation area and surrounding tissue structures to help doctors adjust ablation strategies in a timely manner and reduce the risk of damage to healthy tissues.

[0051] (4) Improve surgical safety: Through multimodal image fusion and efficient three-dimensional reconstruction technology, the accuracy and comprehensiveness of image data can be ensured, thereby improving the safety and effectiveness of surgery. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a schematic diagram of the system module of the present invention.

[0053] Figure 2 It is a schematic flow chart of the method of the present invention. DETAILED DESCRIPTION

[0054] The embodiments of the present invention are described in further detail below in conjunction with the accompanying drawings.

[0055] The high-voltage steep pulse tumor ablation visualization system and method based on augmented reality technology of the present invention are innovative in the acquisition and processing of real-time images during surgery, the application of AR technology, and multimodal image fusion. Specifically, through the intraoperative image acquisition device and the computing processing unit, the real-time acquisition and processing of intraoperative images are realized to ensure the accurate presentation of the ablation area and surrounding tissue structures; the three-dimensional images of the ablation area and surrounding tissue structures are projected into the surgical field of view using augmented reality technology to provide intuitive intraoperative visualization feedback; and the image data fusion of multiple imaging devices (such as ultrasound, CT, MRI) is supported to improve the accuracy and comprehensiveness of the images.

[0056] like Figure 1 As shown, a high-voltage steep pulse tumor ablation visualization system based on augmented reality technology includes an intraoperative image acquisition device, a computing processing unit, an AR projection device, a display device, and a high-voltage steep pulse generating device;

[0057] The intraoperative image acquisition device includes an intraoperative ultrasound device, an intraoperative CT device, and an intraoperative MRI device. The intraoperative imaging device collects data in real time and performs data integration, time synchronization, and spatial calibration.

[0058] The computing processing unit performs data preprocessing, ablation area segmentation, three-dimensional reconstruction, and multi-modal image fusion processing on the data integrated by the intraoperative image acquisition device; wherein the data preprocessing includes image enhancement and noise removal, the ablation area segmentation is achieved by a convolutional neural network image segmentation algorithm, and the three-dimensional reconstruction obtains a three-dimensional reconstructed image R of the intraoperative ablation area and surrounding tissues. 3D Multimodal image fusion combines the image data of different modalities obtained from intraoperative ultrasound equipment, intraoperative CT equipment, and intraoperative MRI equipment to generate a comprehensive image I fusion ;

[0059] The AR projection device reconstructs the three-dimensional image R 3D and integrated image I fusion Through projection, calculation is performed to generate AR projection images and project them into the surgical field of view;

[0060] The display device is an AR glasses or a helmet implanted with AR technology, and the surgeon uses the AR glasses to view the augmented reality images of the ablation area and surrounding tissues in real time;

[0061] The high-voltage steep pulse generating device is used to generate high-voltage steep pulses, and the surgeon performs surgery on the patient with the assistance of AR glasses through the high-voltage steep pulse generating device.

[0062] like Figure 2 As shown, the high-voltage steep pulse tumor ablation visualization method based on augmented reality technology includes the following steps:

[0063] S1) Intraoperative Image Acquisition

[0064] Acquisition and integration of intraoperative ultrasound equipment (I US )、Intraoperative CT equipment (I CT ), intraoperative MRI equipment (I MRI ) image data: I(t)={I US (t), I CT (t), I MRI (t)}; where I(t) is the set of all image data acquired at time t;

[0065] and perform temporal synchronization and spatial calibration on the integrated image data;

[0066] Specifically, intraoperative ultrasound equipment (I US ) is used to obtain real-time ultrasound images of the surgical area, with high frame rate and real-time performance; intraoperative CT equipment (I CT ) is used to obtain high-resolution tomographic images and provide detailed anatomical information of the surgical area; intraoperative MRI equipment (I MRI ) is used to obtain high-contrast soft tissue images to help distinguish tumor tissue from healthy tissue.

[0067] S2) Data preprocessing

[0068] S2-1) Image Enhancement

[0069] Perform image enhancement processing on the collected original image data to improve the contrast and clarity of the image. The formula is I enh (t) = f enh (I(t)); where, I enh (t) is the enhanced image data, f enhis the image enhancement function;

[0070] S2-2) Noise Removal

[0071] Eliminate noise in the image and improve image quality. The formula is I pre (t) = f pre (I enh (t)); where I pre (t) is the denoised image data, f pre is the noise removal function.

[0072] S3) Ablation area segmentation

[0073] S3-1) Data input

[0074] Input preprocessed image data I pre (t), where t is the time variable, I pre (t) represents the image data collected at different time points;

[0075] S3-2) Convolutional neural network structure design

[0076] The network structure includes multiple convolutional layers, pooling layers and fully connected layers. The specific structure is as follows:

[0077] Convolutional layer: extracts local features of the image. The formula is Conv(I pre (t))=∑ i，j I pre (x+i,y+j,t)·W ij +b; where Conv(I pre (t)) is the output of the convolutional layer, I pre (x, y, z) is the pixel value of the preprocessed image data at position (x, y) and time t, i, j is the index of the convolution kernel, W ij is the convolution kernel weight, b is the bias;

[0078] Pooling layer: Reduce the dimension of feature map and reduce the amount of calculation. The formula is Pool(I conv )=max i，j I conv (x+i, y+j); where Pool(I conv ) is the output of the pooling layer, I conv (x, y) is the pixel value at position (x, y) output by the convolutional layer, and max is the maximum pooling operation;

[0079] Fully connected layer: The features extracted by the convolutional layer and the pooling layer are integrated for classification. The formula is FC(I pool )=σ(W fc I pool +b fc);wherein, FC(I pool ) is the output of the fully connected layer, I pool is the output of the pooling layer, W fc is the weight matrix of the fully connected layer, b fc is the bias vector of the fully connected layer, σ is the activation function; S3-3) Image segmentation

[0080] The preprocessed image data I pre (t) is input into the trained convolutional neural network, which will automatically perform feature extraction and region segmentation and output the ablation region R(t), with the formula R(t) = CNN(I pre (t)); where R(t) is the segmentation result at time t;

[0081] S3-4) Post-processing of segmentation results

[0082] The segmentation result R(t) is subjected to morphological processing and edge correction to improve the accuracy of the segmentation result.

[0083] S4) 3D reconstruction

[0084] The 3D reconstruction calculation is based on volume integral rendering, and the formula is R 3D =∫ V R edge (t)dV; where R 3D is the reconstructed three-dimensional image, V is the reconstructed volume, dV is the volume element, R edge (t) is the ablation area after segmentation and edge correction at time t;

[0085] The purpose of three-dimensional reconstruction is to provide a stereoscopic view of the ablation area during surgery, so that the surgeon can better understand the ablation situation.

[0086] S5) Multimodal image fusion

[0087] The image data of different modalities are fused to generate a comprehensive image, the formula is I fusion =αI US (t)+βI CT (t)+γI MRI (t); where I fusion For comprehensive images, I US (t) is the ultrasound image at time t, I CT (t) is the CT image at time t, I MRI (t) is the MRI image at time t, α, β, γ are the reconstruction volume weight coefficients, satisfying α+β+γ=1.

[0088] S6) AR projection

[0089] The three-dimensional reconstructed image R3D and integrated image I fusion The input AR projection device performs calculation through projection to generate an AR projection image;

[0090] The generated AR projection image is projected into the surgical field, and the surgeon uses AR glasses to view the augmented reality image of the ablation area and surrounding tissue structures in real time.

[0091] S7) Tumor ablation surgery

[0092] With the help of AR glasses, surgeons can view the augmented reality images of the ablation area and surrounding tissue structures in real time while performing the surgery using a high-voltage steep pulse generator.

[0093] During the surgical operation, if the real-time image changes, steps S1) to S6) are repeated to obtain updated real-time augmented reality images until the operation is completed.

[0094] Furthermore, in step S1), the time synchronization formula is: US =t CT =t MRI =t; where t US is the timestamp of ultrasound image data, t CT is the timestamp of the CT image data, t MRI It is the timestamp of MRI image data;

[0095] Each imaging device achieves time synchronization through synchronization signals to ensure that the image data acquired at the same time point has time consistency;

[0096] The formula for spatial calibration is: I cal (x, y, z) = f cal (I US (x, y, z), I CT (x, y, z), I MRI (x, y, z)); where I cal (x, y, z) is the calibrated image data, f cal is the spatial calibration function, (x, y, z) is the spatial coordinate;

[0097] The spatial calibration algorithm is used to perform spatial calibration on the image data acquired by different imaging devices to ensure that each image data is aligned in the same spatial coordinate system.

[0098] Furthermore, in step S3-4), the morphological processing formula is R morph (t) = Morph(R(t)); where R morph(t) is the segmentation result after morphological processing, Morph is the morphological processing function, including erosion and dilation;

[0099] The edge correction formula is R edge (t)=Edge(R morph (t)); where R edge (t) is the segmentation result after edge correction, and Edge is the edge detection and correction function.

[0100] The above description is only a preferred embodiment of the present invention. It should be pointed out that ordinary technicians in this technical field can make several improvements and modifications without departing from the concept of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. High-voltage steep pulse tumor ablation visualization system based on augmented reality technology, characterized by: It includes an intraoperative image acquisition device, a computing processing unit, an AR projection device, a display device and a high-voltage steep pulse generating device; The intraoperative image acquisition device includes an intraoperative ultrasound device, an intraoperative CT device, and an intraoperative MRI device. The intraoperative image acquisition device collects data in real time and performs data integration, time synchronization, and spatial calibration. The computing processing unit performs data preprocessing, ablation area segmentation, three-dimensional reconstruction, and multi-modal image fusion processing on the data integrated by the intraoperative image acquisition device; wherein the data preprocessing includes image enhancement and noise removal, the ablation area segmentation is achieved by a convolutional neural network image segmentation algorithm, and the three-dimensional reconstruction obtains a three-dimensional reconstructed image R of the intraoperative ablation area and surrounding tissues. 3D Multimodal image fusion combines the image data of different modalities obtained from intraoperative ultrasound equipment, intraoperative CT equipment, and intraoperative MRI equipment to generate a comprehensive image I fusion ; The AR projection device reconstructs the three-dimensional image R 3D and integrated image I fusion Calculation is performed through projection to generate AR projection images and project them into the surgical field of view; The display device is an AR glasses, through which the surgeon can view the augmented reality images of the ablation area and surrounding tissues in real time; The high-voltage steep pulse generating device is used to generate high-voltage steep pulses, and the surgeon performs surgery on the patient with the assistance of AR glasses through the high-voltage steep pulse generating device.

2. The high-voltage steep pulse tumor ablation visualization system based on augmented reality technology according to claim 1, characterized in that: The method for segmenting the ablation area in the computing processing unit is as follows: S3-1) Data input Input preprocessed image data I pre (t), where t is the time variable, I pre (t) represents the image data collected at different time points; S3-2) Convolutional neural network structure design The network structure includes multiple convolutional layers, pooling layers and fully connected layers. The specific structure is as follows: Convolutional layer: extract local features of the image, the formula is Among them, Conv(I pre (t)) is the output of the convolutional layer, I pre (x, y, t) is the pixel value of the preprocessed image data at position (x, y) and time t, i, j is the index of the convolution kernel, W ij is the convolution kernel weight, b is the bias; Pooling layer: Reduce the dimension of feature map and reduce the amount of calculation. The formula is Pool(I conv )=max i，j I conv (x+i, y+j); where Pool(I conv ) is the output of the pooling layer, I conv (x, y) is the pixel value at position (x, y) output by the convolutional layer, and max is the maximum pooling operation; Fully connected layer: The features extracted by the convolutional layer and the pooling layer are integrated for classification. The formula is FC(I pool )=σ(W fc I pool +b fc );wherein, FC(I pool ) is the output of the fully connected layer, I pool is the output of the pooling layer, W fc is the weight matrix of the fully connected layer, b fc is the bias vector of the fully connected layer, σ is the activation function; S3-3) Image segmentation The preprocessed image data I pre (t) is input into the trained convolutional neural network, which will automatically perform feature extraction and region segmentation and output the ablation region R(t), with the formula R(t) = CNN(I pre (t)); where R(t) is the segmentation result at time t; S3-4) Post-processing of segmentation results The segmentation result R(t) is subjected to morphological processing and edge correction to improve the accuracy of the segmentation result.

3. The high-voltage steep pulse tumor ablation visualization system based on augmented reality technology according to claim 1, characterized in that: The method for three-dimensional reconstruction in the computing processing unit is as follows: The 3D reconstruction calculation is based on volume integral rendering, and the formula is R 3D =∫ V R edge (t)dV; where R 3D is the reconstructed three-dimensional image, V is the reconstructed volume, dV is the volume element, R edge (t) is the ablation area after segmentation and edge correction at time t.

4. The high-voltage steep pulse tumor ablation visualization system based on augmented reality technology according to claim 1, characterized in that: The method for fusing multimodal images in the computing processing unit is as follows: The image data of different modalities are fused to generate a comprehensive image, the formula is I fusion =αI US (t)+βI CT (t)+γI MRI (t); where I fusion For comprehensive images, I US (t) is the ultrasound image at time t, I CT (t) is the CT image at time t, I MRI (t) is the MRI image at time t, α, β, γ are the reconstruction volume weight coefficients, satisfying α+β+γ=1.

5. The high-voltage steep pulse tumor ablation visualization system based on augmented reality technology according to claim 2, characterized in that: In step S3-4), the morphological processing formula is R morph (t) = Morph(R(t)); where R morph (t) is the segmentation result after morphological processing, Morph is the morphological processing function, including erosion and dilation; The edge correction formula is R edge (t)=Edge(R morph (t)); where R edge (t) is the segmentation result after edge correction, and Edge is the edge detection and correction function.

Images