Multi-mode lymphedema evaluation and surgical navigation system based on image processing

Abstract

The invention relates to the technical field of medical equipment, and provides a multi-modal lymphedema assessment and surgical navigation system based on image processing, which comprises a 3D scanning modeling module used for acquiring three-dimensional point cloud data of the body surface of a patient through structured light 3D scanning equipment, an iconography examination analysis module used for performing U-Net image segmentation on CT / MRI image data, and an image processing module used for processing the CT / MRI image data. The ultrasonic result calculation module is used for carrying out Canny edge detection on an ultrasonic image to determine the boundary of a surgical site, and the data integration and navigation generation module is used for realizing multi-modal data fusion through mutual information maximization, generating a surgical path based on FMM and carrying out real-time navigation in combination with an AR technology. According to the method, objective evaluation of lymphedema and accurate navigation of the operation are realized through accurate integration of multi-modal data and an image processing technology, the diagnosis and treatment accuracy is effectively improved, the operation risk is reduced, complications are reduced, and a scientific basis is provided for personalized treatment.

Description

Technical Field

[0001] This invention relates to the field of medical device technology, specifically to a multimodal lymphedema assessment and surgical navigation system based on image processing. Background Technology

[0002] Lymphedema is a common and challenging disease to treat. Current assessment methods for lymphedema, such as circumference measurement and volume measurement using the water displacement method, have many drawbacks. They are neither objective nor can they accurately represent specific locations, causing significant difficulties in clinical diagnosis and treatment. This makes it impossible to accurately determine the severity of the patient's condition and the effectiveness of treatment, thus affecting the formulation and adjustment of subsequent treatment plans. In terms of surgical treatment, existing surgical methods lack precise navigation guidance, which can easily cause unnecessary damage to non-surgical areas, affecting surgical outcomes and patient recovery. Summary of the Invention

[0003] To address the shortcomings of existing technologies, this invention provides a multimodal lymphedema assessment and surgical navigation system based on image processing, which solves the problems of existing lymphedema assessment methods being neither objective nor accurate, and the lack of precise surgical navigation that can easily damage non-surgical sites.

[0004] To achieve the above objectives, the present invention provides the following technical solution: a multimodal lymphedema assessment and surgical navigation system based on image processing, comprising: The 3D scanning and modeling module is used to acquire three-dimensional point cloud data of the patient's body surface through a structured light 3D scanning device and to construct a three-dimensional model based on the Poisson surface reconstruction algorithm. The imaging examination and analysis module is used to perform U-Net image segmentation on CT / MRI image data, optimize the segmentation results by combining the Dice loss function, and realize the spatial registration of image data and 3D model through homogeneous coordinate transformation and iterative ICP algorithm. The ultrasound results calculation module is used to perform Canny edge detection on ultrasound images to determine the boundaries of the surgical site, and to calculate tissue depth by combining ultrasound propagation time; The data integration and navigation generation module is used to maximize multimodal data fusion through mutual information, generate surgical paths based on FMM, and perform real-time navigation in conjunction with AR technology.

[0005] Preferably, in the 3D scanning modeling module, the calculation method for 3D point cloud data is based on the principle of triangulation, establishing a mapping relationship between image point coordinates (u,v) and spatial point coordinates (X,Y,Z) through the camera intrinsic parameter matrix K and extrinsic parameter matrix [R|t]. , Where (u,v) are the pixel coordinates of the image point on the imaging plane. The three-dimensional point cloud is obtained by solving the above equation for a large number of image points.

[0006] Preferably, the Poisson surface reconstruction algorithm converts discrete point clouds into continuous surface models by solving the Poisson equation, and performs Laplace smoothing and hole repair on the models to construct accurate three-dimensional surface models.

[0007] Preferably, in the imaging examination and analysis module, the U-Net network structure includes a contraction path and an expansion path: The shrinkage path extracts deep features of the image through convolution and pooling operations; The expansion path restores the feature map resolution through upsampling and convolution operations, and finally outputs the segmented surgical site image. The network training uses the Dice loss function: , Where A is the true label and B is the predicted result, the accurate segmentation of the surgical site is achieved by minimizing this loss function.

[0008] Preferably, the imaging examination and analysis module transforms the surgical site coordinates in the image coordinate system to the world coordinate system through a homogeneous coordinate transformation matrix, and uses the ICP algorithm to calculate the optimal rigid body transformation (R,t) of the two sets of point cloud data, so as to minimize the sum of squared distances between corresponding points, thereby achieving spatial registration between the image data and the 3D model.

[0009] Preferably, in the ultrasound result calculation module, the Canny edge detection algorithm includes the following steps: Gaussian filtering is applied to the ultrasound images to reduce noise. Calculate the gradient magnitude and direction; Edges are determined using nonmaximum suppression and double threshold detection; The tissue depth d = vt / 2 is calculated by combining the ultrasonic wave propagation speed v and the echo time t.

[0010] Preferably, the data integration and navigation generation module employs a mutual information fusion algorithm, defining the mutual information of the two sets of data X and Y as follows: , Where H(X) is the entropy of X, and H(X|Y) is the conditional entropy of X given Y. The optimal registration parameters for multimodal data are determined by maximizing mutual information, and the data are unified to the 3D model coordinate system.

[0011] Preferably, the surgical path planning is based on FMM, which takes the surgical starting point as the source point and gradually propagates in the three-dimensional mesh by minimizing the cost function to generate the optimal path from the starting point to the target point.

[0012] Preferably, a multimodal lymphedema assessment and surgical navigation method based on image processing includes the following steps: S1. Obtain point cloud data and reconstruct a 3D model using structured light scanning; S2. Perform U-Net segmentation and ICP registration on CT / MRI images; S3. Perform edge detection and depth calculation on the ultrasound images; S4. Based on mutual information fusion multimodal data, the surgical path is generated through FMM and combined with AR technology for real-time navigation.

[0013] This invention provides a multimodal lymphedema assessment and surgical navigation system based on image processing. It has the following beneficial effects: 1. This invention, through comprehensive analysis of multimodal data, can more comprehensively, objectively, and accurately assess the condition of patients with lymphedema, including the precise location, size, depth of the edema, and its relationship with surrounding tissues, providing a more reliable basis for clinical diagnosis.

[0014] 2. The surgical navigation system of the present invention can guide the surgeon in real time, accurately avoid non-surgical sites, reduce surgical risks, improve the success rate of surgery, reduce the occurrence of postoperative complications, and benefit the patient's recovery.

[0015] 3. The various modules of this invention form an organic closed loop. 3D modeling provides a unified spatial benchmark for multimodal data. ICP registration between CT / MRI and 3D models ensures the consistency between images and entities. Canny edge detection and ultrasound depth calculation realize dynamic boundary calibration. This collaborative mechanism solves the problems of data misalignment and navigation lag in existing technologies, standardizes the assessment and surgical procedures, and effectively improves repeatability, providing technical support for the standardized promotion of lymphedema diagnosis and treatment. Attached Figure Description

[0016] Figure 1 This is a system block diagram of the present invention; Figure 2 This is a flowchart of the method of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Example 1: Please see the appendix Figure 1This invention provides a multimodal lymphedema assessment and surgical navigation system based on image processing. Taking a patient with left thigh lymphedema as an example, the system covers the complete process from data acquisition to surgical navigation. The patient is a 50-year-old female who experienced left lower extremity lymphatic drainage obstruction due to post-mastectomy radiotherapy and requires lymphovenous anastomosis surgery, including: I. Implementation Steps of the 3D Scanning and Modeling Module 1. Data Acquisition: The device uses a high-precision structured light 3D scanner. In this embodiment, ArtecEva is selected, with a resolution of 0.1mm and a scanning range of 500mm×500mm×500mm. Procedure: The patient stands with the left thigh exposed. The scanner projects a blue and white structured light pattern. Dual cameras with 13 megapixels simultaneously acquire images from different angles at a frame rate of 15fps. Triangulation principle: The camera intrinsic parameter matrix is , Calibration is performed using a checkerboard calibration board, where pixel coordinates f x =5000,f y =5000, u0=640, v0=480; The extrinsic parameter matrix [R|t] is calibrated by the scanner at the factory and describes the position of the camera coordinate system relative to the world coordinate system; The mapping relationship between the coordinates of a point (u, v) and the coordinates of a point in space (X, Y, Z) is as follows: , Solve the equations for 50,000 pixels to generate three-dimensional point cloud data of the left thigh surface.

[0019] 2. Surface Reconstruction and Optimization Poisson surface reconstruction: Using CloudCompare software, the reconstruction depth parameter was set to 8, and the Poisson equation was solved. ρ is the point cloud density function, which converts the point cloud into a triangular mesh surface; Post-processing: Laplacian smoothing: 10 iterations, smoothing coefficient 0.3, to reduce mesh noise; Hole Repair: Automatically detects holes with a diameter <5mm, and uses triangulation interpolation to fill them, generating a smooth 3D surface model.

[0020] II. Implementation Steps of the Imaging Examination and Analysis Module 1. CT image segmentation Data input: Import the patient's left thigh CT data, slice thickness 1mm, DICOM format, 200 slices in total, resolution 512×512; U-Net network configuration: Shrinkage path: 4 convolutional blocks, each block contains 2 layers of 3×3 convolution (ReLU) + 1 layer of 2×2 max pooling (stride 2), with the number of channels being 64, 128, 256 and 512 respectively; Expansion path: 4 upsampling blocks, each block contains 2×2 deconvolution (upsampling) + 2 layers of 3×3 convolution (ReLU), with the number of channels halved in succession; Output layer: 1×1 convolution, generating a binary segmentation map, with the surgical site as 1 and the background as 0; Training parameters: Loss function: Dice loss Where A represents the actual labels manually marked by the doctor, and B represents the predicted results; Optimizer: Adam, learning rate 1e -4 Batch size 8, training for 200 epochs, and a Dice coefficient of 0.95 on the validation set; 2. Spatial registration and homogeneous coordinate transformation: This involves transforming the x-coordinates of the surgical site vertex in the CT coordinate system into homogeneous coordinates. c y c , z c Convert to world coordinate system: , in The extrinsic parameter matrix for CT equipment calibration; ICP algorithm: Extract approximately 50,000 points from the CT segmentation point cloud and downsample the 3D model point cloud to 50,000 points; Iterative process: Find the closest pair of points with a Euclidean distance < 2mm, calculate the optimal rigid body transformation (R,t), and minimize it. n is the number of point pairs. The process is repeated 10 times until the root mean square error is less than 0.5 mm, at which point the registration is complete.

[0021] III. Implementation Steps of the Ultrasound Result Calculation Module 1. Edge detection in ultrasound images The device uses a Philips Epiq7 ultrasound machine with a linear array probe and a frequency of 10MHz to acquire ultrasound images of the surgical area of ​​the left thigh with a resolution of 640×480. The Canny edge detection process includes: Gaussian filtering: Uses 5×5 kernels to remove speckle noise; Gradient calculation: The Sobel operator is used to calculate the gradient in the x / y directions, and the magnitude is... ,direction ; Non-maximum suppression: Preserves local maxima along the gradient direction and suppresses non-edge pixels; Dual threshold detection: high threshold 0.8, low threshold 0.4, and edge connection through hysteresis tracking to determine the boundary of the surgical site; 2. Tissue Depth Calculation The propagation speed of ultrasound is v=1540m / s, the echo time t is obtained directly from the ultrasound instrument, the depth formula is: d=vt / 2, the depth is calculated for 100 edge points, and the average value is taken as the average depth of the surgical site.

[0022] IV. Implementation Steps for Data Integration and Navigation Generation Module 1. Multimodal data fusion Maximizing Mutual Information: Defining the mutual information between CT / MRI segmentation data (X) and the 3D model (Y) Among them, entropy Using the Powell optimization algorithm, we searched for rotation angles and translations to maximize mutual information values ​​and unified CT / MRI and ultrasound data into the 3D model coordinate system (error < 1 mm). 2. Surgical Path Planning Based on FMM Mesh generation: The 3D model is discretized into a 1mm×1mm×1mm voxel mesh, and the surgical starting point, i.e. the lymphatic vessel cut-off end, is marked with coordinates (100,80,50) and the target point of venous anastomosis, with coordinates (150,120,60). Cost function: Considering tissue stiffness and vascular risk, an optimal path from the starting point to the ending point is generated with a total length of 32mm, avoiding vascular areas; 3. AR Real-time Navigation The device uses a Microsoft HoloLens 2 with a refresh rate of 90Hz and a field of view of 52°. Display content: a semi-transparent red grid representing the three-dimensional boundary of the surgical site, surrounding blood vessels (femoral vein identified by CT), blue tubular structures and nerves (sciatic nerve identified by MRI), yellow lines, and green arrows guiding the real-time location coordinates, with real-time updates of the remaining distance to the target point.

[0023] Example 2: Please see the appendix Figure 2 This invention provides a multimodal lymphedema assessment and surgical navigation method based on image processing, comprising the following steps: S1.3D Modeling: Completes left thigh scan within 4 minutes, generating a high-precision 3D model; S2. Image processing: CT segmentation takes 2 minutes, ICP registration takes 3 minutes, and the error is <1mm; S3. Ultrasonic calculation: Edge detection and depth calculation are completed within 2 minutes to obtain the boundary and depth of the surgical site (12±1mm). S4. Data Fusion and Navigation: Mutual information registration takes 1 minute, FMM path generation takes 2 minutes, and AR devices display navigation information in real time to guide doctors to complete lymphovenous anastomosis surgery. The total operation time is reduced by 20% compared with traditional methods, and the anastomosis accuracy is improved to 0.3mm.

[0024] In summary, this embodiment achieves precise assessment and navigation for lymphedema surgery through technologies such as structured light scanning, U-Net segmentation, ICP registration, Canny detection, mutual information fusion, and FMM path planning. The parameter settings and algorithm implementation details of each module comply with the technical features of the claims. Furthermore, the feasibility and effectiveness of the system have been verified through specific clinical cases, providing a reliable technical solution for the precise treatment of lymphedema.

[0025] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A multimodal lymphedema assessment and surgical navigation system based on image processing, characterized in that, include: The 3D scanning and modeling module is used to acquire three-dimensional point cloud data of the patient's body surface through a structured light 3D scanning device and to construct a three-dimensional model based on the Poisson surface reconstruction algorithm. The imaging examination and analysis module is used to perform U-Net image segmentation on CT / MRI image data, optimize the segmentation results by combining the Dice loss function, and realize the spatial registration of image data and 3D model through homogeneous coordinate transformation and iterative ICP algorithm. The ultrasound results calculation module is used to perform Canny edge detection on ultrasound images to determine the boundaries of the surgical site, and to calculate tissue depth by combining ultrasound propagation time; The data integration and navigation generation module is used to maximize multimodal data fusion through mutual information, generate surgical paths based on FMM, and perform real-time navigation in conjunction with AR technology.

2. The image processing-based multimodal lymphedema assessment and surgical navigation system according to claim 1, characterized in that, In the 3D scanning modeling module, the calculation method for 3D point cloud data is based on the principle of triangulation. A mapping relationship between image point coordinates (u,v) and spatial point coordinates (X,Y,Z) is established through the camera intrinsic parameter matrix K and extrinsic parameter matrix [R|t]. , Where (u,v) are the pixel coordinates of the image point on the imaging plane. The three-dimensional point cloud is obtained by solving the above equation for a large number of image points.

3. The image processing-based multimodal lymphedema assessment and surgical navigation system according to claim 1, characterized in that, The Poisson surface reconstruction algorithm converts discrete point clouds into continuous surface models by solving the Poisson equation, and then performs Laplacian smoothing and hole repair on the models to construct accurate three-dimensional surface models.

4. The image processing-based multimodal lymphedema assessment and surgical navigation system according to claim 1, characterized in that, In the imaging examination and analysis module, the U-Net network structure includes a contraction path and an expansion path: The shrinkage path extracts deep features of the image through convolution and pooling operations; The expansion path restores the feature map resolution through upsampling and convolution operations, and finally outputs the segmented surgical site image. The network training uses the Dice loss function: ， Where A is the true label and B is the predicted result, the accurate segmentation of the surgical site is achieved by minimizing this loss function.

5. The image processing-based multimodal lymphedema assessment and surgical navigation system according to claim 1, characterized in that, The imaging examination and analysis module transforms the surgical site coordinates in the image coordinate system to the world coordinate system using a homogeneous coordinate transformation matrix, and uses the ICP algorithm to calculate the optimal rigid body transformation (R,t) of the two sets of point cloud data, so as to minimize the sum of squared distances between corresponding points, thereby achieving spatial registration between the image data and the 3D model.

6. The image processing-based multimodal lymphedema assessment and surgical navigation system according to claim 1, characterized in that, The Canny edge detection algorithm in the ultrasound result calculation module includes the following steps: Gaussian filtering is applied to the ultrasound images to reduce noise. Calculate the gradient magnitude and direction; Edges are determined using nonmaximum suppression and double threshold detection; The tissue depth d = vt / 2 is calculated by combining the ultrasonic wave propagation speed v and the echo time t.

7. The image processing-based multimodal lymphedema assessment and surgical navigation system according to claim 1, characterized in that, The data integration and navigation generation module employs a mutual information fusion algorithm, defining the mutual information between two sets of data X and Y as follows: ， Where H(X) is the entropy of X, and H(X|Y) is the conditional entropy of X given Y. The optimal registration parameters for multimodal data are determined by maximizing mutual information, and the data are unified to the 3D model coordinate system.

8. The image processing-based multimodal lymphedema assessment and surgical navigation system according to claim 1, characterized in that, The surgical path planning is based on FMM, which takes the surgical starting point as the source point and propagates step by step in the three-dimensional mesh by minimizing the cost function to generate the optimal path from the starting point to the target point.

9. The image processing-based multimodal lymphedema assessment and surgical navigation system according to claim 1, characterized in that, The AR technology displays in real time the following content: the three-dimensional coordinates of the surgical site, the spatial relationship between the boundary contour and surrounding blood vessels and nerve structures, providing visual guidance for surgical navigation.

10. A multimodal lymphedema assessment and surgical navigation method based on image processing, using the multimodal lymphedema assessment and surgical navigation system based on image processing as described in any one of claims 1-9, characterized in that, Includes the following steps: S1. Obtain point cloud data and reconstruct a 3D model using structured light scanning; S2. Perform U-Net segmentation and ICP registration on CT / MRI images; S3. Perform edge detection and depth calculation on the ultrasound images; S4. Based on mutual information fusion multimodal data, the surgical path is generated through FMM and combined with AR technology for real-time navigation.

Images