A surgical assistance system and method based on image processing

Abstract

The present invention relates to a surgical assistance system and method based on image processing. The system obtains preoperative medical images of patients and collects multispectral imaging medical images of the surface structure, blood flow distribution, and tissue elasticity of the surgical area in real time. After preprocessing, image feature extraction is performed to obtain preoperative image features and intraoperative multispectral image features, construct a biomechanical organ model, update the tissue deformation parameters of the biomechanical organ model, align the intraoperative multispectral image features with the biomechanical organ model in time and space, generate a three-dimensional enhanced model containing vascular topology, lesion boundaries, and elasticity distribution, superimpose the three-dimensional enhanced model on the real-time surgical field image, mark the relative position of the surgical instrument and the risk area, generate an intervention signal when the instrument approaches the critical safety distance threshold, and the doctor performs the surgical operation with assistance. The system obtains multidimensional intraoperative information through multispectral imaging to construct a high-precision dynamic biomechanical model, and combines it with augmented reality technology to achieve precise navigation and active intervention.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a surgery assistance system and method based on image processing. Background Art

[0002] In recent years, with the rapid development of medical imaging technology and computer-assisted surgery systems, the importance of real-time surgical navigation technology in the field of surgery has become increasingly prominent. Traditional surgical navigation systems mainly rely on preoperative static images (such as CT and MRI) to construct three-dimensional models, and combine them with intraoperative optical or electromagnetic tracking to achieve instrument positioning. However, such systems have significant limitations: first, intraoperative tissue deformation due to traction, breathing or hemodynamic changes leads to a mismatch between the preoperative model and the real-time anatomical structure; second, the dynamic changes of key structures such as blood vessels and lesions (such as blood flow distribution and tissue elasticity) are difficult to capture in real time through single-modality images; third, existing systems' interactive warnings for surgical instruments and risk areas are mostly based on fixed distance thresholds, lacking predictions of instrument motion trajectories and multimodal feedback mechanisms.

[0003] To address the above-mentioned issues, existing technologies have attempted to make improvements through intraoperative ultrasound and fluorescence imaging. For example, indocyanine green imaging based on near-infrared fluorescence has been used to visualize blood flow in the surgical area, but its spatial resolution is limited and it cannot simultaneously reflect the mechanical properties of tissues; although ultrasound elastography can provide information on tissue hardness, it is difficult to efficiently integrate it with optical images. In addition, non-rigid registration algorithms based on feature point matching (such as B-spline transformation) are often used to compensate for tissue deformation, but there is a problem of cumulative registration errors in the joint modeling of heterogeneous features of multispectral images (such as vascular topology and elastic parameters). In terms of risk warning, existing systems mostly use single visual or sound prompts, which can easily cause delayed response by doctors due to interference with the surgical field.

[0004] In recent years, the introduction of artificial intelligence (AI) technology has provided new insights into image feature extraction and dynamic modeling. Deep learning-based vascular segmentation algorithms have significantly improved the accuracy of identifying tubular structures in complex backgrounds. However, traditional methods have insufficiently explored the correlation between blood flow signals and elastic features in multispectral images. Furthermore, the dynamic coupling of biomechanical models with real-time imaging still faces the challenge of balancing computational efficiency and accuracy, particularly when rapidly updating deformation parameters during surgery, which can easily lead to lag effects.

[0005] Therefore, existing surgical assistance systems still face technical bottlenecks in real-time multimodal image fusion, dynamic deformation compensation, and intelligent risk warning. The urgent technical challenge is how to acquire multidimensional intraoperative information (surface structure, blood flow, and elasticity) through multispectral imaging, construct a high-precision dynamic biomechanical model, and integrate this with augmented reality technology to achieve precise navigation and proactive intervention. Summary of the Invention

[0006] The purpose of the present invention is to provide a surgical assistance system and method based on image processing, which is used to obtain multidimensional intraoperative information (surface structure, blood flow, elasticity) through multispectral imaging, construct a high-precision dynamic biomechanical model, and combine it with augmented reality technology to achieve precise navigation and active intervention.

[0007] In order to solve the above technical problems, the technical solutions adopted by the present invention are as follows:

[0008] In a first aspect, a surgical assistance method based on image processing is provided, comprising the following steps:

[0009] S1: Acquire preoperative medical images of the patient and collect multispectral imaging medical images related to the surface structure, blood flow distribution, and tissue elasticity of the patient's surgical area in real time during surgery;

[0010] S2: Preprocessing and image feature extraction of preoperative medical images and multispectral imaging medical images to obtain preoperative image features and intraoperative multispectral image features;

[0011] S3: constructing a biomechanical organ model based on preoperative image features, and updating tissue deformation parameters of the biomechanical organ model based on intraoperative multispectral image features;

[0012] S4: Spatiotemporally aligning intraoperative multispectral image features with the biomechanical organ model to generate a three-dimensional enhanced model that includes vascular topology, lesion boundaries, and elasticity distribution;

[0013] S5: superimposing the three-dimensional enhanced model onto the real-time surgical field image, marking the relative position of the surgical instrument and the risk area, and generating an intervention signal when the instrument approaches a critical safety distance threshold;

[0014] S6: Doctors perform surgery with the assistance of real-time surgical field images, marking information, and intervention signals.

[0015] Preferably, the specific process of collecting multispectral imaging medical images related to the surface structure, blood flow distribution and tissue elasticity of the patient's surgical area in real time in step S1 is as follows:

[0016] S11: The camera on the visible light endoscope is used to capture the surface structure image of the patient's surgical area;

[0017] S12: Generate blood flow and tumor images based on the near-infrared light of a specified wavelength emitted by the near-infrared fluorescence module to excite indocyanine green;

[0018] S13: Ultrasonic waves are emitted by an ultrasonic elastography device, and the tissue hardness is measured according to the shear wave velocity of the ultrasonic waves.

[0019] Preferably, the specific process of preprocessing the preoperative medical image and the multispectral imaging medical image and extracting image features in step S2 is as follows:

[0020] S21: Adjust the window width and window position of different preoperative medical images to be consistent, and perform isotropic resampling to eliminate the impact of scanning parameter differences on feature extraction. Perform spectral normalization on multispectral imaging medical images, and perform dark current correction and adaptive histogram equalization to enhance vascular intensity contrast.

[0021] S22: Feature extraction is performed on different preoperative medical images to obtain bone landmarks and vascular intersection features. Vascular features are extracted from multispectral images based on indocyanine green fluorescence intensity gradient tracking. Vascular network topology is constructed. Lesion features are also extracted from preoperative medical images and multispectral imaging medical images.

[0022] S23: Stable feature point pairs are selected based on the preset algorithm through the features of bone landmarks, vascular intersections, and tissue features, and a bidirectional matching constraint is established that satisfies both the preoperative medical image and the multispectral imaging medical image.

[0023] Preferably, the specific process of step S22 is as follows:

[0024] S221: Identifying designated key bone landmarks and vascular intersections in the preoperative medical image based on a preset deep neural network architecture, and integrating a multi-resolution attention mechanism into the deep neural network architecture to enhance the recognition accuracy of the designated key bone landmarks and vascular intersections;

[0025] S222: Verifying the positions of the designated key bone landmarks and blood vessel intersections based on the topological structure constraint module. If the topological structure constraint relationship is not satisfied, adjusting the designated key bone landmarks and blood vessel intersections.

[0026] S223: Calculate the second-order derivative of the multispectral image and construct the Hessian matrix. The specific formula is as follows:

[0027] ;

[0028] in, x, y are the two-dimensional space of pixels in the multispectral image. x Axis coordinate values and y Axis coordinate values, Ixx 、 Iyy is the spatial second-order partial derivative of the multispectral image, Ixy is the mixed partial derivative;

[0029] S224: Use Frangi filter to enhance tubular structures and calculate Hessian matrix eigenvalues λ 1. λ 2;

[0030] S225: Using a preset response function to measure tubularity and suppress background noise, the original gradient is corrected based on the Hessian matrix, and threshold segmentation is performed using the response value of the response function. The threshold is set to 30% of the maximum response to generate the initial vascular region. A distance transform is then performed to extract the centerline as the seed point, and the non-vascular region is determined based on the image histogram.

[0031] S226: Based on the vascular branching characteristics of the initial vascular region, a minimum spanning tree is constructed, isolated segments are removed, and morphological closing operations are applied to fill the internal vascular voids to construct the vascular network topology;

[0032] S227: Based on the grayscale distribution characteristics of preoperative medical images and multispectral imaging medical images, the region of interest is delineated, and the lesion probability map is generated through the created U-Net network model. The edge detection and regional statistical information are combined to drive the contour to converge to the lesion boundary to obtain the lesion characteristics.

[0033] Preferably, the specific process of constructing the biomechanical organ model based on the preoperative image features in step S3 is as follows:

[0034] S31: Generate high-resolution 3D mesh using hexahedral elements;

[0035] S32: Extract tissue morphological features and texture features from preoperative medical images, fuse them with bone landmarks and vascular intersection features, and generate a 256-dimensional joint feature vector;

[0036] S33: Generate a biomechanical organ model of the surgical area in the three-dimensional grid based on the 256-dimensional joint feature vector.

[0037] Preferably, in step S4, the specific process of performing spatiotemporal alignment of the intraoperative multispectral image features with the biomechanical organ model to generate a three-dimensional enhanced model including vascular topology, lesion boundary, and elasticity distribution is as follows:

[0038] S41: Build an image pyramid based on image resolution, iteratively optimize from the low-resolution layer, and gradually transfer transformation parameters to the high-resolution layer;

[0039] S42: Use the normalized mutual information function as the optimization objective function to optimize the image at the highest resolution layer. The specific formula is as follows:

[0040] f (A,B)=[H(A)+HI(B)] / H(A,B);

[0041] in, f(A, B) is the normalized mutual information function, A is a specified preoperative medical image, B is a multispectral imaging medical image, H(A) is the information entropy of image A, which quantifies the grayscale distribution complexity of image A, H(B) is the information entropy of image B, which quantifies the grayscale distribution complexity of image B, and H(A, B) is the joint entropy of image A and image B.

[0042] S43: A titanium alloy fiducial of a specified size is implanted in the patient's surgical area. The spatial distance is measured using the preoperative medical image and the multispectral image, and the target registration error is calculated. The specific calculation formula is as follows:

[0043] ;

[0044] Where N is the total number of reference mark points, i is the index of the currently calculated marker pair, i ∈[1, N], The first i The three-dimensional coordinates of the reference mark points, in mm, is the corresponding i The original three-dimensional coordinates of the fiducial markers are the coordinate system before registration. T(·) is the affine transformation matrix, which is the coordinate conversion function that maps the multispectral coordinate system to the CT coordinate system.

[0045] S44: Creating a dynamic deformation compensation model, wherein the dynamic deformation compensation model performs non-rigid registration on the intraoperative multispectral image and the preoperative medical image based on the registration error value by using a feature point matching algorithm.

[0046] Preferably, in step S5, the three-dimensional enhanced model is superimposed on the real-time surgical field image to mark the relative position of the surgical instrument and the risk area, and the specific process of generating an intervention signal when the instrument approaches the critical safety distance threshold is as follows:

[0047] S51: The spatial coordinates of the surgical instrument are obtained through the optical tracking system and dynamically aligned with the coordinate system of the 3D augmented model using the ICP algorithm;

[0048] S52: Using augmented reality projection technology, vascular topology and lesion boundaries are superimposed on the surgical field image in a semi-transparent form, achieving sub-millimeter spatial matching;

[0049] S53: Real-time identification of the instrument tip position using a convolutional neural network, marking contact risks with a pulsating halo in the superimposed image;

[0050] S54: Establish an instrument motion trajectory prediction model, activate a pre-alarm when it is detected that the velocity vector points to a high-risk area, set up a multimodal feedback system, and generate a triple warning signal based on tactile vibration of the instrument handle, visual flashes based on surgical field projection, and auditory prompts based on gradient buzzing.

[0051] In a second aspect, a surgical assistance system based on image processing is provided, which is used to implement any one of the surgical assistance methods based on image processing, including a data acquisition module, a preprocessing module, a feature extraction module, a biomechanical organ model generation module, a registration module, a three-dimensional enhanced model generation module, a three-dimensional enhanced model superposition module, and an early warning module;

[0052] The data acquisition module is used to acquire preoperative medical images of the patient and to collect multispectral imaging medical images related to the surface structure, blood flow distribution and tissue elasticity of the patient's surgical area during surgery in real time;

[0053] The preprocessing module is used to preprocess the preoperative medical image and the multispectral imaging medical image;

[0054] The feature extraction module is used to extract features from images and obtain preoperative image features and intraoperative multispectral image features;

[0055] The biomechanical organ model generation module is used to construct a biomechanical organ model based on preoperative image features and update tissue deformation parameters of the biomechanical organ model based on intraoperative multispectral image features;

[0056] The registration module is used to align the intraoperative multispectral image features with the biomechanical organ model in time and space;

[0057] The three-dimensional enhanced model generation module is used to generate a three-dimensional enhanced model including vascular topology, lesion boundary and elasticity distribution;

[0058] The three-dimensional enhanced model superposition module is used to superimpose the three-dimensional enhanced model onto the real-time surgical field image;

[0059] The early warning module is used to mark the relative position of the surgical instrument and the risk area, and generate an intervention signal when the instrument approaches a critical safety distance threshold.

[0060] The beneficial effects of the present invention include:

[0061] The present invention provides a surgical assistance system and method based on image processing, which obtains preoperative medical images of patients and collects multispectral imaging medical images of the surface structure, blood flow distribution and tissue elasticity of the surgical area in real time. After preprocessing, image feature extraction is performed to obtain preoperative image features and intraoperative multispectral image features, construct a biomechanical organ model, update the tissue deformation parameters of the biomechanical organ model, align the intraoperative multispectral image features with the biomechanical organ model in time and space, generate a three-dimensional enhanced model containing vascular topology, lesion boundaries and elasticity distribution, superimpose the three-dimensional enhanced model on the real-time surgical field image, mark the relative position of the surgical instrument and the risk area, generate an intervention signal when the instrument approaches the critical safety distance threshold, and perform surgical operations with the assistance of doctors. Multi-dimensional intraoperative information is obtained through multispectral imaging to construct a high-precision dynamic biomechanical model, and it is combined with augmented reality technology to achieve precise navigation and active intervention.

[0062] First, intraoperative multispectral imaging is used to achieve real-time visualization of blood vessels, tumors, and elastic parameters. Three-dimensional enhanced model overlay technology is used to control the spatial relationship error between the instrument and the risk structure within a preset range, and a graded early warning system is provided to reduce the rate of accidental damage to key structures.

[0063] Secondly, the feature extraction algorithm based on deep learning effectively shortens the time of blood vessel identification, reduces the dynamic update delay of the biomechanical model, supports respiratory motion compensation, and greatly reduces the time consumption of the multimodal data registration process.

[0064] Finally, the auxiliary system of the present invention can be applied to complex surgical operations, shorten the operation time, provide an intelligent perception-decision-feedback closed loop, and achieve a technological leap from static navigation to dynamic interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 The figure is a flowchart of a surgical assistance method based on image processing according to the present invention.

[0066] Figure 2 Schematic diagram of the architecture of a surgical assistance system based on image processing according to the present invention. DETAILED DESCRIPTION

[0067] The following is combined with Figure 1~Figure 2 The present invention is described in further detail:

[0068] Example 1

[0069] In this embodiment, neurosurgery glioma resection is used as an application scenario for illustration. Figure 1 As shown, a surgical assistance method based on image processing includes the following steps:

[0070] S1: Acquire preoperative medical images of the patient and collect real-time multispectral medical images during surgery, including the surface structure, blood flow distribution, and tissue elasticity of the surgical area. Preoperative medical images are retrieved from a pre-set database. Real-time multispectral imaging is performed using an endoscope equipped with a near-infrared module to capture the surface structure of the surgical area during surgery.

[0071] S2: Preprocess the preoperative medical images and multispectral imaging images and extract image features to obtain preoperative image features and intraoperative multispectral image features. Window width and window position are adjusted for the preoperative and multispectral imaging images, with the window width uniformly adjusted to 1600 HU and the window position to 40 HU. Spectral calibration is performed on the multispectral images to eliminate interference from hemoglobin absorption bands.

[0072] S3: constructing a biomechanical organ model based on preoperative image features, and updating tissue deformation parameters of the biomechanical organ model based on intraoperative multispectral image features.

[0073] S4: Intraoperative multispectral image features are spatiotemporally aligned with the biomechanical organ model to generate a three-dimensional enhanced model that includes vascular topology, lesion boundaries, and elasticity distribution. Four titanium alloy fiducials of specified diameters are marked on the skull. Non-rigid registration is performed using an improved Demons algorithm, and the target registration error is measured. Dual-channel perspective compensation technology is used to improve spatial registration accuracy when projecting vascular enhanced images.

[0074] S5: superimposing the three-dimensional enhanced model onto the real-time surgical field image, marking the relative position of the surgical instrument and the risk area, and generating an intervention signal when the instrument approaches a critical safety distance threshold.

[0075] S6: Doctors perform surgery with the assistance of real-time surgical field images, marking information, and intervention signals.

[0076] Example 2

[0077] The specific process of real-time acquisition of multispectral imaging medical images related to the surface structure, blood flow distribution, and tissue elasticity of the patient's surgical area during surgery in step S1 is as follows:

[0078] S11: The surface structure image of the patient's surgical area is collected based on the camera carried by the visible light endoscope. The visible light endoscope uses an electronic endoscope with an integrated CMOS / CCD image sensor, supports 4K resolution and a wide dynamic range greater than 85dB. It is equipped with a ring-shaped LED light source with a color temperature of 5500K±200K. The illumination of the surgical area is maintained at an adjustable range of 1000-15000 lux through an automatic light intensity adjustment module. A 12mm diameter hard mirror or flexible fiber optic mirror is used, and the field of view is set to 70°-120°. The endoscope integrates a MEMS angular velocity sensor to track changes in azimuth and pitch angles in real time. An AprilTag visual marker array is arranged at the far end of the mirror body with a spacing of 5mm, which cooperates with the surgical field tracking camera to achieve submillimeter spatial positioning.

[0079] S12: Indocyanine green (ICG) is excited by near-infrared light of a specified wavelength emitted by the near-infrared fluorescence module to generate blood flow and tumor images. After intravenous injection of 0.3 mg / kg ICG into the surgical area, the system switches to fluorescence mode and acquires blood flow distribution images at a frame rate of 15 fps. Simultaneously, the ultrasound elastography system is activated, with the shear wave frequency set to 50 Hz, to acquire an elastogram with a spatial resolution of 1 mm.

[0080] S13: Ultrasonic waves are emitted based on the ultrasonic elastography device, and the tissue hardness is measured according to the shear wave velocity of the ultrasonic wave. The ultrasonic probe emits high-frequency sound pulses with a central frequency of 1-5 MHz and a pulse duration of 50-200 μs, forming an adjustable acoustic radiation force pulse focused on the target tissue. The target tissue depth is 20-150 mm. Multi-point focusing is achieved through phased array technology to generate transverse shear waves inside the tissue. The excitation energy is controlled within a safe range of a mechanical index of less than 1.0 to avoid tissue damage. The shear wave propagates in a direction perpendicular to the ultrasonic beam with a velocity range of 1-10 m / s. The propagation depth is affected by the tissue attenuation coefficient, and the effective detection depth is usually less than 70 mm.

[0081] A 128-256-element array probe is used to receive echo signals, capturing tissue particle displacement at a sampling rate greater than 10,000 frames per second. The displacement changes between adjacent frames are calculated using a radio frequency signal cross-correlation algorithm with a resolution of microns, generating a time-displacement curve that captures the time difference between shear waves arriving at different detection points.

[0082] pass Calculate the wave velocity, where Δ d is the distance between adjacent detection points, usually set to 1-3mm, Δ t The multipath weighted average method is used to improve the measurement accuracy, and the standard deviation is controlled within ±0.2m / s. Based on the physical relationship between shear wave velocity c and elastic modulus E: E=3 ρc 2 , tissue density ρ The default setting is 1000kg / m³, which is an approximate value for living soft tissue.

[0083] Example 3

[0084] Based on Example 1 or Example 2, the specific process of preprocessing the preoperative medical image and the multispectral imaging medical image and extracting image features in step S2 is as follows:

[0085] S21: Adjust the window width and window position of different preoperative medical images to be consistent, and perform isotropic resampling to eliminate the impact of scanning parameter differences on feature extraction. Perform spectral normalization on multispectral imaging medical images, and perform dark current correction and adaptive histogram equalization to enhance vascular intensity contrast.

[0086] S22: Feature extraction is performed on different preoperative medical images to obtain bone landmarks and vascular intersection features. Vascular features are extracted from multispectral images based on indocyanine green fluorescence intensity gradient tracking. Vascular network topology is constructed. Lesion features are also extracted from preoperative medical images and multispectral imaging medical images.

[0087] S23: Stable feature point pairs are selected based on the preset algorithm through the features of bone landmarks, vascular intersections, and tissue features, and a bidirectional matching constraint is established that satisfies both the preoperative medical image and the multispectral imaging medical image.

[0088] The specific process of step S22 is as follows:

[0089] S221: Identify designated key bone landmarks and vascular intersections in preoperative medical images based on a preset deep neural network architecture, and integrate a multi-resolution attention mechanism into the deep neural network architecture to enhance the recognition accuracy of designated key bone landmarks and vascular intersections. By integrating a spatial-channel dual attention module through an improved ResNet-50 network, 12 anatomical landmarks such as the edge of the corpus callosum and the intersection of the basal vein are extracted. When calculating the Hessian matrix for the fluorescence image, set σ = 1.2 Gaussian kernel for scale normalization, Frangi filter parameters β =0.5, γ =15, and the segmentation threshold is 30% of the maximum response.

[0090] S222: The positions of designated key bone landmarks and vascular intersections are verified based on the topological constraint module. If the topological constraint relationship is not met, the designated key bone landmarks and vascular intersections are adjusted. Four titanium alloy fiducials with a diameter of 2 mm are marked on the skull, with a spacing greater than 30 mm. Non-rigid registration is performed based on the improved Demons algorithm, and the target registration error TRE is measured to be 0.87±0.12 mm. When projecting vascular enhanced images, dual-channel perspective compensation technology is used, and the spatial registration accuracy reaches 0.3 mm.

[0091] S223: Calculate the second-order derivative of the multispectral image and construct the Hessian matrix. The specific formula is as follows:

[0092] ;

[0093] in, x, y are the two-dimensional space of pixels in the multispectral image. x Axis coordinate values and y Axis coordinate values, Ixx 、 Iyy is the spatial second-order partial derivative of the multispectral image, Ixy is the mixed partial derivative;

[0094] S224: Use Frangi filter to enhance tubular structures and calculate Hessian matrix eigenvalues λ 1. λ 2;

[0095] S225: Using a preset response function to measure tubularity and suppress background noise, the original gradient is corrected based on the Hessian matrix, and threshold segmentation is performed using the response value of the response function. The threshold is set to 30% of the maximum response to generate the initial vascular region. A distance transform is then performed to extract the centerline as the seed point, and the non-vascular region is determined based on the image histogram.

[0096] S226: Based on the vascular branching characteristics of the initial vascular region, a minimum spanning tree is constructed, isolated segments are removed, and morphological closing operations are applied to fill the internal vascular voids to construct the vascular network topology;

[0097] S227: Based on the grayscale distribution characteristics of preoperative medical images and multispectral imaging medical images, the region of interest is delineated, and the lesion probability map is generated through the created U-Net network model. The edge detection and regional statistical information are combined to drive the contour to converge to the lesion boundary to obtain the lesion characteristics.

[0098] The specific process of constructing the biomechanical organ model based on the preoperative image features in step S3 is as follows:

[0099] S31: Generates a high-resolution 3D mesh using hexahedral elements. The Advancing Front method is used to generate a surface quadrilateral mesh. Voxel filling is performed based on an octree structure. Internal nodes are mapped to hexahedral elements using Transfinite Interpolation. The Jacobian matrix determinant is greater than 0.7, and the element aspect ratio is less than 3:1.

[0100] S32: Extract tissue morphology and texture features from preoperative medical images and fuse them with features of skeletal landmarks and vascular intersections to generate a 256-dimensional joint feature vector. Organ volume, surface area, and curvature distribution are calculated. Contrast, energy, and homogeneity are extracted from the GLCM matrix using a 7×7 window size. SCSE-ResNet is then used to locate 12 key points.

[0101] S33: Generate a biomechanical organ model of the surgical area within the three-dimensional mesh based on the 256-dimensional joint eigenvector. Young's modulus and Poisson's ratio are mapped based on the eigenvectors. Anisotropic parameters are set for the vascular region, with an axial / radial stiffness ratio of 3:1. A constitutive equation is constructed: Ku = F, where K is the stiffness matrix integrated hyperelastic model. Boundary conditions are set to fixed skeletal landmarks, and a simulated surgical instrument pressure of 5 N is applied. GPU acceleration, namely CUDA kernel functions, is used to achieve deformation visualization at 30 frames per second.

[0102] In another embodiment of this example, finite element analysis software was established, and a brain tissue model consisting of 256,000 hexahedral elements was constructed within the software. Preoperative DTI fiber bundle imaging data was integrated, and regions with FA values > 0.25 were set as anisotropic materials. The gray matter elastic modulus was set to 3 kPa, the Poisson's ratio to 0.45, and the white matter elastic modulus to 5 kPa. During the operation, when the ultrasound elasticity data was used to update the model, a real-time parameter identification algorithm with a preset filtering algorithm was used to control the convergence time to within 50 ms.

[0103] Example 4

[0104] On the basis of Example 1, Example 2, or Example 3, the specific process of performing spatiotemporal alignment of the intraoperative multispectral image features with the biomechanical organ model in step S4 to generate a three-dimensional enhanced model including vascular topology, lesion boundary, and elasticity distribution is as follows:

[0105] S41: Build an image pyramid based on image resolution, iteratively optimize from the low-resolution layer, and gradually transfer transformation parameters to the high-resolution layer;

[0106] S42: Use the normalized mutual information function as the optimization objective function to optimize the image at the highest resolution layer. The specific formula is as follows:

[0107] f (A,B)=[H(A)+H(B)] / H(A,B);

[0108] in, f (A, B) is the normalized mutual information function, A is a specified preoperative medical image, B is a multispectral imaging medical image, H(A) is the information entropy of image A, which quantifies the grayscale distribution complexity of image A, H(B) is the information entropy of image B, which quantifies the grayscale distribution complexity of image B, and H(A, B) is the joint entropy of image A and image B.

[0109] S43: A titanium alloy fiducial of a specified size is implanted in the patient's surgical area. The spatial distance is measured using the preoperative medical image and the multispectral image, and the target registration error is calculated. The specific calculation formula is as follows:

[0110] ;

[0111] Where N is the total number of reference mark points, i is the index of the currently calculated marker pair, i ∈[1, N], The first i The three-dimensional coordinates of the reference mark points, in mm, is the corresponding i The original three-dimensional coordinates of the fiducial markers are the coordinate system before registration. T(·) is the affine transformation matrix, which is the coordinate conversion function that maps the multispectral coordinate system to the CT coordinate system.

[0112] S44: Creating a dynamic deformation compensation model, wherein the dynamic deformation compensation model performs non-rigid registration on the intraoperative multispectral image and the preoperative medical image based on the registration error value by using a feature point matching algorithm.

[0113] In step S5, the three-dimensional enhanced model is superimposed on the real-time surgical field image to mark the relative position of the surgical instrument and the risk area. The specific process of generating an intervention signal when the instrument approaches the critical safety distance threshold is as follows:

[0114] S51: The spatial coordinates of the surgical instrument are obtained through the optical tracking system and dynamically aligned with the coordinate system of the 3D augmented model using the ICP algorithm;

[0115] S52: Using augmented reality projection technology, vascular topology and lesion boundaries are superimposed on the surgical field image in a semi-transparent form, achieving sub-millimeter spatial matching;

[0116] S53: Real-time identification of the instrument tip position using a convolutional neural network, marking contact risks with a pulsating halo in the superimposed image;

[0117] S54: A model predicting the instrument's motion trajectory is established. A pre-alarm is activated when a velocity vector is detected pointing toward a high-risk area. A multimodal feedback system is implemented, generating a triple warning signal based on tactile vibration of the instrument handle, visual flashes based on the surgical field projection, and auditory cues based on a gradient beep. When the tip of the bipolar electrocoagulation forceps is within 2mm of the anterior choroidal artery, the system triggers a three-level warning: ① The handle vibrates at 120Hz three times per second; ② The vessel outline in the surgical field flashes red (at a frequency of 5Hz); and ③ A 2000Hz gradient-increasing beep is emitted. Experiments using isolated pig brains have demonstrated a warning response delay of <80ms and a false alarm rate of <1.2%.

[0118] A surgical assistance system based on image processing, used to implement any one of the surgical assistance methods based on image processing, see Figure 2 As shown, it includes a data acquisition module, a preprocessing module, a feature extraction module, a biomechanical organ model generation module, a registration module, a three-dimensional enhanced model generation module, a three-dimensional enhanced model superposition module, and an early warning module. The data acquisition module is connected to the preprocessing module and the feature extraction module, the feature extraction module is connected to the biomechanical organ model generation module, the biomechanical organ model generation module is connected to the registration module, the registration module is connected to the three-dimensional enhanced model generation module, the three-dimensional enhanced model generation module is connected to the three-dimensional enhanced model superposition module, and the early warning module is connected to the surgical instrument.

[0119] The data acquisition module is used to acquire preoperative medical images of the patient and to collect multispectral imaging medical images related to the surface structure, blood flow distribution, and tissue elasticity of the surgical area during surgery in real time. The preprocessing module is used to preprocess preoperative medical images and multispectral imaging medical images. The feature extraction module is used to extract features from the images and acquire preoperative image features and intraoperative multispectral image features. The biomechanical organ model generation module is used to construct a biomechanical organ model based on preoperative image features and to update the tissue deformation parameters of the biomechanical organ model based on intraoperative multispectral image features. The registration module is used to spatially and temporally align the intraoperative multispectral image features with the biomechanical organ model. The three-dimensional enhanced model generation module is used to generate a three-dimensional enhanced model that includes vascular topology, lesion boundaries, and elasticity distribution. The three-dimensional enhanced model overlay module is used to overlay the three-dimensional enhanced model onto the real-time surgical field image. The early warning module is used to mark the relative position of the surgical instrument and the risk area and to generate an intervention signal when the instrument approaches a critical safety distance threshold.

[0120] In summary, the present invention provides a surgical assistance system and method based on image processing, which obtains preoperative medical images of patients and collects multispectral imaging medical images of the surface structure, blood flow distribution and tissue elasticity of the surgical area in real time. After preprocessing, image feature extraction is performed to obtain preoperative image features and intraoperative multispectral image features, construct a biomechanical organ model, update the tissue deformation parameters of the biomechanical organ model, align the intraoperative multispectral image features with the biomechanical organ model in time and space, generate a three-dimensional enhanced model including vascular topology, lesion boundary and elasticity distribution, superimpose the three-dimensional enhanced model on the real-time surgical field image, mark the relative position of the surgical instrument and the risk area, generate an intervention signal when the instrument approaches the critical safety distance threshold, and perform surgical operations with the assistance of doctors. Multi-dimensional intraoperative information is obtained through multispectral imaging to construct a high-precision dynamic biomechanical model, and it is combined with augmented reality technology to achieve precise navigation and active intervention.

[0121] Intraoperative multispectral imaging enables real-time visualization of vascular, tumor, and elastic parameters. Three-dimensional enhanced model overlay technology controls the spatial relationship error between the instrument and risk structures within a preset range. A graded early warning system (tactile, visual, and auditory) is provided to reduce the risk of accidental damage to critical structures. A deep learning-based feature extraction algorithm effectively shortens vessel identification time, reduces the latency of dynamic updates of the biomechanical model, supports respiratory motion compensation, and significantly reduces the time-consuming multimodal data registration process. This system is applicable to complex surgical procedures, shortens operative time, and provides an intelligent perception-decision-feedback closed loop, achieving a technological leap from static navigation to dynamic interaction.

Claims

1. A surgical assistance system based on image processing, characterized in that: It includes data acquisition module, preprocessing module, feature extraction module, biomechanical organ model generation module, registration module, 3D enhanced model generation module, 3D enhanced model superposition module, and early warning module; The data acquisition module is used to acquire preoperative medical images of the patient and to collect multispectral imaging medical images related to the surface structure, blood flow distribution and tissue elasticity of the patient's surgical area during surgery in real time; The preprocessing module is used to preprocess the preoperative medical image and the multispectral imaging medical image; The feature extraction module is used to extract features from images and obtain preoperative image features and intraoperative multispectral image features; The biomechanical organ model generation module is used to construct a biomechanical organ model based on preoperative image features and update tissue deformation parameters of the biomechanical organ model based on intraoperative multispectral image features; The registration module is used to align the intraoperative multispectral image features with the biomechanical organ model in time and space; The three-dimensional enhanced model generation module is used to generate a three-dimensional enhanced model including vascular topology, lesion boundary and elasticity distribution; The three-dimensional enhanced model superposition module is used to superimpose the three-dimensional enhanced model on the real-time surgical field image; The early warning module is used to mark the relative position of the surgical instrument and the risk area, and generate an intervention signal when the instrument approaches a critical safety distance threshold.

2. The image processing-based surgical assistance system according to claim 1, characterized in that: The specific process of the data acquisition module collecting multispectral imaging medical images related to the surface structure, blood flow distribution and tissue elasticity of the patient's surgical area in real time is as follows: The camera mounted on the visible light endoscope captures images of the surface structure of the patient's surgical area; the near-infrared light of a specified wavelength emitted by the near-infrared fluorescence module excites indocyanine green to generate blood flow and tumor images; and the ultrasonic elastography device emits ultrasonic waves, and measures tissue hardness based on the shear wave velocity of the ultrasonic waves.

3. The surgical assistance system based on image processing according to claim 1, characterized in that: The specific process of the preprocessing module preprocessing the preoperative medical image and the multispectral imaging medical image and extracting image features is as follows: The window widths and window positions of different preoperative medical images were adjusted to be consistent, and isotropic resampling was performed to eliminate the impact of scanning parameter differences on feature extraction. Multispectral imaging medical images were spectrally normalized, baseline noise was subtracted using a dark current reference frame, and dark current correction and adaptive histogram equalization were performed to enhance vascular intensity contrast, increasing vascular contrast to several times that of the original image. Feature extraction was performed on different preoperative medical images to obtain bone landmarks and vascular intersection features. Vascular features were extracted from multispectral images based on indocyanine green fluorescence intensity gradient tracking, vascular network topology was constructed, and lesion features were extracted from preoperative medical images and multispectral imaging medical images. Stable feature point pairs are screened according to the preset algorithm through the features of bone landmarks, vascular intersections and tissue features, and bidirectional matching constraints that simultaneously meet the needs of preoperative medical images and multispectral imaging medical images are established.

4. The image processing-based surgical assistance system according to claim 3, characterized in that: The specific process of constructing the vascular network topology and extracting lesion features from preoperative medical images and multispectral imaging medical images is as follows: Identify designated key bone landmarks and vascular intersections in preoperative medical images based on a preset deep neural network architecture, and integrate a multi-resolution attention mechanism into the deep neural network architecture to enhance the recognition accuracy of designated key bone landmarks and vascular intersections; The topology constraint module verifies the positions of the designated key bone landmarks and vascular intersections. If the topology constraint relationship is not met, the designated key bone landmarks and vascular intersections are adjusted. Calculate the second-order derivative of the multispectral image and construct the Hessian matrix; Frangi filter is used to enhance the tubular structure and calculate the eigenvalues of the Hessian matrix; A preset response function is used to measure tubularity and suppress background noise. The original gradient is corrected based on the Hessian matrix, and threshold segmentation is performed using the response value of the response function to generate the initial vascular region. A distance transform is then performed to extract the centerline as the seed point, and the non-vascular region is determined based on the image histogram. Based on the vascular branching characteristics of the initial vascular region, a minimum spanning tree is constructed, isolated segments are removed, and morphological closing operations are applied to fill the internal cavities of the blood vessels to construct the vascular network topology. The region of interest is delineated based on the grayscale distribution characteristics of preoperative medical images and multispectral imaging medical images. The lesion probability map is generated through the created U-Net network model. The edge detection and regional statistical information are combined to drive the contour to converge to the lesion boundary and obtain the lesion characteristics.

5. The image processing-based surgical assistance system according to claim 1, characterized in that: The specific process of the biomechanical organ model generation module constructing the biomechanical organ model based on preoperative image features is as follows: Use hexahedral elements to generate high-resolution three-dimensional meshes; Extract tissue morphological features and texture features from preoperative medical images, fuse them with bone landmarks and vascular intersection features, and generate a 256-dimensional joint feature vector; A biomechanical organ model of the surgical area is generated in the three-dimensional grid based on the 256-dimensional joint feature vector.

6. The image processing-based surgical assistance system according to claim 1, characterized in that: The registration module performs spatiotemporal alignment of intraoperative multispectral image features with the biomechanical organ model to generate a three-dimensional enhanced model that includes vascular topology, lesion boundaries, and elasticity distribution. The specific process is as follows: Build an image pyramid based on image resolution, iteratively optimize from the low-resolution layer, and gradually transfer transformation parameters to the high-resolution layer; Image optimization is performed using the normalized mutual information function as the optimization objective function at the highest resolution layer; A titanium alloy fiducial of specified size is implanted in the patient's surgical area, the spatial distance is measured using preoperative medical images and multispectral images, and the target registration error value is calculated; A dynamic deformation compensation model is created, and the dynamic deformation compensation model performs non-rigid registration of the intraoperative multispectral image and the preoperative medical image based on the registration error value through a feature point matching algorithm.

7. The image processing-based surgical assistance system according to claim 1, characterized in that: The 3D enhanced model overlay module overlays the 3D enhanced model onto the real-time surgical field image, and the early warning module marks the relative position of the surgical instrument and the risk area, and generates an intervention signal when the instrument approaches the critical safety distance threshold. The specific process is as follows: The spatial coordinates of the surgical instrument are obtained through an optical tracking system and dynamically aligned with the coordinate system of the 3D augmented model using the ICP algorithm; Using augmented reality projection technology, vascular topology and lesion boundaries are superimposed on the surgical field image in a semi-transparent form, achieving sub-millimeter spatial matching; The device tip position is identified in real time using a convolutional neural network, marking contact risks with a pulsing halo in the superimposed image. A model for predicting the motion trajectory of the instrument is established, and a pre-alarm is activated when the velocity vector is detected pointing to a high-risk area. A multimodal feedback system is set up to generate a triple warning signal based on tactile vibration of the instrument handle, visual flashes based on surgical field projection, and auditory prompts based on gradient buzzing.

Images