Three-dimensional image registration method and system based on binocular optical and photoacoustic endoscopy imaging

Abstract

The application provides a three-dimensional image registration method and system based on binocular optical and photoacoustic endoscopy imaging, which is suitable for blood vessel imaging and lesion identification in abdominal cavity minimally invasive surgery; the method collects left and right view image pairs through binocular endoscopy, obtains a three-dimensional point cloud of a detection target surface through stereo matching and triangulation, and generates a three-dimensional optical surface model through post-reconstruction; meanwhile, a two-dimensional cross-sectional image sequence is collected through photoacoustic endoscopy, a three-dimensional photoacoustic volume model of the detection target is obtained through volume data reconstruction, a registration loss function containing skeleton consistency, surface-volume distance constraint and projection consistency is constructed, and the spatial transformation relationship between the optical surface model and the photoacoustic volume model is iteratively optimized in a three-dimensional space, so that high-precision cross-modal registration of the two kinds of data is realized.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of medical image processing and endoscopic imaging technology, and particularly relates to a three-dimensional image registration method based on binocular optical and photoacoustic endoscopic imaging, which is suitable for multi-modal image registration and tissue structure modeling in minimally invasive surgery scenarios. BACKGROUND

[0002] With the popularization of laparoscopic and other minimally invasive surgeries, the demand for intraoperative lesion positioning, blood vessel identification and tissue functional imaging is increasing. Photoacoustic imaging (PAI) as a new imaging method combining the advantages of optics and ultrasound has high spatial resolution and good tissue contrast, especially in intraoperative blood vessel visualization, blood oxygen detection, lesion boundary identification, etc. At the same time, binocular endoscopes can provide three-dimensional geometric information of the tissue surface, providing intuitive anatomical information for intraoperative navigation. However, existing intraoperative photoacoustic imaging is mostly in the form of handheld free scanning, and the probe will continuously change its pose during operation, resulting in significant perspective deviation and geometric distortion between adjacent frames. Especially in soft tissue surface or irregular organ area, it is easy to cause blood vessel structure dislocation, fracture and topological error reconstruction, which is more prominent when multi-frame photoacoustic image stitching or optical image registration is performed, affecting the structural consistency and anatomical explainability of the fusion result.

[0003] Existing image registration methods usually rely on external positioning devices (such as optical trackers, electromagnetic navigators, etc.), but such devices have limited application in narrow surgical cavities such as laparoscopes, and are easily affected by instrument and tissue shielding. On the other hand, existing methods mostly use rigid or affine transformation to complete registration, which cannot effectively deal with the nonlinear deformation and perspective effect introduced by probe motion, making it difficult to ensure the true alignment of fine anatomical structures. In addition, there are significant differences in signal representation between optical and photoacoustic images, and traditional similarity measures based on grayscale or intensity cannot accurately capture the correspondence between the two. SUMMARY

[0004] The present application aims to overcome the shortcomings of the prior art and proposes a three-dimensional image registration method based on binocular optical and photoacoustic endoscopic imaging, thereby achieving high-precision cross-modal registration of the two types of data.

[0005] The three-dimensional image registration method based on binocular optical and photoacoustic endoscopic imaging comprises the following steps:

[0006] Step S1, acquiring a pair of optical images with left and right perspectives using a binocular endoscope, and reconstructing a three-dimensional optical model of the detection target through stereo matching and triangulation;

[0007] Step S2: Acquire a sequence of two-dimensional cross-sectional images using photoacoustic endoscopy, and reconstruct a three-dimensional photoacoustic body model using volume data;

[0008] Step S3: Extract features such as the vascular skeleton, surface, and bifurcation points from the optical 3D model and photoacoustic 3D volume data, respectively.

[0009] Step S4: Establish a registration transformation model between the 3D optical model and the 3D photoacoustic model. The registration transformation model includes rigid transformation and non-rigid deformation fields acting sequentially. The rigid transformation parameters and the control parameters of the non-rigid deformation fields are used as variables to be optimized. The 3D photoacoustic model is mapped to the coordinate system of the 3D optical model through the registration transformation to obtain the mapping result. Based on the mapping result, the skeleton consistency loss function, the surface-volume distance loss function, and the projection consistency loss function are calculated and weighted to construct the registration loss function. The registration loss function is used as the objective function to form an end-to-end 3D registration optimization model.

[0010] Step S5: Minimize the registration loss function, iteratively update the rigid transformation parameters and the control parameters of the non-rigid deformation field until the preset convergence condition is met, obtain the final rigid transformation and the optimal non-rigid deformation field, and transform the three-dimensional photoacoustic model to obtain the three-dimensional registration result aligned with the three-dimensional optical model space in the same coordinate system.

[0011] Furthermore, the skeleton consistency loss function Defined as:

[0012]

[0013] in, For optical skeleton point set; For the photoacoustic vascular skeleton point set; Weights for the bifurcation points; These are the coordinates of the skeleton points in the optical 3D model. Φ(x) represents the coordinates of the skeleton points in the photoacoustic 3D volume data; Φ(x) represents the spatial transformation function that maps point x in the optical coordinate system to the photoacoustic 3D coordinate system; T(Φ(x)) represents the coordinates of the corresponding point in the photoacoustic volume data obtained after 3D interpolation of the mapped position Φ(x). The skeleton consistency loss is used to perform nearest neighbor matching between the skeleton points of the optical 3D model and the skeleton points of the photoacoustic 3D volume data after mapping, and measures the difference between the two based on Euclidean distance.

[0014] Furthermore, the surface-volume distance loss function Defined as:

[0015]

[0016] in, For optical surface point sets; denotes the spatial coordinates of the optical surface point; is a robust function; is a signed distance field of the photoacoustic volume data.

[0017] Further, the projection consistency loss function comprises two parts: a soft-IoU loss function :

[0018]

[0019] for measuring the area overlap degree between the virtual photoacoustic mask and the optical blood vessel mask ;

[0020] wherein, denotes the pixel position in the two-dimensional image plane; denotes the value of the virtual photoacoustic mask generated by projection according to the three-dimensional photoacoustic volume data at the pixel ; denotes the blood vessel mask value of the optical image at the pixel ; is a positive constant for preventing the denominator from being zero, preferably taking the value of 10 -6 .

[0021] edge direction consistency loss function :

[0022]

[0023] for measuring the edge direction matching degree between the virtual photoacoustic image and the optical image ;

[0024] wherein, denotes the pixel set participating in the edge direction calculation; denotes the two-dimensional image pixel position in the pixel set , used for indexing the corresponding pixel points in the virtual photoacoustic image and the optical image ; denotes the gray value or intensity value of the virtual photoacoustic image rendered or projected from the three-dimensional photoacoustic volume data at the pixel ; denotes the gray value or intensity value of the optical image at the pixel ; , are the gradient vectors thereof, respectively; , are the norms of the above gradient vectors, respectively; To prevent the numerical instability of positive constant caused by zero denominator, the value is preferably 10 -6 .

[0025] Further, the optimization of the registration parameters adopts an end-to-end iterative updating mechanism based on a registration loss function, and specifically includes the following steps:

[0026] The total loss function is constructed:

[0027] ;

[0028] wherein, represents the Lie algebra parameter of the rigid transformation; represents the non-rigid deformation field; s is a scale parameter; represents the non-rigid deformation field after introducing the scale parameter s; represents the defined skeleton consistency loss function; represents the defined surface-volume distance loss function; represents the defined projection consistency loss containing soft-IoU loss and edge direction consistency loss; represents the regularization term applied to the deformation field for limiting the smoothness or complexity of the deformation field; , , , respectively represent the weighting coefficients of the skeleton consistency loss , the surface-volume distance loss , the projection consistency loss , and the regularization loss in the total loss function.

[0029] The gradient of the total loss function with respect to the optimization variable is calculated, and the update direction is obtained through back propagation; the iterative method based on gradient descent or Adam optimizer is used to update the parameters until the loss function converges.

[0030] Further, in the registration process, the vascular skeleton in the registration region is topologically corrected through structure consistency constraint, specifically including:

[0031] Detecting the bifurcation points and end points in the optical and photoacoustic vascular skeletons; establishing the spatial correspondence of the cross-modal key points in a unified coordinate system;

[0032] Correcting the locally misaligned or discontinuous vascular segments based on the topological consistency constraint to maintain the continuity and topological fidelity of the registration result at the anatomical structure level.

[0033] A three-dimensional image registration system based on binocular optical and photoacoustic endoscopic imaging, comprising:

[0034] binocular optical endoscopy module for acquiring left and right optical image pairs and reconstructing three-dimensional optical models;

[0035] photoacoustic endoscopy module for acquiring two-dimensional cross-sectional images and reconstructing three-dimensional photoacoustic volume data;

[0036] feature extraction module for extracting vessel surface, skeleton and bifurcation point structural features from optical and photoacoustic data;

[0037] registration optimization module for performing end-to-end iterative optimization based on a registration loss function of skeleton consistency, surface-volume distance and projection consistency to obtain the final alignment of the two modal data;

[0038] display module for outputting three-dimensional registration and registration results and supporting interactive browsing.

[0039] Further, the registration optimization module comprises:

[0040] a differentiable rendering unit for volume rendering photoacoustic volume data under the optical camera perspective to generate virtual photoacoustic images, and the rendering process is derivable with respect to rigid transformation and non-rigid deformation field parameters;

[0041] a loss construction unit for constructing a registration loss function based on the area overlap of virtual photoacoustic images and optical images, edge direction consistency and vessel skeleton topological relationship;

[0042] an optimization unit for updating registration parameters through a back propagation mechanism and an iterative optimization method.

[0043] Further, the feature extraction module is used to detect and extract bifurcation points and end points in the vessel skeleton, and topological consistency constraints are used in the registration optimization process to ensure the anatomical structure continuity of the registration results.

[0044] Further, the system further comprises a registration optimization module for differentiable volume rendering of three-dimensional photoacoustic volume data under the optical camera perspective to generate virtual photoacoustic images, and constructing a registration loss function based on the area overlap of the virtual photoacoustic images and the optical images, edge direction consistency and vessel skeleton topological constraints, and realizing cross-modal registration optimization through gradient back propagation.

[0045] The above scheme of the present application at least has the following beneficial effects:

[0046] The method first acquires a left and right view image pair through binocular endoscopy, obtains a three-dimensional point cloud of the surface of the detection target through stereo matching and triangulation, and then reconstructs a three-dimensional optical surface model; meanwhile, a two-dimensional cross-sectional image sequence is collected through photoacoustic endoscopy, and a three-dimensional photoacoustic volume model of the detection target is obtained through volume data reconstruction; then, a registration loss function containing skeleton consistency, surface-volume distance constraint and projection consistency is constructed, and the spatial transformation relationship between the optical surface model and the photoacoustic volume model is iteratively optimized in three-dimensional space, so that high-precision cross-modal registration of the two kinds of data is realized; the method can realize complementary fusion of structural information based on the three-dimensional optical surface model and the three-dimensional photoacoustic volume model without frame-by-frame pose estimation, ensure the continuity of vascular topology and the anatomical interpretability, and has the advantages of high precision, good robustness and strong real-time performance, and can be used for intraoperative navigation and auxiliary diagnosis. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 is a flowchart of a three-dimensional image registration method based on binocular optical and photoacoustic endoscopy imaging provided by an embodiment of the present application.

[0048] Figure 2 is a schematic diagram of a three-dimensional image registration system based on binocular optical and photoacoustic endoscopy imaging provided by an embodiment of the present application. DETAILED DESCRIPTION

[0049] The accompanying drawings and specific embodiments further illustrate the present application. It should be understood that these embodiments are only used to assist in illustrating the principles and technical solutions of the present application, and do not constitute a limitation on the protection scope of the present application. In addition, those skilled in the art can make various equivalent replacements or modifications to the present application without departing from the spirit and essence of the present application after reading the present specification, and these equivalent modifications shall be included in the scope of protection of the present application. In view of the problems of photoacoustic image perspective distortion, structure mismatch and uncontrollable pose in the existing intraoperative imaging process, a three-dimensional image registration method based on binocular optical and photoacoustic endoscopy imaging is proposed. The method can realize high-precision three-dimensional fusion of cross-modal data without external positioner, and enhance the structural consistency, geometric accuracy and clinical interpretability of intraoperative image information.

[0050] The technical problem to be solved by the present application is that it no longer depends on white light image pose estimation and cross-channel mapping, but directly based on binocular optical three-dimensional reconstruction and photoacoustic three-dimensional volume data, through constructing a registration loss function and a differentiable rendering mechanism, to realize accurate registration of two modal data.

[0051] The three-dimensional image registration system based on binocular optical and photoacoustic endoscopy imaging is composed of binocular optical endoscopy, photoacoustic endoscopy, image acquisition and synchronization unit, feature extraction module, registration optimization module and display terminal; the binocular endoscopy acquires an optical image pair through left and right cameras and reconstructs a three-dimensional optical model; the photoacoustic endoscopy acquires a two-dimensional B-scan sequence and reconstructs a three-dimensional photoacoustic volume data; the feature extraction module extracts a blood vessel surface and a skeleton from two modal data; the registration optimization module realizes end-to-end optimization based on a registration loss function; and the display terminal is used for outputting a three-dimensional registration result.

[0052] As shown in the three-dimensional image registration method based on binocular optical and photoacoustic endoscopy imaging, the implementation process is as follows: Figure 1

[0053] Step S1, dual-modal data acquisition, simultaneously acquiring a binocular optical image pair and a photoacoustic B-scan sequence under a unified time reference; the optical side obtains a three-dimensional optical point cloud through stereo matching and triangulation; the photoacoustic side obtains a three-dimensional photoacoustic volume data through beamforming and voxel reconstruction; the acquisition result is used as a subsequent registration input, specifically including: starting the image acquisition and synchronization unit, using a high-precision clock synchronization protocol (such as PTP protocol), locking the sampling frequencies of the left and right cameras of the binocular optical endoscope and the signal acquisition module of the photoacoustic endoscope (the recommended sampling frequency matching is 15-30 Hz, ensuring that the single-frame data acquisition time difference is ≤1 ms), avoiding the mismatch of anatomical structure positions caused by acquisition time sequence deviation.

[0054] Device parameter pre-calibration: binocular optical endoscopy calibration: using a checkerboard calibration board, acquiring calibration images at different angles and distances, and solving the intrinsic matrix K (including focal length, principal point coordinates, distortion coefficient) of the left and right cameras and the extrinsic parameters (rotation matrix R, translation vector T) between the two cameras through Zhang Zhengyou calibration algorithm to complete the stereo vision system calibration; photoacoustic endoscopy calibration: through a phantom with known size (such as an agar phantom containing standard blood vessels), the scanning step of the photoacoustic probe, the sound velocity parameter of the ultrasonic transducer and the pixel-physical size mapping relationship of the B-scan image are calibrated to ensure the accuracy of subsequent voxel reconstruction.

[0055] ​The calibrated binocular optical endoscope is inserted into the surgical cavity and aimed at the target anatomical region (such as soft tissue containing blood vessels), and the synchronous unit triggers the left and right cameras to simultaneously collect optical image pairs (recommended resolution 1920x1080, format RGB or grayscale image); the collected image pairs are preprocessed (including grayscale, Gaussian filter denoising, histogram equalization to enhance contrast), and then a feature-based stereo matching algorithm (such as SIFT-SGM algorithm, which takes into account feature robustness and matching efficiency) is used to complete the matching of corresponding points between left and right images, and a disparity map is obtained; combined with the pre-calibrated intrinsic matrix K and extrinsic parameters (R, T), the three-dimensional space coordinates of each matching point are calculated by triangulation principle, and the matching error outliers are removed (using RANSAC algorithm), and finally the initial three-dimensional optical point cloud is generated.

[0056] Photoacoustic B-scan sequence acquisition and volume data sketch generation: the photoacoustic endoscopic probe and the binocular optical endoscope are coaxial or fixed synchronously (avoiding relative pose changes during operation), and the synchronous unit triggers the photoacoustic probe to scan the target region according to the preset path (such as spiral scanning or linear scanning, the scanning range covering the optical imaging area), and the ultrasonic transducer receives the photoacoustic signal (i.e. the ultrasonic signal generated after the light-absorbing body absorbs the pulsed laser); the analog photoacoustic signal is converted into a digital signal through a data acquisition card, and a delay-sum (DAS) beam forming algorithm is used to focus process the digital signal, generating a single two-dimensional photoacoustic B-scan image (recommended resolution 512x512); the probe pose information of each frame of B-scan image is recorded according to the scanning time sequence (based on the built-in encoding module of the probe, without the need for external positioning equipment), and the multiple frames of B-scan images are sequentially arranged according to the scanning path, forming a three-dimensional photoacoustic volume data sketch (without interpolation optimization).

[0057] The processed initial three-dimensional optical point cloud and the three-dimensional photoacoustic volume data sketch are standardized in format (unified as PLY format or DICOM format, adapted to the subsequent registration module), serving as input data for the three-dimensional model reconstruction in step S2.

[0058] Step S2, three-dimensional model reconstruction; on the optical side, disparity estimation is performed on the binocular image pair, and three-dimensional coordinates are calculated in combination with the intrinsic matrix K to generate an optical point cloud ; on the photoacoustic side, B-scan sequences are superimposed according to the probe scanning path, and a voxel grid is formed through interpolation to generate a three-dimensional photoacoustic volume data , specifically including: for the initial disparity map generated in S1, a sub-pixel interpolation algorithm is used to improve the accuracy of disparity calculation (the accuracy of disparity is improved from pixel level to sub-pixel level), and at the same time, the error disparity in the occluded area is removed through left-right consistency check (the disparity in the occluded area does not satisfy the left-right consistency, and is directly marked as invalid area), and an optimized high-precision disparity map is generated.

[0059] Three-dimensional coordinate accurate calculation and point cloud optimization, based on the optimized parallax map, combined with the pre-calibrated intrinsic matrix K, the three-dimensional coordinate calculation formula is derived through the pinhole camera model (X=(u-cx) x Z / f, Y=(v-cy) x Z / f, where Z is the depth value, which is calculated by parallax, (u, v) is the pixel coordinate, (cx, cy) is the principal point coordinate, and f is the focal length), the three-dimensional space coordinates of each effective pixel are calculated; post-processing of the generated initial point cloud: statistical filtering is used to remove the remaining outliers (set the number of neighborhood points and distance threshold), the point cloud density is reduced by voxel downsampling (retain key structural features and improve subsequent processing efficiency), and finally a high-quality three-dimensional optical point cloud S is generated, which clearly retains the surface anatomical structure of the target region (such as the blood vessel surface contour).

[0060] B-scan sequence alignment and completion, based on the probe pose information of each frame of B-scan image recorded by S1, all B-scan images are aligned to a unified world coordinate system (consistent with the optical point cloud coordinate system) using rigid transformation (rotation, translation); for the data missing area that may appear in the scanning process (such as the gap caused by uneven probe scanning step), linear interpolation or cubic spline interpolation is used to supplement the missing pixel data, ensuring the continuity of the B-scan sequence.

[0061] Voxel grid generation and volume data construction, according to the actual size of the target region, set the resolution of the voxel grid (recommended voxel size is 0.1-0.5mm, balance accuracy and efficiency), the aligned B-scan sequence is stacked layer by layer into the voxel grid according to the scanning path; the pixel value of the B-scan image is mapped to each voxel of the voxel grid using trilinear interpolation algorithm, generating three-dimensional photoacoustic volume data V; Gaussian filter denoising processing is performed on the volume data to enhance the contrast of target structures such as blood vessels and background, clearly presenting the internal vascular structure of the target region (such as blood vessel skeleton, blood vessel lumen).

[0062] This invention avoids the perspective deviation between adjacent frames caused by relative pose changes during the operation of the photoacoustic probe and optical endoscope through high-precision time synchronization and coaxial fixation with the device. This provides temporally and spatially consistent dual-modal data for subsequent registration, improving the structural consistency of the fusion results. Through dual-optical endoscope self-calibration and photoacoustic endoscope's built-in encoding module recording pose, it eliminates the need for external devices such as optical trackers and electromagnetic navigators, adapting to the narrow surgical cavity environment of laparoscopy and avoiding the impact of instrument and tissue obstruction on positioning, thus improving the feasibility of intraoperative application. The optical side employs high-precision calibration... Parallax optimization and point cloud post-processing, along with B-scan alignment, interpolation completion, and voxel-based structured reconstruction on the photoacoustic side, effectively address the challenges of reconstructing soft tissue and irregular organ regions, avoiding vascular structure misalignment, breakage, and topological errors, and enhancing anatomical interpretability. The generated 3D optical point cloud S (preserving surface structure) and 3D photoacoustic volume data V (preserving internal structure) provide high-quality data sources for the subsequent feature extraction module (extracting vascular surface and skeleton), helping to solve the problem that traditional grayscale / intensity-based similarity metrics are difficult to capture bimodal correspondences.

[0063] Step S3: Blood vessel feature extraction. The optical 3D model is processed by blood vessel enhancement filtering and segmentation to obtain the blood vessel skeleton point set. And identify the set of bifurcation points. The photoacoustic data is processed through threshold segmentation and skeletonization to obtain a skeleton point set. and the set of bifurcation points These features are used for geometric and topological constraints.

[0064] Step S4, loss function construction: To achieve cross-modal registration, this invention defines the following loss function:

[0065] Skeleton consistency loss function :

[0066]

[0067] in, For optical skeleton point set; For the photoacoustic vascular skeleton point set; Belongs to Optical skeleton points; Belonging to The photoacoustic skeleton points; For a non-rigid deformation field, Φ(x) represents the point in the optical coordinate system. The point mapped to the photoacoustic coordinate system; T is a rigid transformation or three-dimensional interpolation operator, and T(Φ(x)) represents the corresponding point obtained after performing three-dimensional interpolation on the mapped point Φ(x) in the photoacoustic volume data; The bifurcation point weighting function for the backbone point is used to increase the weight of the bifurcation point in the loss calculation. This loss is used to ensure that the blood vessel skeleton in the optical model is consistent with the blood vessel skeleton in the photoacoustic three-dimensional body data in spatial position, thereby realizing cross-modal geometric structure alignment.

[0068] Surface-volume distance loss function:

[0069]

[0070] wherein, is a set of surface points in the optical three-dimensional model; is an optical surface point belonging to ;

[0071] is a non-rigid deformation field, denotes the point after mapping the optical point to the photoacoustic coordinate system;

[0072] denotes the corresponding three-dimensional point obtained by three-dimensional interpolation of the mapping point in the photoacoustic body data; is a signed distance field function constructed from the photoacoustic body data, used to describe the signed distance of any three-dimensional point to the boundary of the photoacoustic surface; is a robust function used to suppress the influence of abnormal points on the loss value. This loss ensures that the optical surface is consistent with the boundary of the photoacoustic data, and realizes geometric alignment of the two modalities in the surface structure.

[0073] Projection consistency loss function: a virtual photoacoustic image is generated by a differentiable renderer ; ;

[0074] soft-IoU measures the degree of region overlap:

[0075]

[0076] wherein, denotes the pixel position index in the two-dimensional image plane, used to access the corresponding pixel position in the two masks; denotes the value of the virtual photoacoustic mask generated by the projection of the three-dimensional photoacoustic body data at pixel ; denotes the blood vessel mask value of the optical image at pixel ; is a positive constant introduced to prevent numerical instability caused by a zero denominator, preferably with a value of 10 -6 . The soft-IoU loss is used to ensure the consistency of the two modalities at the region level. In the soft-IoU loss, traverse all pixel positions; in the gradient direction consistency loss, only traverse pixel positions in the pixel set Ω.

[0077] The average gradient direction cosine measures the edge direction consistency:

[0078]

[0079] where, is the pixel set participating in the edge direction consistency calculation; represents the pixel position belonging to Ω; represents the gray value or intensity value of the virtual photoacoustic image rendered by the three-dimensional photoacoustic body data at the pixel ; represents the gray value or intensity value of the optical image at the pixel p; , are their gradient vectors, respectively; , are the norms of the above gradient vectors, respectively; is a positive constant to prevent numerical instability caused by a zero denominator, preferably 10 -6 . This loss is used to measure the consistency of the virtual photoacoustic image and the optical image in the edge direction, thereby improving the matching accuracy of the cross-modal images in the edge structure.

[0080] Combining the above two loss functions gives: ; where, and are the weight coefficients for balancing the two loss terms;

[0081] The regularization function term :

[0082] ;

[0083] is used to constrain the prior of the rigid parameter, the smoothness of the non-rigid deformation field, the rationality of the Jacobian, and the variation range of the scale parameter; , , , represent the weight coefficients of each regularization subterm, respectively; represents the Lie algebra parameter of the current rigid transformation; represents the prior value of the rigid parameter, which is used to constrain the rigid transformation not to deviate from the expected range; represents the Laplacian (i.e., the second-order smoothing operator) of the non-rigid deformation field Φ, which is used to constrain the spatial smoothness of the deformation field; represents the Jacobian determinant of the deformation field Φ at position x, which is used to limit the local volume change and avoid deformation folding; denotes a positive constant for setting the allowed minimum determinant lower bound in the Jacobian regularizer; s is a scale parameter, denotes its logarithmic form for limiting the scale too large or too small; denotes the L2 norm for measuring the size of a vector or tensor, which is defined as the square root of the sum of squares of each component.

[0084] The core goal of this step is to build a multi-dimensional loss function that integrates geometric, topological, and projection constraints to achieve end-to-end optimization of rigid transformation and non-rigid deformation. It addresses the problems of non-linear deformation and perspective effect that traditional rigid / affine transformation cannot handle. It extracts topological / geometric features such as vessel skeletons and bifurcation points (instead of gray scale / intensity), avoids the differences in optical and photoacoustic signal expressions, and accurately captures the homologous correspondence of dual-modality data. Bifurcation points serve as topological key nodes, providing strong constraints for registration to avoid misplacement and fracture of blood vessel branches, and improving the anatomical interpretability of the reconstruction results. The feature extraction process balances accuracy and efficiency (single-frame processing time < 200 ms) to meet the real-time needs of laparoscopic surgery. Skeleton consistency constraints, surface-volume distance constraints on spatial boundaries, and projection consistency constraints on imaging angles all ensure the structural consistency of the registration results. The registration is achieved through the geometric / topological constraints of the loss function itself, eliminating the need for external devices such as optical trackers and electromagnetic navigators. This design is suitable for the narrow surgical cavity of laparoscopy and avoids the influence of instrument / tissue occlusion. The robust function and soft-IoU design alleviate the interference of outliers, have stronger gradient continuity, and enable end-to-end optimization without the need for step-by-step processing, resulting in a more than 30% improvement in registration efficiency.

[0085] Step S5, registration optimization, combines the above items into a total goal:

[0086] ;

[0087] A multi-scale strategy is adopted, first optimizing rigidity, then gradually introducing non-rigidity, and using optimizers such as Adam for iterative updates; denotes the common region of interest (ROI) of photoacoustic (PA) and optical (Opt) three-dimensional data, focusing on blood vessels and lesion areas, and excluding background noise areas; denotes the voxel gray value of the photoacoustic image at spatial position ; denotes the gradient vector of the photoacoustic image at position , representing the blood vessel contour / edge information; denotes the gradient modulus value of the optical image at position , reflecting the structural sharpness of the optical image; denotes the gradient modulus value of the photoacoustic image at position , reflecting the structural sharpness of the photoacoustic image.

[0088] Initialize the registration core parameters:

[0089] rigid transformation parameters, translation vector (initially a zero vector), rotation matrix (initially a unit matrix); non-rigid transformation parameters, using a B-spline deformation model , control point displacement initialized to 0; loss weight, , , , (fine-tuned by cross-validation); optimizer configuration, Adam optimizer, initial learning rate , weight decay ; early stopping if iteration does not converge for more than 1000 rounds, output the current optimal parameters and mark for manual review. Randomly select 100 feature points (branch points, lesion boundary points) to verify the coordinate deviation after registration (mean , maximum ); topological continuity verification: check the direction of blood vessel branches, connection of branch points, and absence of isolated branches or false connections through a skeleton topological algorithm; fine-tune the transformation parameters in combination with intraoperative anatomical marker points (such as blood vessel anastomosis) to match the actual anatomical location. Rigid matrix, non-rigid deformation field control point data; align the transformed optical model with the photoacoustic volume data.

[0090] Step S6, registration display, superimpose the registered optical three-dimensional model and photoacoustic three-dimensional volume data, and the doctor can view the individual modalities or registration results as needed to achieve intraoperative blood vessel navigation and lesion identification, which specifically includes: converting the registered optical three-dimensional mesh (triangular facets) to GLTF format and converting the photoacoustic volume data to DICOMVolume format; performing gradient enhancement and threshold segmentation on the photoacoustic volume data (highlighting blood vessels) and performing surface smoothing on the optical model. The bottom layer is the photoacoustic volume data (volume rendering / surface rendering, transparency 0-100% adjustable), and the upper layer is the optical blood vessel model (wireframe / solid rendering, custom color marking blood vessels / lesions); ensure that the real-time interactive frame rate during surgery is ≥30fps to meet the dynamic viewing needs of surgery. Multi-modal superimposed visualization intuitively presents the spatial correspondence of blood vessels / lesions, real-time interaction and measurement functions during surgery, enabling precise blood vessel navigation and lesion identification, reducing surgical risk, and improving decision-making efficiency. It solves the problems of geometric mismatch and topological rupture of optical-photoacoustic multi-modal images, providing high-precision and interactive image support for intraoperative precise navigation.

[0091] A three-dimensional image registration system based on binocular optical and photoacoustic endoscopic imaging includes:

[0092] A binocular optical endoscopic module is used to acquire left and right optical image pairs and reconstruct a three-dimensional optical model;

[0093] a photoacoustic endoscopy module for acquiring two-dimensional cross-sectional images and reconstructing three-dimensional photoacoustic volume data;

[0094] a feature extraction module for extracting vessel surface, skeleton and bifurcation point structural features from optical and photoacoustic data;

[0095] a registration optimization module for performing end-to-end iterative optimization based on a registration loss function of skeleton consistency, surface-volume distance and projection consistency to obtain the final alignment of the two modal data;

[0096] a display module for outputting three-dimensional registration and registration results and supporting interactive browsing.

[0097] Further, the registration optimization module comprises:

[0098] a differentiable rendering unit for volume rendering photoacoustic volume data under the optical camera perspective to generate virtual photoacoustic images, and the rendering process is derivable with respect to rigid transformation and non-rigid deformation field parameters;

[0099] a loss construction unit for constructing a registration loss function based on the area overlap, edge direction consistency and vessel skeleton topological relationship of the virtual photoacoustic image and the optical image;

[0100] an optimization unit for updating the registration parameters through a back propagation mechanism and an iterative optimization method.

[0101] Further, the feature extraction module is used to detect and extract bifurcation points and end points in the vessel skeleton, and topological consistency constraints are used in the registration optimization process to ensure the anatomical structure continuity of the registration results.

[0102] Further, the system further comprises a registration optimization module for differentiable volume rendering of three-dimensional photoacoustic volume data under the optical camera perspective to generate virtual photoacoustic images, and constructing a registration loss function based on the area overlap, edge direction consistency and vessel skeleton topological constraints of the virtual photoacoustic image and the optical image, and realizing cross-modal registration optimization through gradient back propagation.

[0103] It should be noted that the system corresponds to the above method, and all implementation manners in the above method embodiments are applicable to this embodiment and can achieve the same technical effects. The above is the preferred embodiment of the present application. It should be noted that for ordinary skilled persons in the technical field, without departing from the principles of the present application, a number of improvements and refinements can be made, which should be considered as the protection scope of the present application.

Claims

1. A method of three-dimensional image registration based on binocular optical and photoacoustic endoscopic imaging, characterized in that, The process includes the following steps: Step S1, acquiring optical image pairs from left and right perspectives using binocular endoscopy, and reconstructing a three-dimensional optical model of the target using stereo matching and triangulation; Step S2, acquiring a sequence of two-dimensional cross-sectional images using photoacoustic endoscopy, and generating a three-dimensional photoacoustic volume model through volume data reconstruction; Step S3, extracting vascular skeleton, surface, and bifurcation point features from the optical three-dimensional model and the photoacoustic three-dimensional volume data, respectively; Step S4, establishing a registration transformation model between the three-dimensional optical model and the three-dimensional photoacoustic volume model. The registration transformation model includes sequentially acting rigid transformations and non-rigid deformation fields, using the rigid transformation parameters and the control parameters of the non-rigid deformation fields as variables to be optimized. The registration transformation maps the 3D photoacoustic model to the coordinate system of the 3D optical model to obtain the mapping result. Based on the mapping result, the skeleton consistency loss function, surface-volume distance loss function, and projection consistency loss function are calculated and weighted to construct the registration loss function. The registration loss function is used as the objective function to form an end-to-end 3D registration optimization model. In step S5, the registration loss function is minimized, and the rigid transformation parameters and the control parameters of the non-rigid deformation field are iteratively updated until the preset convergence condition is met. The final rigid transformation and the optimal non-rigid deformation field are obtained, and the 3D photoacoustic model is transformed to obtain the 3D registration result aligned with the 3D optical model space in the same coordinate system.

2. The three-dimensional image registration method based on binocular optical and photoacoustic endoscopic imaging according to claim 1, characterized in that, The backbone consistency loss function is defined as: ； wherein, is a set of optical skeleton points; is a set of photoacoustic vascular skeleton points; is a bifurcation point weight; is a skeleton point coordinate in an optical three-dimensional model; is a skeleton point coordinate in a photoacoustic three-dimensional volume data; Φ(x) represents a spatial transformation function of mapping a point x in an optical coordinate system to a photoacoustic three-dimensional coordinate system; T(Φ(x)) represents a photoacoustic volume data corresponding point coordinate obtained by three-dimensional interpolation on the mapped position Φ(x).

3. The three-dimensional image registration method based on binocular optical and photoacoustic endoscopic imaging according to claim 2, characterized in that, The surface-volume distance loss function is defined as: ； wherein, is a set of optical surface points; is a spatial coordinate of an optical surface point; is a robust function; is a signed distance field of the photoacoustic volume data.

4. The three-dimensional image registration method based on binocular optical and photoacoustic endoscopic imaging according to claim 3, characterized in that, The projection consistency loss function includes the following two parts: soft-IoU loss function. : Used to measure virtual photoacoustic mask With optical vascular mask Regional overlap; Represents the pixel position in a two-dimensional image plane. This indicates that the virtual photoacoustic mask generated by projection based on 3D photoacoustic volume data is located at the pixel level. The value at; Indicates the optical image in pixels Values ​​of the vascular mask at the location; A positive constant to prevent numerical instability caused by a denominator of zero; Edge direction consistency loss function : Used to measure virtual photoacoustic images With optical images The degree of edge direction matching; This represents the set of pixels involved in edge direction calculations; Indicates the pixel position belonging to Ω; This represents a virtual photoacoustic image obtained by rendering or projecting three-dimensional photoacoustic volume data in pixels. The grayscale or intensity value at that location; Indicates the optical image in pixels The grayscale or intensity value at that location; , These are their gradient vectors; , These are the magnitudes of the gradient vectors mentioned above.

5. The three-dimensional image registration method based on binocular optical and photoacoustic endoscopic imaging according to claim 4, characterized in that, The optimization of the registration parameters adopts an end-to-end iterative update mechanism based on the registration loss function, which specifically includes the following steps: Construct the total loss function: ； in, Denotes the Lie algebra parameters of a rigid transformation; Represents a non-rigid deformation field; s For scale parameters; This represents the non-rigid deformation field after introducing the scale parameter s; This represents the defined skeleton consistency loss function; This represents the defined surface-volume distance loss function; This represents the defined projection consistency loss, which includes soft-IoU loss and edge orientation consistency loss. Represents the deformation field The applied regularization term is used to limit the smoothness or complexity of the deformation field; , , , These represent the skeleton consistency loss in the total loss function, respectively. Surface-volume distance loss Projection consistency loss Regularization loss Weighting coefficients; Calculate the gradient of the total loss function with respect to the optimization variables and obtain the update direction through backpropagation; update the parameters using an iterative method based on gradient descent or Adam optimizer until the loss function converges.

6. The three-dimensional image registration method based on binocular optical and photoacoustic endoscopic imaging according to claim 5, characterized in that, The registration process involves topological correction of the vascular skeleton within the registration area using structural consistency constraints, specifically including: Detecting bifurcation points and endpoints in optical and photoacoustic vascular skeletons; establishing spatial correspondences of cross-modal key points in a unified coordinate system; correcting locally misaligned or discontinuous vascular segments based on topological consistency constraints to maintain the continuity and topological fidelity of registration results at the anatomical level.

7. A three-dimensional image registration system based on binocular optical and photoacoustic endoscopic imaging, the system being used to implement the method as described in any one of claims 1 to 6, characterized in that, include: The binocular optical endoscope module is used to acquire left and right optical image pairs and reconstruct a three-dimensional optical model; The photoacoustic endoscope module is used to acquire two-dimensional cross-sectional images and reconstruct three-dimensional photoacoustic body data; The feature extraction module is used to extract structural features of blood vessel surface, skeleton and bifurcation points from optical and photoacoustic data; The registration optimization module is used to perform end-to-end iterative optimization based on the registration loss function of skeleton consistency, surface-volume distance and projection consistency to obtain the final alignment of the two modal data; The display module is used to output 3D registration and registration results, and supports interactive browsing.

8. The three-dimensional image registration system based on binocular optical and photoacoustic endoscopic imaging according to claim 7, characterized in that, The registration optimization module includes: Differentiable rendering units are used to perform volumetric rendering of photoacoustic data from the perspective of an optical camera, generating virtual photoacoustic images. The rendering process is differentiable with respect to rigid transformation and non-rigid deformation field parameters. Loss construction unit is used to construct a registration loss function based on the regional overlap, edge direction consistency and vascular skeleton topology of virtual photoacoustic image and optical image; The optimization unit is used to update the registration parameters through backpropagation and iterative optimization methods.

9. The three-dimensional image registration system based on binocular optical and photoacoustic endoscopic imaging according to claim 8, characterized in that, The feature extraction module is used to detect and extract bifurcation points and endpoints in the vascular skeleton, and to ensure the continuity of the anatomical structure of the registration result through topological consistency constraints during the registration optimization process.

10. The three-dimensional image registration system based on binocular optical and photoacoustic endoscopic imaging according to claim 9, characterized in that, The system also includes a registration optimization module, which is used to perform differentiable volume rendering on the three-dimensional photoacoustic volume data from the perspective of the optical camera to generate a virtual photoacoustic image. Based on the regional overlap, edge direction consistency and vascular skeleton topological constraints of the virtual photoacoustic image and the optical image, a registration loss function is constructed, and cross-modal registration optimization is achieved through gradient backpropagation.

Images