A method and apparatus for intraluminal optical multi-modal image three-dimensional registration

Abstract

The application discloses a method and device for intracavity optical multimodal image three-dimensional registration, comprising: calibrating a multimodal endoscope system, constructing a world coordinate system; laser marking in the cavity, three-dimensional reconstruction by binocular camera, sensing three-dimensional topography of the lesion area; expanding the balloon, extending into the OCT probe along the guide wire, scanning the cavity by retracting, and detecting the balloon marker; in the process, the binocular camera identifies the periodic flash point at the top of the probe, calculates the retraction trajectory of the probe in real time, and obtains the eccentricity between the balloon and the guide wire; then, the photoacoustic probe is replaced, the scanning process is repeated, and the same balloon marker is identified synchronously; finally, based on the marker features and system parameters shared by each mode, three-dimensional registration and information fusion of binocular vision, OCT and photoacoustic imaging are realized. The method can realize fast and accurate registration of optical multimodal images, effectively fuse multimodal information, and significantly improve the detection and auxiliary diagnosis ability of intracavity early cancer.

Description

Technical Field

[0001] This invention relates to the field of endoscopy technology, and more specifically to a method and apparatus for three-dimensional registration of intracavitary optical multimodal images. Background Technology

[0002] Early clinical detection and assessment of treatment boundaries are fundamental to endocavitary tumor therapy. Current mainstream endocavitary optical imaging modalities each possess unique advantages, revealing structural and functional information of tissues from different dimensions. For example, wide-field imaging features a large field of view, high frame rate, and intuitive visualization, enabling rapid coverage of wide cavity regions. Optical coherence tomography (OCT) boasts micrometer-level axial resolution and millimeter-level penetration depth, non-invasively presenting the microscopic tissue structure of the mucosa and submucosa. Photoacoustic imaging combines optical contrast with ultrasound penetration depth, allowing label-free acquisition of physiological functional information such as blood supply and oxygenation. However, the information scale, physical mechanisms, and spatial dimensions represented by these modalities differ significantly. Without high-precision three-dimensional spatial registration, multimodal data can only be interpreted in isolation, failing to achieve structure-function-morphology correlation analysis under a unified anatomical coordinate system, severely weakening their potential for collaborative diagnosis.

[0003] Currently, multimodal fusion in clinical practice and research largely relies on simple overlay, depending on mechanical encoders for coarse axial alignment, or manual intervention to select registration feature points. This not only limits registration accuracy but also results in low automation and poor robustness. More importantly, the complex intracavitary environment, significant tissue deformation, and non-ideal linear probe trajectories, coupled with the heterogeneity of imaging principles across different modalities and the mismatch between resolution and field of view, make cross-modal feature extraction and spatial mapping extremely challenging. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a method and apparatus for three-dimensional registration of intracavitary optical multimodal images. This method innovatively introduces binocular vision as a three-dimensional spatial mediator, achieving unified spatial anchoring of OCT and photoacoustic modalities. Simultaneously, combined with a periodic flash structure at the tip of the OCT probe, a binocular camera tracks the probe's retraction trajectory in real time, accurately quantifying the eccentricity between the balloon and the guidewire, and using this to spatially compensate for axial scanning data. Ultimately, this achieves rapid and accurate registration of binocular vision, OCT, and photoacoustic imaging. The entire registration process requires no manual intervention, relying on hardware design and algorithmic collaboration to achieve fast, stable, and repeatable three-dimensional alignment, suitable for real-time clinical diagnostic scenarios.

[0005] This invention provides a method for three-dimensional registration of intracavity optical multimodal images, comprising the following steps: S1: Calibrate the multimodal endoscopic system and construct a unified world coordinate system; S2: Laser marking is performed on the surface of tissue adjacent to the lesion, and the three-dimensional morphology of the lesion area is reconstructed based on binocular vision; S3: Advance the balloon catheter to the target lesion segment, inflate the balloon, insert the OCT probe along the guidewire into the balloon, perform a retraction scan, and detect the first balloon marker; record the axial length L covered by the OCT scan. OCT ; S4: The periodic flashing point at the tip of the OCT probe is identified by a binocular camera, and the retraction trajectory of the OCT probe is calculated in real time to obtain the eccentricity between the balloon and the guidewire. S5: Replace the OCT probe with a photoacoustic probe to perform a retraction scan, reconstruct a three-dimensional photoacoustic image, and simultaneously detect the second balloon marker; record the axial length L covered by the photoacoustic scan. PAI ; S6: Based on the eccentricity between the balloon and the guidewire, for L OCT L PAI Spatial position compensation is performed to correct the actual scanning starting position and axial distribution of OCT and photoacoustic scans in the world coordinate system; based on the first and second balloon markers, rotation correction is performed on the OCT or photoacoustic images to achieve synchronous alignment of the two modalities in the circumferential angle; the transformation matrix of the OCT image relative to the binocular vision coordinate system is calculated to achieve spatial alignment of the OCT and binocular images; the photoacoustic data is mapped to the binocular vision coordinate system to achieve registration of the OCT, photoacoustic and binocular vision trimodal images in a unified three-dimensional space.

[0006] Further, step S1 includes the following sub-steps: S11: Place the checkerboard calibration plate at the exit of the binocular endoscope clamp channel and adjust the spatial orientation of the checkerboard calibration plate so that it is perpendicular to the optical axis of the lens. S12: Collect several sets of synchronized left and right image pairs at different angles and distances on the chessboard calibration board; S13: Calculate the intrinsic parameter matrix, distortion coefficients, rotation matrix, and translation vector of the stereo camera based on the acquired images; S14: Set the left camera coordinate system to the world coordinate system, and the extension direction of the pincer is the positive Z-axis direction.

[0007] Further, step S2 includes the following sub-steps: S21: After confirming the location of the lesion, a laser is projected onto the surface of the tissue adjacent to the lesion to form an artificial feature marker; S22: Acquire left and right eye images within the cavity using a binocular camera; perform distortion correction on the left and right eye images within the cavity based on the intrinsic parameter matrix and distortion coefficients; calculate the stereo correction transformation using the rotation matrix and translation vector. S23: Dense stereo matching and 3D reconstruction are performed using a semi-global block matching algorithm to obtain a dense 3D point cloud of tissue adjacent to the lesion.

[0008] Furthermore, in sub-step S21, the tissue adjacent to the lesion refers to the normal tissue area within 2-5 mm beyond the edge of the lesion; the laser is projected through a semiconductor laser emitter integrated into the sidewall of the duct, with a wavelength of 532 nm and a power of <50 mW.

[0009] Further, step S3 includes: The balloon catheter is pushed to the target lesion segment, and sterile saline is injected through the catheter lumen using an external pressure pump to slowly inflate the balloon until the outer wall of the balloon evenly adheres to the inner wall of the lumen. After inserting the OCT probe into the balloon, the lumen is scanned by retraction, and continuous OCT slices of each cross section and the three-dimensional reconstruction results composed of each OCT slice are returned. A marker generatrix is ​​integrated axially on the outer surface of the balloon, extending continuously along the length of the balloon and parallel to its axis. When the OCT probe retracts within the balloon, this generatrix appears as a shadow signal in each frame of the OCT cross-sectional image. Image processing algorithms automatically identify this generatrix shadow and extract its angular position in the OCT cross-sectional image. The axial length L covered by the OCT scan is recorded. OCT .

[0010] Furthermore, during the retraction process of the OCT probe, at a frequency Periodic flashes; simultaneously, the binocular camera flashes at a frequency Image acquisition is performed to ensure that two frames are captured within each complete bright-dark cycle, one frame corresponding to the OCT probe being lit and the other frame corresponding to the OCT probe being off, thus obtaining an image pair of alternating bright and dark images at the OCT probe end.

[0011] Further, in step S4, the eccentricity ε between the balloon and the guidewire is ε = |C_balloon - C_wire| / R_balloon, where C_balloon is the coordinate of the center of the balloon cross-section, C_wire is the coordinate of the center of the guidewire cross-section, and R_balloon is the radius of the balloon; the eccentricity is used to evaluate the centering of the guidewire within the balloon and correct the rotation center deviation of the OCT image.

[0012] Further, step S5 includes: After completing the OCT scan and withdrawing the probe, the photoacoustic imaging probe is advanced into the target lesion area via the balloon guidewire. An automatic retraction device is then activated, retracting the photoacoustic imaging probe at a constant speed to achieve continuous axial sampling. All acquired photoacoustic cross-sectional images are stacked in spatial order to construct three-dimensional photoacoustic volume data of the lesion area. Image processing algorithms are used to detect the angular position of the balloon generatrix shadow in the photoacoustic cross-sectional image. The axial length L covered by the photoacoustic scan is recorded. PAI .

[0013] Further, step S6 includes the following sub-steps: S61: Convert the volumetric 3D structure of OCT and photoacoustics into point cloud form; identify voxels with gray values ​​below the gray threshold in the image as cavity regions and extract their boundary contours; convert the surface structure of the cavity into 3D point cloud data through a 3D edge detection algorithm; S62: Based on the eccentricity between the balloon and the guidewire, combined with the physical expansion model of the balloon, the axial length L covered by the OCT scan is... OCT axial length L covered by photoacoustic scanning PAI Spatial position compensation is performed to correct the actual scanning starting position and axial distribution of OCT scan and photoacoustic scan in the world coordinate system, so that the OCT image and photoacoustic image are spatially aligned in the direction of catheter advancement. S63: Detect the angular position of the balloon generatrix shadow in the two modal images respectively, calculate its angular deviation in the cross section, and then perform rotation correction on the OCT image or photoacoustic image so that the two modalities can be synchronously aligned in the circumferential angle. S64: Employs a point-to-point registration algorithm to calculate the transformation matrix of the OCT image relative to the binocular vision coordinate system, achieving high-precision spatial alignment between the OCT image and the binocular image; maps photoacoustic data to the binocular vision coordinate system, achieving registration of OCT, photoacoustic, and binocular vision trimodal images in a unified three-dimensional space; S65: Maps high-resolution functional and structural information from OCT and photoacoustic images to the 3D reconstructed surface area of ​​binocular vision, achieving information fusion and visualization.

[0014] The present invention also provides an apparatus for three-dimensional registration of intracavity optical multimodal images, comprising one or more processors for implementing the above-described method for three-dimensional registration of intracavity optical multimodal images.

[0015] The beneficial effects of this invention are as follows: 1. By designing the axial marking generatrix on the balloon surface, a common radial reference feature that runs through the entire scanning process is provided for OCT and photoacoustic imaging, effectively achieving precise radial alignment across modes; 2. Introduce a periodic flash structure at the tip of the OCT probe to achieve dynamic trajectory tracking and eccentricity correction; 3. Using the 3D point cloud of the cavity surface reconstructed by binocular endoscope and the laser marking points as the macroscopic spatial reference, the microstructural information of OCT and the functional metabolic information of photoacoustics are uniformly mapped to the same world coordinate system through point cloud registration algorithms such as Iterative Closest Point (ICP), realizing the three-in-one multimodal high-precision fusion and visualization of "macromorphology-microstructure-physiological function". Attached Figure Description

[0016] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a schematic diagram of the binocular vision three-dimensional reconstruction result of the esophageal phantom according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the marking signal of the balloon generatrix under the OCT cross-sectional image in an embodiment of the present invention; Figure 4 This is a schematic diagram of the OCT probe image acquisition and differential positioning results in an embodiment of the present invention; wherein, Figure 4 In the diagram, A represents the previous frame captured by the left camera, showing alternating light and dark areas of the OCT probe. Figure 4 B in the image is a schematic diagram of the alternating light and dark frames of the OCT probe taken by the left camera. Figure 4 C in the diagram is a probe position obtained by the finite difference method; Figure 5 This is a schematic diagram of the retraction trajectory of the OCT probe in an embodiment of the present invention; Figure 6 This is a schematic diagram of the structure of the device of the present invention. Detailed Implementation

[0017] The present invention will now be described in detail with reference to the accompanying drawings and preferred embodiments. The objectives and effects of the present invention will become clearer from this description. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of the invention. The present invention will be described in detail below with reference to the accompanying drawings. Unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0018] This example uses early-stage esophageal cancer.

[0019] like Figure 1 As shown, the present invention proposes a method for three-dimensional registration of intracavity optical multimodal images, comprising the following steps: S1: Calibrate the multimodal endoscopic system and establish a unified world coordinate system; this includes the following sub-steps: S11: Prepare a checkerboard calibration plate of known size, place it directly in front of the exit of the binocular endoscope's clamp channel, and adjust its spatial orientation so that the normal direction of the calibration plate's plane is basically coincident with the mechanical axis of the clamp channel, that is, the calibration plate is perpendicular to the optical axis of the lens, and ensure that its Z-axis direction is consistent with the forward observation direction of the endoscope.

[0020] S12: Acquire 10-20 sets of synchronized left and right image pairs at different angles and distances on the calibration plate.

[0021] S13: Calculate the intrinsic parameter matrix of the stereo camera based on the acquired images. K L , K R ), distortion coefficient (dist L dist R ) and extrinsic parameters (rotation matrix) R Translation vector T ).in, K L , K R These represent the intrinsic parameter matrices of the left and right cameras, respectively, containing parameters such as focal lengths f_x and f_y and principal point coordinates c_x and c_y, obtained through Zhang Zhengyou's calibration method; dist L dist R These are the radial distortion coefficient vectors for the left and right cameras, respectively, which are obtained through calibration along with the intrinsic parameter matrix.

[0022] S14: Set the left camera coordinate system as the world coordinate system, and the extension direction of the pincers is the positive Z-axis. This setting ensures that all subsequent 3D reconstruction results are directly expressed in this unified coordinate system, facilitating spatial alignment with other modal data.

[0023] S2: Laser marking is performed on the surface of tissue adjacent to the lesion, and the three-dimensional shape reconstruction of the lesion area is completed based on binocular vision; this includes the following sub-steps: S21: After confirming the location of the lesion, a laser is projected onto the surface of the tissue adjacent to the lesion to form an artificial feature marker. The tissue adjacent to the lesion refers to the normal tissue area within 2-5 mm beyond the edge of the lesion; the laser is projected through a semiconductor laser emitter integrated into the sidewall of the duct, with a wavelength of 532 nm and a power of <50 mW, forming a visible marker.

[0024] S22: Acquire left and right eye images within the cavity using a binocular camera. Based on the intrinsic parameter matrix and distortion coefficients obtained in step S1, perform distortion correction on the left and right eye images within the cavity. Then, utilize the rotation matrix obtained in step S1... R Translation vector T Calculate the stereo correction transformation.

[0025] S23: Dense stereo matching and 3D reconstruction are performed using a semi-global block matching (SGBM) algorithm to obtain a dense 3D point cloud of tissue adjacent to the lesion, accurately reflecting the microscopic morphology of the intracavitary tissue surface, such as... Figure 2 As shown.

[0026] S3: Inflate the balloon and insert the OCT probe to complete the retraction scan and identify the balloon markers; specifically including the following sub-steps: S31: After the binocular camera has acquired the image in sub-step S22, the integrated balloon catheter is pushed to the target lesion segment. Sterile saline is injected through the catheter lumen using an external pressure pump to slowly inflate the balloon until the outer wall of the balloon evenly adheres to the inner wall of the lumen.

[0027] S32: After the OCT probe is inserted into the balloon, it scans the lumen by retracting and returns continuous OCT slices of each cross section, as well as the three-dimensional reconstruction results composed of each OCT slice, which includes information about the lesion tissue.

[0028] S33: A high-contrast marker generatrix is ​​integrated axially on the outer surface of the balloon, extending continuously along the length of the balloon and parallel to its axis. When the OCT probe retracts within the balloon for scanning, this generatrix appears as a distinct shadow signal in each frame of the OCT cross-sectional image, such as... Figure 3 As shown. The shadow can be automatically identified by image processing algorithms, and the angular position of the generatrix shadow in the cross section can be extracted; this angular information serves as a spatial reference feature shared by OCT and photoacoustic modes, providing a key radial alignment basis for subsequent multimodal 3D registration.

[0029] S34: During the OCT probe retraction process, the motor's built-in encoder records the probe's movement displacement in real time. After scanning is completed, the control system automatically reads the total stroke data fed back by the motor, denoted as L. OCT This indicates the axial length covered by the current OCT scan. The motor incorporates an absolute rotary encoder with a resolution of 12 bits, or 4096 pulses per revolution. The stepper motor is connected to the OCT probe via a drive shaft, and the control system sends pulse signals via serial port / USB to control the retraction speed and distance.

[0030] S4: Based on the periodic flashes at the tip of the OCT probe, the trajectory of the OCT probe is tracked in real time by a binocular camera to calculate the balloon-guidewire eccentricity; the eccentricity between the balloon and the guidewire ε=|C_balloon-C_wire| / R_balloon, where C_balloon is the coordinate of the center of the balloon cross-section, C_wire is the coordinate of the center of the guidewire cross-section, and R_balloon is the radius of the balloon; the eccentricity is used to evaluate the centering of the guidewire within the balloon and correct for rotation center deviation in the OCT image. Specifically, it includes the following sub-steps: S41: The OCT probe, during the retraction process in step S3, will operate at a frequency... The camera flashes periodically. Simultaneously, the binocular camera also flashes at a similar frequency. Image acquisition is performed to ensure that two frames are captured within each complete "bright-dark" cycle: one frame corresponds to the probe being lit and the other frame corresponds to the probe being off, thus obtaining an image pair of alternating bright and dark images at the probe end. Figure 4 A and Figure 4 In the diagram, B represents the bright and dark frames of the left camera. Due to the slow retraction speed and short image acquisition interval, the physical displacement of the OCT probe in space between adjacent frames is negligible, meaning its position is considered essentially unchanged. At this point, the only significant difference between the two frames lies in whether the probe tip emits light. This characteristic provides ideal conditions for subsequent feature point extraction using the differential method, effectively suppressing background texture interference and motion blur.

[0031] S42: Left graph for time t I L ( t ) and the right figure I R ( t ), and the left image at the next moment. I L ( t+1 ) and the right figure I R ( t+ 1 The image is stereo-corrected using the intrinsic and extrinsic parameters obtained in step S1 to obtain the corrected left image. RI L ( t The corrected right figure RI R ( t (and the left image at the next corrected moment) RI L ( t+1 The right figure after correction at the next moment. RI R ( t+1 ).

[0032] S43: Since the only difference between the two frames is the bright spot at the probe location, the position of the probe, i.e., the bright spot at time t in the left image, can be obtained using a formula. ,like Figure 4 As shown in C:

[0033] The operation represents taking the absolute value of the pixel-by-pixel difference between two images. Similarly, the probe's position information in the right image... It can be obtained from the following formula:

[0034] By setting a threshold θ, morphological opening operations and connected component analysis are performed on the difference image to extract the centroid of the largest connected region, thus accurately locating the pixel coordinates of the bright spot in the left and right images. It should be noted that the threshold θ can range from [50, 150], based on an 8-bit grayscale image; in this embodiment, θ = 80 is used. A threshold that is too small will introduce noisy bright spots, while a threshold that is too large will cause the true bright spot to be filtered out.

[0035] S44: Since the image has undergone stereo correction, the center point of the bright spot is in the same row of the left and right images. The bright spot can be obtained using the formula, which is the parallax of the OCT probe at time t. d :

[0036] in, This indicates that the center of the bright spot is on the horizontal axis of the left image. The center of the bright spot is indicated by the x-coordinate in the right figure; the depth value of the OCT probe at time t is further obtained using the formula. Z :

[0037] in, To correct the focus of the camera, bs This serves as the camera baseline.

[0038] Since the system uses the left camera coordinate system as the world coordinate system, the coordinates of the probe in the world coordinate system at time t can be directly obtained using a formula. :

[0039]

[0040] in, The pixel coordinates of the center of the highlight in the left image. Let be the coordinates of the principal point of the left camera. Therefore, the position of the OCT probe at each time t can be mapped to a three-dimensional spatial point, constituting its instantaneous pose during its motion within the balloon.

[0041] S45: By sampling the entire pullback process at high frequency, all moments are... 3D coordinates The data is sorted and interpolated to generate a continuous and smooth spatial trajectory curve, which represents the actual path of the OCT probe within the cavity. Figure 5 As shown in the figure. This trajectory not only reflects the stability of the guidewire advancement, but can also be used to analyze its deviation from the central axis of the balloon.

[0042] S5: Replace the OCT probe with a photoacoustic probe to perform a pullback scan, reconstruct a three-dimensional photoacoustic image, and simultaneously detect the balloon marker; specifically including the following sub-steps: S51: After completing the OCT scan and withdrawing the probe, the photoacoustic imaging probe is inserted into the target lesion area via a balloon guidewire. Subsequently, the automatic retraction device is activated, and the photoacoustic imaging probe is withdrawn at a constant speed to achieve continuous axial sampling. All acquired photoacoustic cross-sectional images are stacked in spatial order to construct three-dimensional photoacoustic volume data of the lesion area.

[0043] S52: Detect the angular position of the busbar shadow in the photoacoustic cross-sectional image using an image processing algorithm.

[0044] S53: After the scan is completed, the axial length covered by the photoacoustic scan in this operation by the control system is recorded as L. PAI .

[0045] S6: Based on common marker features and system parameters, achieve 3D registration and information fusion of binocular vision, OCT, and photoacoustic images; specifically including the following sub-steps: S61: Convert the 3D volumetric structure obtained from OCT and photoacoustic imaging into point cloud format. Set a grayscale threshold, identify voxels with grayscale values ​​below this threshold as cavity regions, and extract their boundary contours. Use a 3D edge detection algorithm to convert the surface structure of the cavities into 3D point cloud data.

[0046] S62: During actual scanning, the actual working distance L between the OCT probe and the photoacoustic detector is... OCT L PAI Deviation will occur due to guidewire offset. Using the eccentricity parameter calculated in step S4, combined with the physical expansion model of the balloon, the L value fed back by the motor is... OCT L PAI Spatial position compensation is performed to correct the actual scanning starting position and axial distribution of OCT and photoacoustic scans in the world coordinate system. After correction, the OCT and photoacoustic images are spatially aligned in the direction of catheter advancement (i.e., axial direction), ensuring that the two have a consistent anatomical correspondence in the longitudinal position.

[0047] S63: By detecting the angular position of the busbar shadow in the two modal images respectively, calculating its angular deviation in the cross section, and then performing rotation correction on the OCT or photoacoustic image, the two modalities are synchronized in the circumferential angle.

[0048] S64: Due to the laser marking in step S1, both the binocular image and the OCT image have obvious feature points. A point-to-point registration algorithm such as ICP is used to calculate the transformation matrix of the OCT image relative to the binocular visual coordinate system. RtThis achieves high-precision spatial alignment between OCT and binocular images. Since the photoacoustic image has been aligned with the OCT through sub-steps S62 and S63, the photoacoustic data can be indirectly mapped to the binocular coordinate system, ultimately achieving complete registration of the OCT, photoacoustic, and binocular vision trimodal images in a unified three-dimensional space.

[0049] S65: Based on the completion of multimodal spatial alignment, high-resolution functional and structural information in OCT and photoacoustic images are mapped to the three-dimensional reconstructed surface area of ​​binocular vision to achieve information fusion and visualization.

[0050] See Figure 6 The present invention also provides an apparatus for three-dimensional registration of intracavity optical multimodal images, including one or more processors, for implementing the method for three-dimensional registration of intracavity optical multimodal images in the above embodiments.

[0051] The embodiments of the device for three-dimensional registration of intracavity optical multimodal images of the present invention can be applied to any device with data processing capabilities, such as a computer. The device embodiments can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 6 The diagram shown is a hardware structure diagram of any device with data processing capabilities, including the apparatus for three-dimensional registration of intracavity optical multimodal images according to the present invention. Except for... Figure 6 In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0052] It will be understood by those skilled in the art that the above descriptions are merely preferred examples of the invention and are not intended to limit the invention. Although the invention has been described in detail with reference to the foregoing examples, those skilled in the art can still modify the technical solutions described in the foregoing examples or make equivalent substitutions for some of the technical features. All modifications and equivalent substitutions made within the spirit and principles of the invention should be included within the scope of protection of the invention.

Claims

1. A method for three-dimensional registration of intracavity optical multimodal images, characterized in that, Includes the following steps: S1: Calibrate the multimodal endoscopic system and construct a unified world coordinate system; S2: Laser marking is performed on the surface of tissue adjacent to the lesion, and the three-dimensional morphology of the lesion area is reconstructed based on binocular vision; S3: Advance the balloon catheter to the target lesion segment, inflate the balloon, insert the OCT probe along the guidewire into the balloon, perform a retraction scan, and detect the first balloon marker; record the axial length L covered by the OCT scan. OCT ; S4: The periodic flashing point at the tip of the OCT probe is identified by a binocular camera, and the retraction trajectory of the OCT probe is calculated in real time to obtain the eccentricity between the balloon and the guidewire. S5: Replace the OCT probe with a photoacoustic probe to perform a retraction scan, reconstruct a three-dimensional photoacoustic image, and simultaneously detect the second balloon marker; record the axial length L covered by the photoacoustic scan. PAI ; S6: Based on the eccentricity between the balloon and the guidewire, for L OCT L PAI Spatial position compensation is performed to correct the actual scanning starting position and axial distribution of OCT and photoacoustic scans in the world coordinate system; based on the first and second balloon markers, rotation correction is performed on the OCT or photoacoustic images to achieve synchronous alignment of the two modalities in the circumferential angle; the transformation matrix of the OCT image relative to the binocular vision coordinate system is calculated to achieve spatial alignment of the OCT and binocular images; the photoacoustic data is mapped to the binocular vision coordinate system to achieve registration of the OCT, photoacoustic and binocular vision trimodal images in a unified three-dimensional space.

2. The method for three-dimensional registration of intracavity optical multimodal images according to claim 1, characterized in that, Step S1 includes the following sub-steps: S11: Place the checkerboard calibration plate at the exit of the binocular endoscope clamp channel and adjust the spatial orientation of the checkerboard calibration plate so that it is perpendicular to the optical axis of the lens. S12: Collect several sets of synchronized left and right image pairs at different angles and distances on the chessboard calibration board; S13: Calculate the intrinsic parameter matrix, distortion coefficients, rotation matrix, and translation vector of the stereo camera based on the acquired images; S14: Set the left camera coordinate system to the world coordinate system, and the extension direction of the pincer is the positive Z-axis direction.

3. The method for three-dimensional registration of intracavity optical multimodal images according to claim 2, characterized in that, Step S2 includes the following sub-steps: S21: After confirming the location of the lesion, a laser is projected onto the surface of the tissue adjacent to the lesion to form an artificial feature marker; S22: Acquire left and right eye images within the cavity using a binocular camera; perform distortion correction on the left and right eye images within the cavity based on the intrinsic parameter matrix and distortion coefficients; calculate the stereo correction transformation using the rotation matrix and translation vector. S23: Dense stereo matching and 3D reconstruction are performed using a semi-global block matching algorithm to obtain a dense 3D point cloud of tissue adjacent to the lesion.

4. The method for three-dimensional registration of intracavity optical multimodal images according to claim 3, characterized in that, In sub-step S21, the tissue adjacent to the lesion refers to the normal tissue area within 2-5 mm beyond the edge of the lesion; the laser is projected through a semiconductor laser emitter integrated into the sidewall of the duct, with a wavelength of 532 nm and a power of <50 mW.

5. The method for three-dimensional registration of intracavity optical multimodal images according to claim 1, characterized in that, Step S3 includes: The balloon catheter is pushed to the target lesion segment, and sterile saline is injected through the catheter lumen using an external pressure pump to slowly inflate the balloon until the outer wall of the balloon evenly adheres to the inner wall of the lumen. After inserting the OCT probe into the balloon, the lumen is scanned by retraction, and continuous OCT slices of each cross section and the three-dimensional reconstruction results composed of each OCT slice are returned. A marker generatrix is ​​integrated axially on the outer surface of the balloon, extending continuously along the length of the balloon and parallel to its axis. When the OCT probe retracts within the balloon, this generatrix appears as a shadow signal in each frame of the OCT cross-sectional image. Image processing algorithms automatically identify this generatrix shadow and extract its angular position in the OCT cross-sectional image. The axial length L covered by the OCT scan is recorded. OCT .

6. The method for three-dimensional registration of intracavity optical multimodal images according to claim 1, characterized in that, During the retraction of the OCT probe, at a frequency Periodic flashes; simultaneously, the binocular camera flashes at a frequency Image acquisition is performed to ensure that two frames are captured within each complete bright-dark cycle, one frame corresponding to the OCT probe being lit and the other frame corresponding to the OCT probe being off, thus obtaining an image pair of alternating bright and dark images at the OCT probe end.

7. The method for three-dimensional registration of intracavity optical multimodal images according to claim 1, characterized in that, In step S4, the eccentricity ε between the balloon and the guidewire is ε = |C_balloon - C_wire| / R_balloon, where C_balloon is the coordinate of the center of the balloon cross-section, C_wire is the coordinate of the center of the guidewire cross-section, and R_balloon is the radius of the balloon. The eccentricity is used to evaluate the centering of the guidewire within the balloon and correct the rotation center deviation of the OCT image.

8. The method for three-dimensional registration of intracavity optical multimodal images according to claim 5, characterized in that, Step S5 includes: After completing the OCT scan and withdrawing the probe, the photoacoustic imaging probe is advanced into the target lesion area via the balloon guidewire. An automatic retraction device is then activated, retracting the photoacoustic imaging probe at a constant speed to achieve continuous axial sampling. All acquired photoacoustic cross-sectional images are stacked in spatial order to construct three-dimensional photoacoustic volume data of the lesion area. Image processing algorithms are used to detect the angular position of the balloon generatrix shadow in the photoacoustic cross-sectional image. The axial length L covered by the photoacoustic scan is recorded. PAI .

9. The method for three-dimensional registration of intracavity optical multimodal images according to claim 8, characterized in that, Step S6 includes the following sub-steps: S61: Convert the volumetric 3D structure of OCT and photoacoustics into point cloud form; identify voxels with gray values ​​below the gray threshold in the image as cavity regions and extract their boundary contours; The surface structure of the cavity is converted into three-dimensional point cloud data using a three-dimensional edge detection algorithm; S62: Based on the eccentricity between the balloon and the guidewire, combined with the physical expansion model of the balloon, the axial length L covered by the OCT scan is... OCT axial length L covered by photoacoustic scanning PAI Spatial position compensation is performed to correct the actual scanning starting position and axial distribution of OCT scan and photoacoustic scan in the world coordinate system, so that the OCT image and photoacoustic image are spatially aligned in the direction of catheter advancement. S63: Detect the angular position of the balloon generatrix shadow in the two modal images respectively, calculate its angular deviation in the cross section, and then perform rotation correction on the OCT image or photoacoustic image so that the two modalities can be synchronously aligned in the circumferential angle. S64: Employs a point-to-point registration algorithm to calculate the transformation matrix of the OCT image relative to the binocular vision coordinate system, achieving high-precision spatial alignment between the OCT image and the binocular image; maps photoacoustic data to the binocular vision coordinate system, achieving registration of OCT, photoacoustic, and binocular vision trimodal images in a unified three-dimensional space; S65: Maps high-resolution functional and structural information from OCT and photoacoustic images to the 3D reconstructed surface area of ​​binocular vision, achieving information fusion and visualization.

10. A device for three-dimensional registration of intracavity optical multimodal images, characterized in that, It includes one or more processors for implementing the method for three-dimensional registration of intracavity optical multimodal images according to any one of claims 1-9.

Images