Calibration of imaging systems with combined optical coherence tomography and visualization modules

Abstract

An imaging system (10, 100, 200) includes a housing assembly (20) having a head unit (18) configured to be at least partially directed toward a target site (16, 116, 216). An optical coherence tomography (OCT) module (14, 114, 214) and a visualization module (12, 112, 212) are located in the housing assembly (20) and are configured to obtain OCT data and visualization data of a target site (16, 116, 216), respectively. The system (10, 100, 200) includes a controller (C) configured to generate a scan pattern for a calibration region selected in a calibration target. And synchronously acquiring OCT data of the calibration area. The controller (C) is configured to obtain, based on the OCT data, a projected two-dimensional OCT image of the calibration region as an average intensity inverse projection. The controller (C) is configured to register the projected two-dimensional OCT image to a corresponding view extracted from the visualization data via a cascaded image registration process having a coarse registration phase and a fine registration phase.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority to U.S. Provisional Application No. 63 / 391,235, filed on July 21, 2022, which is hereby incorporated by reference in its entirety. Technical Field

[0003] The present disclosure relates to the calibration of an imaging system having a combined optical coherence tomography module and a visualization module. Various imaging modalities are commonly used throughout the world to image various parts of the human body. In some cases, the imaging modalities can be combined. For example, an optical coherence tomography (OCT) unit can be integrated with a visualization device (such as a camera) to provide better guidance during surgery. In order to provide accurate, real-time and three-dimensional perceptual guidance in such an integrated system, the two imaging modalities should be well aligned. However, there are some challenges in aligning images from the OCT unit with other imaging modalities, including the complexity of mechanical scanning in a point scanning OCT system. In addition, manual calibration and semi-automatic calibration can be time-consuming and cumbersome. For example, manual calibration requires experienced technical support personnel to run various step-by-step calibration procedures. Summary of the invention

[0004] An imaging system is disclosed herein, the imaging system having a housing assembly having a head unit, the head unit being configured to be at least partially directed toward a target site. An optical coherence tomography (OCT) module and a visualization module are located in the housing assembly and are configured to obtain OCT data and visualization data of the target site, respectively. A controller communicates with the OCT module and the visualization module. The controller has a processor and a tangible non-transitory memory having instructions for a calibration method recorded thereon.

[0005] The controller is configured to generate a scan pattern for a calibration area selected in a calibration target in a coordinate system of the visualization module (referred to as visualization space). The OCT data of the calibration area is collected synchronously with the scan pattern. The controller is configured to obtain a projected two-dimensional OCT image of the calibration area based on the OCT data. The projected two-dimensional OCT image is an average intensity back projection on a frontal viewing plane. The projected two-dimensional OCT image is superimposed with a corresponding view extracted from the visualization data. The controller is configured to register the projected two-dimensional OCT image to the corresponding view via a cascaded image registration process having a coarse registration stage and a fine registration stage.

[0006] In one embodiment, the visualization module is a surgical microscope. The visualization module can be a stereo camera, wherein the second set of data includes a first view and a second view of the target site. The target site can be an eye. Before aligning the projected two-dimensional OCT image with the corresponding view extracted from the visualization data, the controller can be configured to perform automatic image resizing. The scanning mode can be an orthogonal raster scanning mode. The robotic arm can be operably connected to the head unit and configured to selectively move the head unit. The robotic arm can be selectively operated to expand the observation range of the OCT module in three dimensions.

[0007] The controller may be adapted to calculate corresponding transformation parameters of the coarse registration stage based on a translation transformation matrix that compensates for the shift mismatch. In one embodiment, the controller is adapted to calculate corresponding transformation parameters of the fine registration stage based on an affine diffusion tensor image (DTI) registration, wherein the corresponding transformation parameters are based in part on a rotation matrix, a shear matrix, and a scaling matrix.

[0008] The controller may be adapted to compensate for relatively small shift mismatches introduced during at least one of rotation, shear, and scale alignment operations. The rotation matrix may be represented as Where θ is the corresponding transformation parameter of the rotation. The shear matrix can be expressed as Where Shx and Shy are the shearing parameters along the first lateral direction and the second lateral direction. The scaling matrix can be expressed as Wherein, Cx and Cy are scaling parameters along the first lateral direction and the second lateral direction.

[0009] In one embodiment, the controller is adapted to selectively perform a verification procedure, wherein the controller is adapted to select a region of interest in the visualization space and obtain corresponding voltages for the OCT scan based in part on the region of interest and corresponding transformation parameters. The verification procedure includes obtaining an acquired OCT image based on these corresponding voltages and comparing the acquired OCT image with the region of interest. The region of interest can be a line in the visualization space corresponding to a cross-sectional B-frame in the OCT space. The region of interest can be a quadrilateral in the visualization space corresponding to a three-dimensional volume in the OCT space.

[0010] Disclosed herein is a method for calibrating an imaging system having a housing assembly and a controller having a processor and a tangible non-transitory memory. The method includes placing an optical coherence tomography (OCT) module and a visualization module in the housing assembly to obtain OCT data and visualization data of a target site, respectively. The head unit is at least partially directed to the target site. The method includes generating a scan pattern for a calibration area selected in a calibration target in a coordinate system (referred to as visualization space) of the visualization module. The OCT data of the calibration area is collected synchronously with the scan pattern.

[0011] The method includes obtaining a projected two-dimensional OCT image of the calibration area based on the OCT data. The projected two-dimensional OCT image is an average intensity back projection on a frontal viewing plane. The projected two-dimensional OCT image is superimposed with a corresponding view extracted from the visualization data. The method includes registering the projected two-dimensional OCT image to the corresponding view via a cascaded image registration process having a coarse registration phase and a fine registration phase.

[0012] The above features and advantages and other features and advantages of the present disclosure are readily apparent from the following detailed description of the best modes for carrying out the present disclosure when taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a schematic partial stereogram of an imaging system having an optical coherence tomography (OCT) module, a visualization module, and a controller;

[0014] Figure 2 According to the first embodiment Figure 1 a schematic diagram of a portion of a system;

[0015] Figure 3 According to the second embodiment Figure 1 a schematic diagram of a portion of a system;

[0016] Figure 4A and Figure 4B yes Figure 1 A schematic partial stereogram of an example scanning area of ​​an OCT module;

[0017] Figure 4C yes Figure 1 A schematic partial top view of an example scanning mode of an OCT module;

[0018] Figure 5 It can be Figure 1 A flowchart of an example method performed by a controller;

[0019] Figure 6 yes Figure 5 a flow chart of a portion of a method;

[0020] Fig. 7A It can be Figure 1 A schematic diagram of an example calibration target employed by a system of FIG.

[0021] Figure 7B yes Fig. 7A A schematic diagram of a captured image of a calibration target;

[0022] Figure 8 yes Figure 1 A schematic diagram of an example target site of a system of FIG. 1 ; and

[0023] Fig. 9 is through Figure 5 Example graph of a calibrated scan pattern obtained using the method.

[0024] Representative embodiments of the present disclosure are shown in the accompanying drawings by way of non-limiting examples and are described in more detail below. However, it should be understood that the novel aspects of the present disclosure are not limited to the specific forms shown in the drawings enumerated above. On the contrary, the present disclosure will cover modifications, equivalents, combinations, sub-combinations, permutations, groupings, and alternatives that fall within the scope of the present disclosure, such as those covered by the appended claims. DETAILED DESCRIPTION

[0025] Referring to the drawings, wherein like reference numerals refer to like parts, Figure 1 An imaging system 10 having a visualization module 12 and an optical coherence tomography module 14 (hereinafter referred to as "OCT module 14") is schematically shown. The imaging system 10 (hereinafter referred to as "system 10") is configured to image a target site 16. In some embodiments, the visualization module 12 is a stereo camera configured to record a first image and a second image of the target site 16, which can be used to generate a real-time two-dimensional stereoscopic view of the target site 16. The visualization module 12 can be a digital surgical microscope system integrated with the OCT module 14. In other embodiments, the visualization module 12 can be a single camera. It should be understood that the system 10 can take many different forms and include multiple components and facilities and / or alternative components and facilities.

[0026] refer to Figure 1 , the visualization module 12 can be located in the head unit 18 of the housing assembly 20, wherein the head unit 18 is configured to be at least partially directed toward the target site 16. The target site 16 can be an anatomical location of a patient, a laboratory biological sample, a calibration slide / template, etc. In one example, reference Figure 8 , the target site 16 is an eye 600 having a pupil 602 , an iris 604 , and a sclera 606 .

[0027] System 10 provides a target region of interest (e.g. Figure 8 The system 10 can obtain an accurate OCT scan of the region of interest 670 in the OCT module 14, which is co-registered with the view from the visualization module 12, thereby providing valuable information during ophthalmic surgery, such as layer structure, tissue thickness, incision, position or size of the bracket and intraocular lens, etc. As described below, the controller C in the system 10 is adapted to perform cascaded image registration and obtain a transformation matrix connecting the corresponding spaces (i.e., coordinate systems) of the OCT module 14 and the visualization module 12.

[0028] refer to Figure 1 , at least one input device 22 (hereinafter "at least one") is operatively connected to the visualization module 12 (e.g., at the head unit 18) to allow a user to manually position the visualization module. The input device 22 may include corresponding controls for activating or selecting specific features, such as focusing, zooming in, adjusting the amount / type of light projected onto the target area 16, and other features. It should be understood that the number and form of the input device 22 can be varied. For example, the input device 22 may include a joystick, wheel, mouse, or touch screen device. In some embodiments, the input device 22 can be controlled via a remote control unit 23 (see Figure 1 ).

[0029] In some embodiments, the system 10 may include a robotic arm 24 operatively connected to the head unit 18 and configured to selectively move the head unit. Figure 1 , the robot arm 24 can be selectively operated to extend the viewing range of the OCT module 14 along the X direction, the Y direction, and the Z direction. Figure 1 , the head unit 18 can be mechanically coupled to the robotic arm 24 via a coupling plate 26. The robotic arm 24 can include one or more joints, such as a first joint 30 and a second joint 32, which are configured to provide further positioning and / or orientation of the head unit 18. Figure 1 , a corresponding joint motor (such as joint motor 31) and a corresponding joint sensor (such as joint sensor 33) can be connected to each joint. The joint motor 31 is configured to rotate the first joint 30 around the axis, and the joint sensor 33 is configured to transmit the position of the first joint 30 (in 3D space).

[0030] The head unit 18 may be connected to a cart 34 having at least one display medium (which may be a monitor, terminal, or other form of two-dimensional visualization), such as Figure 1 The first display 36 and the second display 38 are shown. The housing assembly 20 may be independent and movable between different positions. Return Figure 1, the first display 36 can be connected to the cart 34 via a flexible robotic arm 40 having one or more joints to enable flexible positioning. The flexible robotic arm 40 can be configured to be long enough to extend above the patient during surgery, thereby providing a relatively close viewing for the surgeon. The first display 36 and the second display 38 can include any type of display, including a high-definition television, an ultra-high-definition television, smart glasses, a projector, one or more computer screens, a laptop computer, a tablet computer, and / or a smart phone, and can include a touch screen.

[0031] refer to Figure 1 The system 10 includes a controller C having at least one processor P and at least one memory M (or a non-transitory tangible computer-readable storage medium) having recorded thereon a method 400 for executing the operation calibration mode 50 (hereinafter referred to as Figure 5 The memory M may store a controller executable instruction set, and the processor P may execute the controller executable instruction set stored in the memory M. The OCT module 14 and the visualization module 12 may include an integrated processor that communicates with the controller C.

[0032] refer to Figure 1 , the controller C may be configured to process signals for broadcasting on the first display 36 and the second display 38. The first display 36 and the second display 38 may include a stereoscopic display system, in which a two-dimensional display has separate images for the left eye and the right eye, respectively. To view the stereoscopic display, the user may wear special glasses that are used in conjunction with the first display 36 and the second display 38 to display a left view to the user's left eye and a right view to the user's right eye.

[0033] The visualization module 12 is configured to collect images of the target area 16, which can be presented in different forms, including but not limited to captured static images, real-time images and / or digital video signals. "Real-time" as used herein generally refers to updating information at the same rate as the received data. More specifically, "real-time" refers to collecting, processing and transmitting image data at a sufficiently high data rate and a sufficiently low delay so that when the data is displayed, the object moves smoothly without jitter or delay noticed by the user. Typically, this occurs when new images are collected, processed and transmitted at a rate of at least about 30 frames per second (fps) and displayed at about 60fps and when the delay of the combined processing of the video signal does not exceed about one-thirtieth of a second.

[0034] The system 10 calibrates the respective spaces or coordinate systems (OCT coordinate space and visualization coordinate space) of the two imaging modalities from the perspective of the acquired images (i.e., end-to-end calibration). This covers mismatches from multiple sources including mechanical scanning, imaging optics, and mismatches due to the surgical environment. The processing algorithms in the calibration are much faster than those achieved through 2D galvo motion simulation, manual calibration, or semi-automatic calibration.

[0035] System 10 does not require accurate image segmentation, which reduces the burden of additional image processing. As described below, method 400 is used to obtain a specific working distance W (see Figure 1 ) and / or OCT calibration at a specific zoom ratio can be automatically completed by clicking a button, where all data acquisition, post-processing and matrix calculations are performed by the controller C. In the case where the visualization module 12 and the OCT module 14 operate at various zoom ratios, a list of repeated calibrations at multiple depths can be performed and stored in a lookup table.

[0036] The system 10 performs the following steps by processing an image (e.g., Fig. 7A The calibration target 500 shown in FIG. 1 is used for step-by-step automatic resizing and translation registration, using intensity-based cascaded multimodal image registration. Registration can be understood as the alignment of the (multiple) views of the visualization module 12 with the (multiple) views of the OCT module 14 in terms of rotation, translation and scaling, as well as the matching of various perspectives. The controller C is adapted to superimpose the OCT section B scan with the corresponding front view area in the image obtained through the visualization space using the calibration parameters (i.e., verification of the calibration).

[0037] The system 10 can employ an affine diffusion tensor for image registration. After image registration, calibration parameters (e.g., rotation, shear, scaling, and shift parameters) are summarized as an affine matrix in a homogeneous coordinate space, which can be further used to transform any region of interest in a visualization space (e.g., camera space) into a well-registered OCT scan pattern. This allows simultaneous display of depth-resolved cross-sections (registered) of the target site 16. Affine transformation is a linear mapping method that preserves points, lines, and planes.

[0038] Reference now Figure 2 , according to a first embodiment, a portion of a system 10 is shown. Figure 2 The device 100 includes an exemplary visualization module 112 that is integrated with an OCT module 114 through a shared objective lens 102. The device 100 includes a dichroic mirror 104. The visualization module 112 may include the dichroic mirror 104, a set of magnification / focusing optics 106, and a high-resolution two-dimensional camera 108 for imaging a target site 116.

[0039] refer to Figure 2 , the OCT module 114 includes a beam expander optical device 120, a two-dimensional scanner 122, a collimator 124, and an OCT engine 126. The two-dimensional scanner 122 can be an XY galvanometer scanner group, a resonant scanner group, a micro-electromechanical system (MEMS) scanner, or other scanners capable of performing raster, orthogonal, or equivalent two-dimensional beam scanning. The galvanometer scanner (also known as a galvanometer optical scanner) includes a motorized mirror mount for laser beam steering or scanning applications. The OCT engine 126 can be spectral domain OCT, swept source OCT, or time domain OCT using spot scanning or point detection techniques.

[0040] refer to Figure 2 , the target site 116 is laterally scanned by a first light beam B1 originating from the OCT module 14. The scanning area of ​​the first light beam B1 at least partially overlaps with the second imaging path B2 at the target site 116 (which is illuminated by a light source alone and imaged by a camera in the visualization module 112). Optical fiber can be used to transmit and / or guide the first light beam B1 and guide it to fall on an appropriate region of interest in the target site 116 in the form of a point scan. The device 100 may include any suitable additional optical or mechanical components available to those skilled in the art for manipulating the light beam and automatically adjusting. The scanned OCT data set (which may be three-dimensional or two-dimensional) and image of the target site 116 can be sent to the controller C.

[0041] Reference now Figure 3 , according to a second embodiment, a portion of a system 10 is shown. Figure 3 The device 200 includes an exemplary visualization module 212 that is integrated with an OCT module 214 via a shared objective lens 202. The device 200 includes a dichroic mirror 204. Figure 3 The visualization module 212 includes two zoom optical device groups 206A, 206B (for beams B2 and B3, respectively), two beam splitters 208A, 208B, two sets of magnification / focusing optical devices 210A, 210B, and two high-resolution two-dimensional cameras 218A, 218B for imaging the target site 216.

[0042] In this embodiment, the visualization module 212 is a binocular surgical microscope. Figure 3 The visualization module 212 includes two eyepieces 230A and 230B, which are connected to the lens tube groups 232A and 232B, respectively. By transmitting the images from the dual cameras 218A and 218B to the first display 36 and the second display 38 (see Figure 1 ), these images can be combined into a heads-up 3D visualization system. This allows ophthalmologists to replace the eyepieces of surgical microscopes with high-resolution stereo cameras. Figure 3As shown, the OCT module 214 may include a beam expander optic 220 (for beam B1 ), a two-dimensional scanner 222 , a collimator 224 , and an OCT engine 226 .

[0043] FIG. 4A to FIG. 4C 2 shows an example scanning area that can be used for the OCT modules 114, 214. It should be understood that FIG. 4A to FIG. 4C The views shown in are intended as non-limiting examples. Figure 4A and 4B is a schematic partial stereogram, and Figure 4C is a schematic partial top view of an example scanning pattern. Figure 4A , a single scan of the point scan S (of the target site 116, 216) produces a depth scan 302 of the physical sample structure into which the light beam (e.g., light beam B1) originating from the OCT module 114, 214 is directed along the incident direction. Figure 4A , the depth scan 302 may be referred to as an “A-scan” and is configured to scan along an axial direction A to a detection depth 304 .

[0044] refer to Figure 4A , Figures 2 to 3 The light beam B1 in the A-scanning system can be moved in a continuous manner around the target site 116, 216 to achieve a second depth scan 306, a third depth scan 308, a fourth depth scan 310, and a fifth depth scan 312 along the first transverse scanning range 314. Such an A-scan line can be referred to as a B-scan or a line scan 316. Figure 4C By appropriately controlling the optical path along the first lateral scanning range 314 and then performing a "step and repeat" path control along the raster pattern 318 to repeat the cycle at the starting point 320 and subsequent rows, a depth scanning grid can be drawn along the first lateral scanning range 314 and the second lateral scanning range 322. Figure 4B , which results in a three-dimensional sampling volume 324, which may have the shape of a cuboid.

[0045] The movement of beam B1 and the processing of each A-scan (e.g., the second depth scan 306, the third depth scan 308, the fourth depth scan 310, and the fifth depth scan 312) can be synchronized with the rest of the system 10 via the controller C and / or the OCT engines 126, 226 so that downstream processes can reassemble the scans in the same order and relative positions during the reconstruction process.

[0046] refer to Figure 4C , the diameter of the spot scan S can be represented by a first set of dimensions 340 and 342. In one example, the first set of dimensions 340 and 342 are equal and in the range of about 15 μm to 80 μm. In other examples, they are not equal. Figure 4C, the spacing between adjacent A-scans or depth scans 302 can be represented by a second set of dimensions 344 and 346.

[0047] Reference now Figure 5 , shows the operation for Figure 1 Flowchart of an example method 400 for the calibration mode 50 of FIG. 400 may be implemented as a memory device stored in FIG. Figure 1 The method 400 is a computer readable code or instruction on a controller C and partially executable by the controller. The method 400 does not need to be applied in the specific order described herein and can be executed dynamically. In addition, it should be understood that some steps can be eliminated. The method 400 can be executed periodically or at predetermined time intervals.

[0048] according to Figure 5 At block 402 , the controller C is programmed to select a region of calibration (ROC) for a calibration target in a visualization space (eg, camera coordinate space). Fig. 7A An example calibration target 500 that can be employed by the system 10 is shown. The calibration target 500 can be a low-reflection or high-reflection grid point distortion target. The calibration target 500 can include a plurality of dots 502 of various sizes arranged in a regular fixed frequency grid. In one embodiment, the calibration target 500 is a "3×3" low-reflection grid point distortion target with a dot size of 0.25 mm. Figure 7B is a schematic diagram of a captured image 550 of a calibration target 500 illustrating an example calibration area 560 .

[0049] From block 402, method 400 proceeds to blocks 404 and 406. According to block 404, a corresponding image (e.g., Figure 7B 550 in the calibration target 500) and streams it to the controller C. According to box 406, the method 400 includes generating a scan pattern for the calibration area 560 selected in the calibration target 500. Depending on the galvanometer field of view of the calibration area 560, the scan pattern can be a raster orthogonal scan pattern. In a raster scan, the light beam can be swept horizontally from a first end to a second end at a steady rate, then blank and quickly move back to the first end, where it starts again and sweeps out the next line. During this period, the vertical position is steadily increased at a slower rate, where one vertical sweep is performed for each image frame and one horizontal sweep is performed for each line of resolution.

[0050] Continue to Figure 5In block 408, in synchronization with the scanning, the controller C is programmed to acquire OCT raw data of the scanned area. A two-dimensional or three-dimensional scanned OCT data set and corresponding images of the calibration target 500 from the visualization module 12 can be sent to the controller C. According to various scanning modes, the scanned OCT data set can be a 3D volume or a two-dimensional B frame including a sequentially scanned A-scan (depth scan).

[0051] from Figure 5 The block 408 proceeds to block 410 where the raw OCT data is post-processed through the OCT reconstruction pipeline, including background removal, spectral windowing, dispersion compensation, fast Fourier transform, and logarithmic compression to form a depth-resolved 3D volume. Although some processing techniques are described above, it should be understood that other techniques may also be used. The 3D OCT volume is further processed by back-projecting the mean intensity onto the frontal view plane as the XY plane (see Figure 1 ) in a projection view or OCT image. The frontal view plane can be defined as the face or frontal facing forward. Figure 7B An example superimposed frontal view display is shown. Projected 2D OCT image (by Figure 7B ) is superimposed with a corresponding view extracted from the visualization data (represented by the first set of points 552).

[0052] From blocks 404 and 410, method 400 proceeds to block 414, as indicated by line 412. Figure 5 At block 414, controller C may be programmed to perform automatic image resizing. Since the two images are from different imaging modalities, have different pixel sizes and different sample counts, it may be necessary to resize the image based on the repetitive pattern being imaged (e.g., Fig. 7A The horizontal and vertical peak counts of the dot pattern in the calibration target 500 shown in FIG. 5 are automatically adjusted to fit the two images into the same graphical user interface (GUI) widget.

[0053] Go to Figure 5 At block 416, the method 400 includes finding a detailed correlation or transformation between the corresponding spaces (or coordinate systems) of the visualization module 12 and the OCT module 14 through a cascaded image registration process. In the illustrated embodiment, the corresponding image from the visualization module 12 may be considered as a fixed image, while the corresponding image from the OCT module 14 may be considered as a moving image to be registered with the fixed image.

[0054] As mentioned above, Figure 7B The first set of points 552 (unshaded) in FIG. 5 represents the corresponding image of the calibration target 500 in the visualization space, while the second set of points 554 (shaded) represents the average intensity back-projection of the raster scanned OCT volume. Figure 7BIn the example shown, there is a mismatch between the first set of points 552 and the second set of points 554 , which will be corrected by the cascaded image registration process in block 416 .

[0055] The cascaded image registration process includes a coarse registration stage and a fine registration stage. The coarse registration stage is based on a translation transformation matrix that corrects for relatively large shift mismatches. The fine registration stage is based on an affine diffusion tensor image (DTI) registration that handles mismatches in rotation, shear, and scaling. The transformation matrix can be expressed as:

[0056] The controller C is adapted to compensate for the relatively small displacements that may be introduced during changes in the first three deformations (rotation, shear and scale).

[0057] Considering relatively small shifts alone, the affine matrix can be expressed as: The affine matrix is ​​the product of the rotation matrix, the shear matrix, and the scale matrix. The rotation matrix can be expressed as The shear matrix can be expressed as And the scaling matrix can be expressed as Here, θ is the corresponding transformation parameter of the rotation, while Shx, Shy and Cx, Cy are the shearing parameters and scaling parameters along the first lateral direction T1 and the second lateral direction T2, respectively. The affine DTI transformation matrix (as shown in equation (1)) is decomposed into matrices describing rotation, shearing, scaling and shifting, which can be shown as in equation (2) below.

[0058]

[0059] Go to Figure 5 In block 418, the controller C is programmed to calculate the transformation parameters according to the matrix during the cascade image registration process. Since the transformation matrix of the coarse registration stage can be regarded as a subset of the affine matrix, for the combined two-step registration, if the two steps share the same transformation center 556 (see Figure 7B ), the result of the two-step registration can be summarized as a single affine matrix.

[0060] Go to Figure 5 At block 420, method 400 may optionally include executing a verification procedure, such as Figure 6 See details in the Reference Figure 6 , block 420 may include submodules 422, 424, and 426. Figure 6 In submodule 422, method 400 includes selecting a region of interest (ROI) in a corresponding space (e.g., camera space) of visualization module 12. In one embodiment, the region of interest is a line in visualization space corresponding to a cross-sectional B-frame in OCT space.

[0061] The surgeon can draw (and move or rotate) certain lines of interest on the camera image during or before surgery to obtain relevant depth information through the OCT module 14. Figure 8 In the example shown, the region of interest 670 in the eye 600 is a line indicating a region of the iridocorneal angle useful for glaucoma diagnosis.

[0062] In another embodiment, the region of interest can be a quadrilateral (e.g., a square) in the visualization space corresponding to the 3D volume in the OCT space. Using the transformation parameters obtained in block 418, the verification program can correct the mismatch through coordinate transformation and generate a new scan pattern (i.e., a voltage list for the scanner), which can follow the following equation.

[0063] Voltage = Affine -1 [Image Coordinates – Transformation 仿射 -center 仿射 ]+[Center 仿射 – Transform 平移 ].

[0064] Fig. 9 is through Figure 5 Example graph of a calibrated scan pattern obtained by method 400. Fig. 9 , the horizontal axis 702 indicates time, and the vertical axis 704 indicates amplitude. Trace 706 indicates that the B frame trigger signal has a flat line interrupted by regular spikes. Trace 708 and trace 710 indicate the corresponding pattern of the XY galvanometer scanner or the voltage conversion factor of the X-axis and Y-axis, respectively. Figure 5 The submodule 424 of the controller C is programmed to transform the coordinates to obtain the coordinates sent to the control scanner (e.g., Figure 2 The voltage of the two-dimensional scanner 122 in the OCT controller (eg, Figure 2 In other words, the location of the region of interest 670 (corresponding to Figure 7B Line 570 in captured image 550 can be converted to a new galvanometer voltage calibration line in OCT coordinate space.

[0065] Continue to Figure 6In submodule 426, controller C is programmed to compare the acquired OCT image with the region of interest 670 in the visualization space. The OCT data before and after calibration can be displayed side by side for verification and comparison. In one embodiment, this verification can be achieved by directly using the calibration parameters in real-time data acquisition. In another embodiment, this verification can be achieved by slicing / resampling the easily acquired OCT volume data. Image matching can be evaluated using various methods available to those skilled in the art (such as the Dice coefficient). For example, an evaluation score of 532 and a Dice coefficient of 0.96 may indicate a successful calibration (with a confidence level of 80%), compared to a low similarity of 0.08 before calibration.

[0066] In some embodiments, the operations of loading a camera image, demarcating a calibration area, loading an OCT image, automatically adjusting the image size, registration, and evaluation can be simplified to clicking a button. Depending on the size of the data, OCT data acquisition and projection to the frontal plane may take from a few seconds to a few minutes. The calibration process (e.g., pattern generation, image resizing, image registration, and coordinate transformation) may take a few seconds. If the optical alignment of the visualization module 12 and the OCT module 14 is above a predefined threshold, a semi-automatic pre-alignment proof can be performed to achieve a more robust calibration. For example, the pre-alignment step may involve using a mouse click on the camera image and the OCT projection image, respectively, to limit the registration boundaries. Depending on the environmental conditions in the operating room, the frequency of instrument use, and the stability of the visualization module 12 and / or the surgical environment, the calibration frequency can vary from a one-time manufacturing step to a daily routine.

[0067] Figure 1 The controller C may include or otherwise access information downloaded from a remote source and / or executable program. Figure 1 , the controller C can be configured to communicate with a remote server 60 and / or a cloud unit 62 via a network 64. The remote server 60 can be a private or public information source maintained by an organization such as a research institute, a company, a university, and / or a hospital. The cloud unit 62 can include one or more servers hosted on the Internet to store, manage, and process data.

[0068] The network 64 may be a serial communication bus in the form of a local area network. The local area network may include, but is not limited to, a controller area network (CAN), a controller area network with flexible data rate (CAN-FD), Ethernet, Bluetooth, WIFI, and other forms of data. The network 64 may be a wireless local area network (LAN) that links multiple devices using a wireless distribution method, a wireless metropolitan area network (MAN) that connects several wireless LANs, or a wireless wide area network (WAN) that covers a large area such as adjacent towns and cities. Other types of connections may be used.

[0069] In summary, system 10 is capable of automatically calibrating an optical coherence tomography (OCT) system to other imaging modalities, such as microscope camera imaging, to provide accurate 3D visualization during surgery. System 10 utilizes an intensity-based cascaded image registration process to achieve fully automatic OCT calibration. Controller C is adapted to obtain corresponding transformation parameters, including rotation, shear, scaling, and shift parameters. Affine diffusion tensor transformation is used to summarize the transformation relationship between OCT space and visualization space. The affine matrix can be further decoupled into rotation, shear, scaling, and shift matrices to further evaluate each type of mismatch. System 10 minimizes unnecessary manual involvement and improves efficiency and repeatability. In one embodiment, system 10 is used to calibrate the lateral scanning of paired XY galvanometer groups in a point scanning OCT system. OCT images can include A scans, B cross-sectional scans, and 3D volume scans.

[0070] System 10 provides technical advantages over other calibration methods that use feedback signals from integrated position detectors. The accuracy of the feedback signal can be adversely affected by environmental conditions such as temperature and humidity, electromagnetic interference noise, and high frequency (1-20kHz) jitter. In addition, the feedback signal provided by the position detector is typically an optical measurement, where the signal fluctuation depends on the rotation angle of the motor shaft bonding element between the LED light source and the photodetector that casts a shadow on the detector. Therefore, the position detector integrated in the galvanometer scanner can evaluate the scanning angle, but cannot evaluate any mismatch in the imaging optics or environment (which will be addressed by system 10).

[0071] Figure 1 The controller C can be an integral part of other controllers integrated with the OCT module 14 and the visualization module 12, or an independent module operatively connected to these other controllers. The controller C includes a computer-readable medium (also referred to as a processor-readable medium), which includes a non-transient (e.g., tangible) medium that participates in providing data (e.g., instructions) that can be read by a computer (e.g., by a processor of a computer). Such media can take a variety of forms, including but not limited to non-volatile media and volatile media. Non-volatile media can include, for example, optical or magnetic disks and other persistent memories. Volatile media can include, for example, dynamic random access memory (DRAM), which can constitute a main memory. Such instructions can be transmitted through one or more transmission media, including coaxial cables, copper wires, and optical fibers, including wires that include a system bus connected to a processor of a computer. Some forms of computer-readable media include, for example, floppy disks, floppy disks, hard disks, tapes, other magnetic media, CD-ROMs, DVDs, other optical media, RAMs, PROMs, EPROMs, FLASH-EEPROMs, other storage chips or boxes, or other media that can be read by a computer.

[0072] The lookup tables, databases, data repositories, or other data stores described herein may include various mechanisms for storing, accessing, and retrieving various data, including hierarchical databases, a set of files in a file storage system, an application database in a proprietary format, a relational database management system (RDBMS), and the like. Each such data storage device may be contained in a computing device that employs a computer operating system (such as one of the above-mentioned operating systems) and may be accessed via a network in one or more of various ways. The file system may be accessed from the computer operating system and may include files stored in various formats. The RDBMS may employ structured query language (SQL), as well as languages ​​for creating, storing, editing, and executing stored programs, such as the PL / SQL language mentioned above.

[0073] The flowchart shown in the figure shows the architecture, function and operation of the possible implementation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each frame in the flowchart or block diagram can represent a part of a module, a fragment or a code, which includes one or more executable instructions for implementing (multiple) specific logical functions. It should also be noted that each frame in the block diagram and / or the flowchart illustration and the combination of frames in the block diagram and / or the flowchart illustration can be implemented by a combination of a dedicated hardware-based system or dedicated hardware and computer instructions that perform a specified function or action. These computer program instructions can also be stored in a computer-readable medium, which can guide a controller or other programmable data processing device to work in a particular manner, so that the instructions stored in the computer-readable medium produce an article of instruction including instructions for implementing the function / action specified in the flowchart and / or block diagram frame.

[0074] The numerical values ​​of parameters (e.g., amounts or conditions) in this specification (including the appended claims) are understood to be modified by the term "approximately" in each corresponding instance, regardless of whether "approximately" actually appears before the numerical value. "Approximately" indicates that the numerical value is slightly imprecise (the value is somewhat close to precision; approximately or fairly close to the value; almost close). If the imprecision provided by "approximately" is not otherwise understood as this ordinary meaning in the art, "approximately" as used herein at least represents the changes that may be caused by the common methods of measuring and using such parameters. In addition, the disclosure of the range includes the disclosure of each value within the entire range or the further divided range. Each value within the range and the endpoints of the range are thus disclosed as separate embodiments.

[0075] The specific embodiments and the drawings or figures are supportive and descriptive for the present disclosure, but the scope of the present disclosure is limited only by the claims. Although some of the best modes and other embodiments for implementing the claimed disclosure have been described in detail, there are various alternative designs and embodiments to practice the disclosure defined in the appended claims. In addition, the features of the embodiments shown in the drawings or the multiple different embodiments mentioned in this specification are not necessarily to be understood as independent embodiments of each other. Instead, each feature described in one of these examples of the embodiment can be combined with one or more other desired features from other embodiments to produce other embodiments that are not described in words or with reference to the drawings. Therefore, such other embodiments fall within the scope framework of the appended claims.

Claims

1. An imaging system, comprising: a housing assembly having a head unit configured to be at least partially directed toward a target site; an optical coherence tomography (OCT) module and a visualization module, the modules being located in the housing assembly and configured to obtain OCT data and visualization data of the target site, respectively; a controller in communication with the OCT module and the visualization module, the controller having a processor and a tangible non-transitory memory having recorded thereon instructions for a calibration method; Wherein, the controller is configured as: generating a scan pattern in a visualization space for a calibration region selected in a calibration target; Acquiring OCT data of the calibration area synchronously with the scanning mode; Obtaining a projected two-dimensional OCT image of the calibration area based on the OCT data, wherein the projected two-dimensional OCT image is a back projection of the average intensity on a frontal viewing plane; superimposing the projected two-dimensional OCT image with a corresponding view extracted from the visualization data; and The projected two-dimensional OCT image is registered to the corresponding view via a cascaded image registration process having a coarse registration stage and a fine registration stage.

2. The imaging system of claim 1, wherein: The visualization module is a stereo camera, and the second set of data includes a first view and a second view of the target site.

3. The imaging system of claim 1, wherein: The visualization module is a surgical microscope.

4. The imaging system of claim 1, wherein: The scanning mode is an orthogonal raster scanning mode.

5. The imaging system of claim 1, wherein: The target site is the eye.

6. The imaging system of claim 1, wherein: Prior to registering the projected two-dimensional OCT image with the corresponding view extracted from the visualization data, the controller is configured to perform automatic image resizing.

7. The imaging system of claim 1, wherein: The controller is adapted to calculate corresponding transformation parameters of the coarse registration stage based on a translation transformation matrix compensating for the shift mismatch.

8. The imaging system of claim 1, wherein: The controller is adapted to calculate corresponding transformation parameters of the fine registration stage based on an affine diffusion tensor image (DTI) registration; and The corresponding transformation parameters are based in part on a rotation matrix, a shear matrix, and a scaling matrix.

9. The imaging system of claim 8, wherein: The controller is adapted to compensate for relatively small shift mismatches introduced during operations of at least one of rotation, shear, and scale alignment.

10. The imaging system of claim 8, wherein: The rotation matrix is ​​expressed as where θ is the corresponding transformation parameter of the rotation.

11. The imaging system of claim 8, wherein: The shear matrix is ​​expressed as Wherein, Shx and Shy are shear parameters along the first transverse direction and the second transverse direction.

12. The imaging system of claim 8, wherein: The scaling matrix is ​​expressed as Wherein, Cx and Cy are scaling parameters along the first lateral direction and the second lateral direction.

13. The imaging system of claim 1, wherein: The controller is adapted to selectively perform a verification procedure, the controller being adapted to: selecting a region of interest in the visualization space; obtaining a corresponding voltage for an OCT scan based in part on the region of interest and corresponding transformation parameters; Obtaining an acquired OCT image based on the corresponding voltage; and The acquired OCT image is compared with the region of interest.

14. The imaging system of claim 1, wherein: The region of interest is a line in the visualization space corresponding to a cross-sectional B-frame in OCT space.

15. The imaging system of claim 1, wherein: The region of interest is a quadrilateral in the visualization space corresponding to the three-dimensional volume in the OCT space.

16. The imaging system of claim 1, further comprising: a robotic arm operatively connected to the head unit and configured to selectively move the head unit; and Wherein, the robotic arm can be selectively operated to expand the observation range of the OCT module in three dimensions.

17. An imaging system, comprising: a housing assembly having a head unit configured to be at least partially directed toward a target site; an optical coherence tomography (OCT) module and a visualization module, the modules being located in the housing assembly and configured to obtain OCT data and visualization data of the target site, respectively; a controller in communication with the OCT module and the visualization module, the controller having a processor and a tangible non-transitory memory having recorded thereon instructions for a calibration method; Wherein, the controller is configured as: generating a scan pattern in a visualization space for a calibration region selected in a calibration target; Acquiring OCT data of the calibration area synchronously with the scanning mode; Obtaining a projected two-dimensional OCT image of the calibration area based on the OCT data, wherein the projected two-dimensional OCT image is a back projection of the average intensity on a frontal viewing plane; superimposing the projected two-dimensional OCT image with a corresponding view extracted from the visualization data; registering the projected two-dimensional OCT image to the corresponding view via a cascaded image registration process having a coarse registration stage and a fine registration stage; and The corresponding transformation parameters of the fine registration stage are calculated based on affine diffusion tensor image (DTI) registration.

18. The imaging system of claim 17, wherein: The controller is adapted to calculate corresponding transformation parameters of the coarse registration stage based on a translation transformation matrix compensating for the shift mismatch.

19. The imaging system of claim 18, wherein: The controller is adapted to selectively perform a verification procedure, the controller being adapted to: selecting a region of interest in the visualization space; obtaining a corresponding voltage for an OCT scan based in part on the region of interest and the corresponding transformation parameters; Obtaining an acquired OCT image based on the corresponding voltage; and The acquired OCT image is compared with the region of interest.

Images