A wide baseline stereo vision three-dimensional imaging method and device for intraoperative radiotherapy

Abstract

The application provides a wide-baseline stereo vision three-dimensional imaging method and device for intraoperative radiotherapy, which comprises: chessboard calibration by a camera to realize binocular camera calibration and image epipolar rectification; linear equations are established according to three-dimensional coordinates of chessboard corner points in a left camera coordinate system and three-dimensional coordinates in a reference point coordinate system, and a singular value decomposition algorithm is used to calculate a rotation matrix and a translation vector between the two three-dimensional coordinate systems to realize coordinate conversion between the two coordinate systems; a deep learning network for end-to-end disparity estimation is trained using a medical optical image and a depth image dataset of an intraoperative radiotherapy scene, so that the network can predict disparity for an organ optical image taken by a binocular camera during surgery; the intraoperative organ image taken by the wide-baseline binocular camera is subject to mutual occlusion of biological tissues, and local features are extracted using SIFT features, and the matching degree of corresponding points of left and right images is evaluated according to the distance of feature vectors, so that disparity values with large calculation errors are screened out; three-dimensional coordinates of the organ are calculated and coordinate system conversion is performed to obtain three-dimensional coordinates of the organ in the reference point coordinate system; the lesion mask image of the organ is subjected to epipolar rectification, and the corresponding three-dimensional points are marked.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of stereovision three-dimensional reconstruction technology, and particularly relates to a wide-baseline stereovision three-dimensional imaging method and device for intraoperative radiotherapy. BACKGROUND

[0002] Intraoperative radiotherapy is a method of directly applying radiation therapy to the surgical area during surgery. The main advantage of intraoperative radiotherapy is that it can accurately irradiate the tumor bed and its surrounding high-risk area under direct vision, thereby reducing the radiation exposure of normal tissues and improving local tumor control rate.

[0003] In order to improve the accuracy and effect of intraoperative radiotherapy, three-dimensional reconstruction technology plays an important role in this process. Through three-dimensional reconstruction technology, the morphology and position of the tumor can be more accurately depicted, which can provide more intuitive and accurate surgical and radiotherapy plans for doctors, and is also an important part of intraoperative radiotherapy automation and intelligentization.

[0004] The accuracy of traditional three-dimensional reconstruction algorithms depends largely on the calibration accuracy of the camera and the calculation of the disparity. This method uses the classic checkerboard calibration algorithm to achieve high-precision binocular camera calibration, calculates the disparity through a deep learning network algorithm trained based on a large amount of medical image data, and evaluates the accuracy of the disparity through a SIFT feature matching algorithm, thereby achieving high-precision organ three-dimensional reconstruction and obtaining high-quality organ three-dimensional clouds, which provides an important reference for intraoperative radiotherapy. SUMMARY

[0005] In view of the demand of intelligent intraoperative radiotherapy instruments for high-quality and high-precision organ three-dimensional point clouds and the dependence of traditional three-dimensional reconstruction algorithms on the calibration accuracy of the camera and the calculation accuracy of the disparity, the following method is proposed.

[0006] In the first aspect, in view of the rapid development of deep learning networks and their strong ability to extract image features, a deep learning network is built to extract multi-scale features of binocular images, a disparity calculation module is constructed using cross-attention mechanism and cost volume, a medical three-dimensional reconstruction dataset is constructed using optical medical images taken in the intraoperative scene and depth maps taken by a depth camera, and the depth is used as information to supervise the training of the deep learning network for disparity calculation, so that the network has superior performance in calculating the disparity of binocular optical medical images in the intraoperative scene. The organ image pair after epipolar calibration is input into the deep learning network, and the disparity of the corresponding pixel points of the left image of the organ is calculated.

[0007] Secondly, even if the trained deep learning network shows superior performance on the training dataset, there are still some errors in the disparity inference process of real data, so it is necessary to filter out the points with errors. Since the baseline of the binocular camera of the interventional radiotherapy instrument involved in the method is relatively long, the occlusion between the biological tissues in the corresponding region causes a large difference in the pixel values of the corresponding positions, so the previous disparity evaluation algorithm based on the difference in pixel values of the corresponding points is no longer applicable. Therefore, the method uses the SIFT feature extraction method to extract the local features of the region around the corresponding pixel points, and uses the Euclidean distance between the two local features as the evaluation index.

[0008] Thirdly, the three-dimensional point coordinates are converted from the left camera coordinate system to the instrument reference point coordinate system. Since the coordinates of the corner points of the chessboard are easy to extract on the image and are arranged in order, the disparity of the corner points is calculated directly after the epipolar rectification, and the three-dimensional coordinates of the corner points in the left camera are calculated. Under the premise that the three-dimensional coordinates of the corner points of the chessboard in the reference point coordinate system are known, the rotation matrix and the translation vector are obtained by using the SVD algorithm. BRIEF DESCRIPTION OF DRAWINGS

[0009] Figure 1 The flowchart of the whole process of the wide-baseline stereo vision three-dimensional imaging method and device for interventional radiotherapy proposed by the method.

[0010] Figure 2 The structure diagram of the disparity calculation deep learning network built by the method.

[0011] Figure 3 The simple structure diagram of the interventional radiotherapy instrument of the method and the reference point coordinate system. The position of the reference point is the emission point of the interventional radiotherapy radiation, and the negative direction of the z-axis is the radiation emission direction. When shooting organ images, the images should be distributed around the plane where the camera optical axis intersection point is located to ensure the clarity of the images and the presence of more same regions between the organ image pairs. When shooting the chessboard images, the chessboard should be placed on the plane where the binocular camera optical axis intersection point is located, and the range of the multiple pairs of chessboard images should cover the common field of view of the binocular camera as much as possible. DETAILED DESCRIPTION

[0012] The whole specific process can be seen in Figure 1 .

[0013] Firstly, camera calibration. Use the chessboard calibration method. When shooting the chessboard images, the chessboard should be placed on the plane where the binocular camera optical axis intersection point is located, and the range of the multiple pairs of chessboard images should cover the common field of view of the binocular camera as much as possible. Correspondingly, when shooting the organ pictures during the operation, the binocular camera should be moved so that the shot organ is on the plane where the binocular camera optical axis intersection point is located.

[0014] Second, the training of the depth learning network for disparity estimation. The optical medical images and depth maps captured by the depth camera are used to construct the medical three-dimensional reconstruction dataset. When constructing the dataset, attention should be paid to the pixel-level registration of the depth map and the optical image. Because the network uses depth values as supervision information for training, specifically, the depth values are converted into disparity values for supervised training.

[0015] Third, the calculation of the rotation matrix and translation vector for coordinate system conversion.

[0016] Step S11: Place the chessboard on the xOy plane of the reference point, and rotate the chessboard so that the long and short sides of the chessboard are parallel to the x and y axes, respectively. Figure 1

[0017] Step S12: Figure 1 The binocular camera and the reference point are relatively static. The binocular camera is translated along the z-axis upward by a height h, so that the chessboard is located on the intersection plane of the camera optical axis. The size of the standard chessboard is known, and the x and y coordinates of the corresponding chessboard corner points in the reference point coordinate system can be directly obtained. The z coordinate is -h, so the coordinates of the chessboard corner points in the reference point coordinate system are known. At this time, the chessboard image is captured.

[0018] Step S13: Perform epipolar rectification on the chessboard image captured in step S12, extract the chessboard corner point coordinates and calculate the disparity of the corner points. The three-dimensional coordinates of the chessboard corner points in the left camera coordinate system are calculated, and the linear equations are solved using SVD to obtain the rotation matrix and translation vector from the left camera coordinate system to the reference point coordinate system.

[0019] Fourth, three-dimensional reconstruction of the organ.

[0020] Step S21: Capture multiple pairs of organ images, perform epipolar rectification using camera calibration data, and input the trained depth learning network to calculate the disparity.

[0021] Step S22: Obtain the corresponding points of each pixel point in the left image in the right image through the calculated disparity, extract the SIFT features of the point pairs, calculate the Euclidean distance, and if it is less than 510, it is considered as a matching point, that is, the disparity calculation is correct, and if it is greater than or equal to 510, it is considered as an error in the disparity calculation.

[0022] Step S23: After filtering out the incorrect disparity, calculate the three-dimensional coordinates of the remaining pixel points in the left camera coordinate system to obtain multiple point clouds. According to the camera pose parameters during shooting, the pose relationship between different point clouds is calculated. Taking the point cloud of the last pair of organ images as the reference, the other point clouds are rotated and translated to the point cloud coordinate system of the last pair of organ images to realize point cloud fusion.

[0023] ​Step S24: obtain the lesion mask according to the fluorescence image of the last pair of organ images, calculate the three-dimensional point corresponding to the mask, and mark it as a lesion;

[0024] Step S25: convert the three-dimensional coordinates of the fused point cloud from the left camera coordinate system to the reference point coordinate system through the rotation matrix and translation vector obtained in step S13.

Claims

1. A wide-baseline stereo vision three-dimensional imaging method for intraoperative radiotherapy, characterized in that, The method comprises the following steps: Step 1: A batch of checkerboard image pairs are photographed by using a binocular camera with fixed relative position, for single target calibration and double target calibration of the camera, to obtain calibration result data for polar correction of the images; Step 2: Use a binocular camera to capture a reference checkerboard image. Given that the three-dimensional coordinates of the checkerboard corner points in the reference coordinate system are known, use binocular calibration data to correct the reference checkerboard image. Calculate the disparity between the corresponding checkerboard corner points in the left and right images after correction, thereby obtaining the three-dimensional coordinates of the checkerboard corner points in the left camera coordinate system. The corrected image can be considered as being captured by a binocular parallel camera. Based on the imaging principle of the binocular parallel camera and the similar triangle theorem, the three-dimensional coordinates of the corner points in the left camera coordinate system are obtained through formula (1). In formula (1), (x... world ,y world ,z world (x) represents the 3D coordinates of the pixel in the left image within the left camera coordinate system. pixel ,y pixel () represents the pixel coordinates of the pixel in the left image. is the intrinsic parameter matrix of the left camera obtained by chessboard calibration, b is the baseline length of the binocular camera obtained after chessboard calibration, and d is the disparity of the corresponding chessboard corner points of the left and right images after correction. Step 3: According to the three-dimensional coordinates of the corner points in the left camera coordinate system and the reference point coordinate system of the intraoperative radiotherapy system, a linear equation set is established, and a singular value decomposition method is used to calculate the rotation matrix and translation vector required for mutual conversion between the three-dimensional coordinates of the same three-dimensional point in the two reference coordinate systems; Step 4: The polar correction of the organ image pair is performed using the checkerboard calibration result data of step 1, and the trained deep learning network calculator is used to calculate the parallax of the corresponding pixel points of the left image of the organ; Step 5: According to the principle of parallel camera imaging, the corresponding pixel points of the left image of the organ in the right image can be obtained, the SIFT features of the corresponding pixel points are calculated, and the feature vector distance is calculated; the parallax value corresponding to the pixel points with a vector distance exceeding the threshold value is regarded as a calculation error, and the corresponding pixel points do not participate in the subsequent three-dimensional coordinate calculation; Step 6: The three-dimensional coordinates of the corresponding pixel points of the left image of the organ in the left camera coordinate system are calculated through the parallax value and the calibration data, and the coordinates are converted to the reference point coordinate system set by the intraoperative radiotherapy instrument; Step 7: The polar correction of the lesion mask image corresponding to the organ image is performed, the three-dimensional coordinates corresponding to the lesion mask image are calculated, and the corresponding three-dimensional points are marked, so that the intraoperative radiotherapy instrument can perform treatment on the lesion area; Step 8: A plurality of image pairs are obtained by photographing the same organ, a plurality of organ point cloud files are obtained, the point cloud files are fused, and finally a high-quality organ point cloud with lesion marking is obtained.

2. The wide-baseline stereo vision three-dimensional imaging method for intraoperative radiotherapy according to claim 1, characterized in that, Chessboard corner points in the left camera coordinate system and the reference system coordinate system three-dimensional coordinates known under the premise of the transformation relationship between the two coordinate systems, so that the subsequent reconstruction of the organ surface three-dimensional coordinates can be converted from the left camera coordinate system to the reference coordinate system; coordinate system conversion of linear equations formula as formula (2), wherein (x r 1,y r 1,z r 1),(x r 2,y r 2,z r 2),(x r 3,y r 3,z r 3)…for the corresponding three-dimensional point coordinates in the reference point coordinate system, its corresponding three-dimensional point coordinates in the left camera coordinate system is (x c 1,y c 1,z c 1),(x c 2,y c 2,z c 2),(x c 3,y c 3,z c 3)……;through singular value decomposition method can be calculated 3×3 rotation matrix R and 3×1 translation vector T; 3. The wide-baseline stereo vision three-dimensional imaging method for intraoperative radiotherapy of claim 1, wherein, The deep learning network for parallax calculation extracts multi-scale features of the image, uses a cross-attention mechanism to obtain a similarity matrix of the binocular image for low-dimensional features with high resolution, and inputs the features of different scales into corresponding scale parallax calculation modules, so that the parallax results are transmitted between adjacent scale parallax calculation modules to enhance the interaction of low-dimensional features with high-dimensional features. Specifically, the high-dimensional parallax graph of small size is improved in resolution after upsampling and convolution, the pseudo image of the left camera perspective after pixel offset is obtained from the feature graph of the right image according to the parallax graph with improved resolution, the cost volume is calculated from the left feature graph and the pseudo image, and then the cost volume, the similarity matrix and the left feature graph are spliced in the channel dimension and input into a convolution layer to obtain a parallax graph. The network realizes multiple physical constraints through multiple loss functions, converts the true depth graph into a true parallax graph as supervision information, calculates the L1 loss between the predicted parallax graph and the true parallax graph, uses the SSIM similarity to construct a loss function, and pre-trains the cross-attention module using the L1 loss.

4. The wide-baseline stereo vision three-dimensional imaging method for use in intraoperative radiotherapy according to claim 1, characterized in that, The basic principle of screening the wrong parallax value is the SIFT feature similarity of the corresponding pixel points; after the epipolar correction, the corresponding points of the left and right images are on the same horizontal line, and the definition of the parallax value is the difference between the pixel coordinates of the corresponding points of the left and right images after the epipolar correction in the horizontal direction; the pixel coordinates of the corresponding points of the left image in the right image are calculated according to the parallax value; the SIFT feature vectors of the corresponding points in the left and right images are extracted, the Euclidean distance between the vectors is calculated, and the smaller the distance is, the higher the similarity is; if the distance is greater than 510, it is considered that the similarity of the corresponding points is too low, and the parallax calculation is inaccurate.

5. A wide-baseline stereo vision three-dimensional imaging device for intraoperative radiotherapy, characterized in that, The method of any one of claims 1-4, wherein the point cloud fusion uses a method of recording the pose parameters of the camera during the imaging of the organ images, taking the position of the camera imaging the last pair of organ images as the reference, and performing translation and rotation on the organ point cloud obtained by three-dimensional reconstruction, so as to make it fused with the point cloud of the last pair of organ images imaged by the instrument.

Images