Binocular endoscope panoramic scanning reconstruction method based on magnetic navigation

Abstract

The invention discloses a binocular endoscope panoramic scanning reconstruction method based on magnetic navigation. The binocular endoscope panoramic scanning reconstruction method comprises the following steps: 1) reading a picture by using a binocular endoscope; 2) processing the two pictures, realizing depth map estimation by using epipolar search and a depth filter, and optimizing the depth map after the depth map estimation is completed; 3) verifying the accuracy of the depth image, namely verifying the accuracy of the depth image according to the relative position relation of the extracted feature points through triangularization matching of the two images, if the accuracy of the depth image is low, regenerating the depth image, and if the accuracy is high, reading the next group of binocular camera images and repeating the process; and 4) after the generation of the depth map is completed, matching the image in the binocular camera with the depth map to obtain accurate point cloud depth color information, and performing three-dimensional reconstruction display according to the obtained point cloud data with the depth color information.

Description

Technical Field

[0001] The invention belongs to the field of automatic control technology, and in particular relates to a binocular endoscope panoramic scanning and reconstruction method based on magnetic navigation. Background Art

[0002] During the current cystoendoscopy, doctors need to observe the patient's lesions based on the images captured by the camera, and when generating a cystoendoscopy report, doctors need to circle the lesions based on the existing images to generate a report that the patient can understand. However, the existing technology has the following shortcomings: patients cannot directly see the location of their own lesions in the bladder, and can only know the size and location of the lesion area based on the doctor's description, and the doctor circles the lesions manually based on the images taken during the shooting, which is a time-consuming and labor-intensive step, and the location of the lesion tissue inside the bladder is not intuitive. Summary of the invention

[0003] The present invention mainly utilizes the inspection images of the bladder binocular endoscope to perform three-dimensional reconstruction scanning of the inside of the bladder. The generated reconstructed point cloud model allows the patient to see the location of the lesion tissue inside the bladder more clearly and intuitively. This step makes the report generated by the hospital more intelligent, which can reduce the workload of the doctor in drawing the lesion area after the report is generated, and allows the doctor to explain the subsequent treatment process more clearly. Therefore, the purpose of the binocular endoscope panoramic scanning and reconstruction method based on magnetic navigation proposed by the present invention is not to diagnose and treat diseases, but to visualize and intuitively view the inspection results for easy understanding by people.

[0004] The present invention provides a binocular endoscope panoramic scanning and reconstruction method based on magnetic navigation. The method estimates the position of the internal tissue of the bladder in three-dimensional space and obtains the corresponding point cloud data through the images collected by the binocular camera, and then realizes the three-dimensional reconstruction of the bladder based on the generated point cloud data. In this process, the polar line search and depth filter are used to generate the depth map of the binocular image. The generated depth map is associated with the reference image, and the associated image is used to extract the point cloud data in the image, and the three-dimensional reconstruction of the inside of the bladder is realized based on the point cloud data extracted from multiple pairs of associated images.

[0005] The present invention provides a binocular endoscope panoramic scanning reconstruction method based on magnetic navigation, the method comprising the following steps:

[0006] (1) Use a binocular endoscope to read the image.

[0007] (2) The two images are processed, and the epipolar line search and the depth filter are used to realize the estimation of the depth map. After the depth map estimation is completed, the depth map is optimized.

[0008] (3) Verify the accuracy of the depth map by triangulating the two images and verifying the accuracy of the depth map based on the relative position relationship of the extracted feature points. If the depth map accuracy is low, regenerate the depth map. If the depth map accuracy is high, read the next set of binocular camera images and repeat the above process.

[0009] (4) After the depth map is generated, the image in the binocular camera is matched with the depth map to obtain accurate point cloud depth and color information, and three-dimensional reconstruction and display are performed based on the obtained point cloud data with depth and color information. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 A flowchart of a binocular endoscope panoramic scanning and reconstruction method based on magnetic navigation provided in an embodiment of the present invention.

[0011] Figure 2 A schematic diagram of epipolar line search in a binocular endoscope panoramic scanning and reconstruction method based on magnetic navigation provided in an embodiment of the present invention.

[0012] Figure 3 A schematic diagram of triangulation in a binocular endoscope panoramic scanning and reconstruction method based on magnetic navigation provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0013] See attached Figure 1 The embodiment of the present invention provides a binocular endoscope panoramic scanning reconstruction method based on magnetic navigation, the method comprising the following steps:

[0014] (1) Use a binocular endoscope to read the image.

[0015] (2) The two images are processed, and the epipolar line search and the depth filter are used to realize the estimation of the depth map. After the depth map estimation is completed, the depth map is optimized.

[0016] (3) Verify the accuracy of the depth map by triangulating the two images and verifying the accuracy of the depth map based on the relative position relationship of the extracted feature points. If the depth map accuracy is low, regenerate the depth map. If the depth map accuracy is high, read the next set of binocular camera images and repeat the above process.

[0017] (4) After the depth map is generated, the image in the binocular camera is matched with the depth map to obtain accurate point cloud depth and color information, and three-dimensional reconstruction and display are performed based on the obtained point cloud data with depth and color information.

[0018] The epipolar line search in step (2) includes:

[0019] like Figure 2 As shown in the figure, the epipolar line search is to use a monocular camera to quickly search for corresponding pixels in the image, obtain the pixel point p1 of the feature point and the corresponding epipolar line l2 in the reference image, and then search for the feature point p2 with high similarity along the epipolar line l2 in the second image. The feature point represents a point in the image with unique properties that can be reliably detected and described. The pixel distribution around the pixel point and the gradient direction are often selected for detection and description. The epipolar line refers to the straight line connected by the projection points corresponding to the specific spatial point on the image plane of two different perspectives. The specific spatial point in the extreme search is the feature point p1 and the searched feature point p2. Because when the images of the two perspectives are determined, the only epipolar line can be calculated based on the feature point p1 and the coordinates of the two images. The feature point p2 is on this epipolar line. The epipolar line search greatly shortens the number of searched pixels and improves efficiency. By comparing the local structure of the pixel point and the feature point p1, such as the distribution of the neighborhood pixels and the gradient direction, it is determined whether the pixel point is the feature point p2 that is desired to be selected during the search process.

[0020] The depth filter in step (2) includes:

[0021] After finding a series of pixel points through epipolar search, the correlation of these points is calculated by the NCC normalized cross-correlation method:

[0022]

[0023] where ∑ i,j A(i,j)B(i,j) represents the sum of the products of the corresponding elements of matrix A and matrix B, that is, the sum of the products of the pixel values ​​of the corresponding areas of image A and image B. It is the square root of the sum of squares of the elements of matrix A and matrix B. The closer the NCC value calculated by the formula is to 1, the greater the correlation between the two feature points.

[0024] After finding the two feature points with the greatest correlation, it means that these two points are the projections of the actual coordinate points in the space in the two images, that is, the specific position of p2 in the second image is determined. Based on the specific positions of these two points in the two images, triangulation or parallax method is used to determine the three-dimensional coordinates of this point in the actual space and obtain the depth data. After completing the depth data estimation of the entire image, the depth estimation of the image is completed.

[0025] The triangulation in step (3) includes:

[0026] Triangulation is mainly used to verify the accuracy of depth maps. Figure 3As shown, triangulation mainly estimates the position of feature points through the position change of pixel points between two frames of images, as well as the parameters of camera internal parameters and camera posture change. In this process, K represents the intrinsic parameters of the camera. The camera intrinsic parameters are the properties of the camera itself. The camera intrinsic parameters determine the focal length of the camera and the size of the camera captured image. In this triangulation process, the camera intrinsic parameters are known by default. p and p′ represent the pixel coordinates of the feature points in the camera captured image, which are directly obtained based on the captured image. The coordinate translation R and coordinate rotation T of the camera from position O1 to O2 are calculated using the magnetic navigation on the endoscope. Using these data, the coordinates of point P in three-dimensional space can be calculated.

[0027] Using two frames of pictures, we get the following equation

[0028] p=K[I0]P

[0029] Where I represents the identity matrix and 0 represents the translation matrix is ​​a zero matrix.

[0030] p′=K′[RT]P

[0031] Expand this system of equations

[0032]

[0033] Where u represents the horizontal coordinate of the pixel point p in the image coordinates, and v represents the vertical coordinate of the pixel point p in the image coordinates. The intrinsic parameter matrix and the extrinsic parameter matrix can be combined into a parameter matrix M

[0034] p=K[I 0]P=MP

[0035] Assume that the parameters of M and M' are as follows, where m1 represents the first row of the parameter matrix, m2 represents the second row of the parameter matrix, and m3 represents the third row of the parameter matrix

[0036] ,

[0037] The following relationship is obtained:

[0038]

[0039] have to

[0040] AP=0

[0041] in

[0042]

[0043] By performing singular value decomposition on the matrix A, we can obtain:

[0044] A=UDV T

[0045] The three-dimensional coordinates of the point P are the last column of the V matrix.

[0046] The point cloud data in step (4) is extracted according to the following method:

[0047] After obtaining the original image read by the camera and obtaining the depth map of the original image through epipolar search and depth filter, the three-dimensional coordinate information and color information of the point cloud are obtained. The three-dimensional spatial information of the pixel point is calculated through the depth map, and then the color information of the pixel point is collected in the original RGB image, so as to combine and display the three-dimensional coordinates and corresponding colors of the point cloud.

[0048] The binocular endoscope panoramic scanning and reconstruction method based on magnetic navigation proposed in the present invention is based on an innovative algorithm structure. This algorithm can be applied to different endoscopes. Whether it is a monocular endoscope or a binocular endoscope, this algorithm can be used to estimate the depth data inside the bladder. Under a binocular endoscope, the polar line search and the depth filter can be converted into a disparity map estimation to achieve iterative estimation of the depth map, while under a monocular endoscope, the polar line search and the depth filter can be used to achieve depth estimation inside the bladder. In addition, magnetic navigation data is used in this algorithm to achieve accurate measurement of the camera position. In the depth estimation process, the direct use of magnetic navigation data can speed up the algorithm and reduce the amount of calculation for dense reconstruction. The application of magnetic navigation solves the difficulty that the camera will generate cumulative errors in pure visual depth estimation, and provides a theoretical basis for accurate reconstruction of the robot.

Claims

1. A binocular endoscope panoramic scanning reconstruction method based on magnetic navigation, characterized in that: The method comprises the following steps: (1) Use a binocular endoscope to read the image; (2) Processing the two images, using epipolar line search and depth filter to estimate the depth map, and optimizing the depth map after the depth map is estimated; (3) Verify the accuracy of the depth map by triangulating the two images and verifying the accuracy of the depth map based on the relative position relationship of the extracted feature points. If the depth map accuracy is low, regenerate the depth map. If the depth map accuracy is high, read the next set of binocular camera images and repeat the above process. (4) After the depth map is generated, the image in the binocular camera is matched with the depth map to obtain accurate point cloud depth and color information, and three-dimensional reconstruction and display are performed based on the obtained point cloud data with depth and color information.

2. The binocular endoscope panoramic scanning and reconstruction method based on magnetic navigation according to claim 1 is characterized in that: The depth filter in step (2) includes: After finding a series of pixel points through epipolar search, the correlation of these points is calculated by the NCC normalized cross-correlation method: where ∑ i,j A(i,j)B(i,j) represents the sum of the products of the corresponding elements of matrix A and matrix B, that is, the sum of the products of the pixel values ​​of the corresponding areas of image A and image B. It is the square root of the sum of squares of the elements of matrix A and matrix B. The closer the NCC value calculated by the formula is to 1, the greater the correlation between the two feature points. After finding the two feature points with the greatest correlation, it means that these two points are the projections of the actual coordinate points in the space in the two images, that is, the specific position of p2 in the second image is determined; based on the specific positions of these two points in the two images, triangulation or parallax method is used to determine the three-dimensional coordinates of this point in the actual space, and the depth data is obtained. After completing the depth data estimation of the entire picture, the depth estimation of the picture is completed.

3. The binocular endoscope panoramic scanning and reconstruction method based on magnetic navigation according to claim 1 is characterized in that: The triangulation in step (3) includes: Triangulation estimates the position of feature points by using the position change of pixels between two frames of images, as well as the internal parameters of the camera and the parameters of the camera posture change. In this process, K represents the intrinsic parameters of the camera, which are the properties of the camera itself. The camera intrinsic parameters determine the focal length of the camera and the size of the camera-captured image. In this triangulation process, the camera intrinsic parameters are assumed to be known. p and p′ represent the pixel coordinates of the feature points in the camera-captured image, which are directly obtained from the captured image. The coordinate translation R and coordinate rotation T of the camera from position O1 to O2 are calculated using the magnetic navigation on the endoscope. Using these data, the coordinates of point P in three-dimensional space can be obtained. Using two frames of pictures, we get the following equation p=K[I0]P Where I represents the identity matrix, and 0 represents that the translation matrix is ​​a zero matrix; p′=K′[RT]P Expand this system of equations Where u represents the horizontal coordinate of the pixel point p in the image coordinates, and v represents the vertical coordinate of the pixel point p in the image coordinates; the intrinsic parameter matrix and the extrinsic parameter matrix are combined into a parameter matrix M p=K[I0]P=MP Assume that the parameters of M and M' are as follows, where m1 represents the first row of the parameter matrix, m2 represents the second row of the parameter matrix, and m3 represents the third row of the parameter matrix The following relationship is obtained: have to AP=0 in By performing singular value decomposition on the matrix A, we can obtain: A=UDV T The three-dimensional coordinates of the point P are the last column of the V matrix.

4. The binocular endoscope panoramic scanning and reconstruction method based on magnetic navigation according to claim 1 is characterized in that: The point cloud data in step (4) is extracted according to the following method: After obtaining the original image read by the camera and obtaining the depth map of the original image through epipolar search and depth filter, the three-dimensional coordinate information and color information of the point cloud are obtained; the three-dimensional spatial information of the pixel point is calculated through the depth map, and then the color information of the pixel point is collected in the original RGB image, so as to combine and display the three-dimensional coordinates and corresponding colors of the point cloud.

Images