Abdominal cavity reconstruction and focus positioning method, system and equipment based on binocular endoscope

Abstract

The invention belongs to the technical field of medical image three-dimensional reconstruction, and discloses an abdominal cavity reconstruction and focus positioning method based on a binocular endoscope, which comprises the following steps of: calibrating and correcting internal reference of the binocular endoscope; inputting the trained depth estimation network to obtain a disparity map and processing the disparity map into a picture according to frames; the luminosity error is minimized through a coarse-to-fine pyramid optimization method, and a transformation matrix of the endoscope pose between two continuous frames is calculated through nonlinear iterative optimization; light beam adjustment optimization is carried out on the motion trail of the endoscope, a depth map is superposed, outliers are removed, and three-dimensional reconstruction of the abdominal cavity tissue is achieved; recognizing a focus, extracting three-dimensional information according to the disparity map, and visually guiding the operation arm to move to a focus point in combination with a calibrated hand-eye matrix so as to realize positioning. According to the method, rapid and dense three-dimensional reconstruction of the peritoneal environment before the operation and accurate focus positioning based on visual guidance during the operation are realized, the accuracy of motion track estimation of the endoscope is improved, and autonomous development of the surgical robot is expected to be promoted.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional reconstruction of medical images, and in particular to a binocular endoscope-based abdominal cavity reconstruction and lesion positioning method, system and equipment. Background Art

[0002] Minimally invasive surgery has been widely used in clinical practice in the past decade or so, thanks to its advantages of small trauma, low infection rate, few complications, less pain for patients, and quick recovery, which has promoted milestone progress in medicine. However, compared with traditional open surgery, it has problems such as limited field of view, inaccurate endoscope positioning, and lack of information about the surrounding environment. The traditional monocular laparoscope commonly used in minimally invasive surgery can only capture two-dimensional scenes, resulting in insufficient estimation of the advancement distance of surgical instruments and the depth of the body cavity. It is impossible to intuitively and three-dimensionally display the relationship between lesions, blood vessels, and surrounding organ tissues in the abdominal environment, which greatly increases the difficulty and complexity of the operation and is more dependent on the doctor's experience.

[0003] With the maturity of binocular vision theory and the development of computer technology, binocular endoscopes are used to calculate the depth information of the body cavity environment, stereoscopically locate the lesion area, and reconstruct the collected body cavity information in real time and densely through computers. Simultaneous Localization and Mapping (SLAM) technology is used to establish an abdominal organ tissue model and track the laparoscope trajectory, which more accurately and truly reflects the abdominal tissue structure and the spatial relationship of the lesion, provides doctors with visual depth information, realizes endoscopic positioning and navigation, and greatly improves the scientificity and success rate of laparoscopic minimally invasive surgery. In addition, the endoscope cannot observe the information under the surface of the organ. After using SLAM to quickly and densely reconstruct the abdominal environment, it can be fused and registered with the preoperative diagnosis CT or MRI images to display the patient's anatomical structure during the operation. At the same time, it can be combined with AR technology to superimpose additional information in the three-dimensional scene, such as annotations of target lesions, contour distances of organs, and tumor measurements, which can also achieve the expansion of the current endoscopic field of view. With the development of robotic technology, the autonomy of surgical procedures is an inevitable trend. Rapid and intensive three-dimensional reconstruction of abdominal tissue and lesion localization have promoted the use of surgical robots to complete autonomous surgical operations. Therefore, there is an urgent need for abdominal reconstruction and lesion localization methods based on binocular endoscopes. Summary of the invention

[0004] In view of the above defects or improvement needs of the prior art, the present invention provides a binocular endoscope-based abdominal cavity reconstruction and lesion localization method.

[0005] To achieve the above object, according to one aspect of the present invention, a binocular endoscope-based abdominal reconstruction and lesion localization method is provided, comprising the following steps:

[0006] S1: Improve the hand-eye calibration of the binocular endoscope internal parameters based on binocular constraints and data loops, obtain the left and right eye images collected by the binocular endoscope and perform corrections based on the obtained internal parameters;

[0007] S2: Input the corrected left and right eye images in step S1 into the trained depth estimation network based on the disparity attention mechanism, output the disparity map, use the binocular endoscope to collect the video data set of the abdominal phantom or the environment close to the reconstructed tissue, and process it into images frame by frame;

[0008] S3: The image obtained in step S2 is subjected to a pyramid optimization method from coarse to fine to minimize the photometric error, and a transformation matrix of the endoscope posture between two consecutive frames is calculated by nonlinear iterative optimization;

[0009] S4: obtaining the endoscope motion trajectory through the transformation matrix obtained in step S3, performing beam adjustment optimization on the endoscope motion trajectory, superimposing the depth map and removing outliers, so as to realize three-dimensional reconstruction of abdominal tissue;

[0010] S5: Based on the three-dimensional reconstruction of step S4, the lesion is identified, and the three-dimensional information is extracted according to the disparity map. Combined with the calibrated hand-eye matrix, the surgical arm is visually guided to move to the lesion point to achieve positioning.

[0011] Preferably, the method for calibrating the internal parameters of the binocular endoscope in step S1 is as follows: the left and right eyes of the binocular endoscope are respectively connected to the computer through the USB interface using a video acquisition card, the calibration board is placed on the robotic arm base platform, the position of the end of the robotic arm is changed, and multiple groups of chessboard images and the corresponding angle data of each joint angle of the robotic arm are recorded; at the same time, the focal length, optical center, distortion coefficient and internal parameters of the left and right eye conversion matrix of the binocular endoscope are calibrated.

[0012] Preferably, the correction method in step S1 is as follows: constructing a hand-eye calibration equation of AX=XB, adding data loop processing, the transformation matrix of the left eye relative to the right eye is obtained through internal parameter calibration and kept constant, and adding binocular constraints to optimize the calibration result.

[0013] Preferably, the depth estimation network based on the disparity attention mechanism in step S2 is obtained by offline training by the following method, which specifically includes the following steps:

[0014] S21: Collect a set of data sets as input;

[0015] S22: Input the data set of step S21 into the hourglass-shaped feature extraction network to obtain a feature map;

[0016] S23: Input the feature map obtained in step S22 into the cascaded disparity attention module, and perform matching cost regression in a coarse-to-fine manner;

[0017] S24: the regression graph obtained in step S23 is passed through an hourglass-type disparity refinement network;

[0018] S25: Output the disparity map.

[0019] As a preferred embodiment, the specific steps of step S23 are: using the 1×1 convolution obtained in step S22 to extract the left and right eye feature maps I respectively left and I right , size is H×W×C, where H is the batch of matrix multiplications, each matrix has W points, and the feature dimension of each point is C;

[0020] Perform geometric product and SoftMax function operations on the feature map, and encode the feature similarity of any two positions along the epipolar line into the disparity attention map M right→left and M left→right In the example, the size is H×W×W, and the mask is obtained by removing the occluded pixels according to the point pair matching correlation;

[0021] The disparity attention mechanism is used instead of the cost volume to reduce the amount of computation and memory usage. There is no need to set a fixed maximum disparity value. Left-right consistency and cycle consistency are introduced to obtain a reliable matching relationship, calculate left-right consistency, and finally output the disparity map.

[0022] Preferably, the specific steps of step S3 are: based on the assumption that the grayscale value of the same spatial point is the same under different camera postures, focus on the grayscale value gradient of image pixels, adopt a coarse-to-fine pyramid structure to prevent the fast movement speed from falling into the local minimum, perform nonlinear iterative optimization through the Gauss-Newton method, minimize the photometric error instead of the reprojection error, and directly calculate the camera motion information.

[0023] As a preference, the specific steps of step S4 are: assuming that the motion trajectory includes m camera postures and n landmark points, and the postures are represented by Lie algebra, then the optimization objective is:

[0024]

[0025] Among them, z ij represents the actual pixel position of the jth feature point of the ith camera in the original image, ξ i represents the position of the i-th camera, p i Represents the world coordinate system position of the i-th feature point. The LM algorithm is used and the Schur elimination method is used to sort and solve the optimized data. According to the optimized endoscope motion trajectory, the depth map is superimposed at a certain number of frames to obtain the three-dimensional reconstruction of the abdominal tissue.

[0026] Preferably, the specific steps of step S5 are as follows: a needle is fixed on the designed binocular endoscope clamp, and the conversion relationship between the tool coordinate system and the endoscope coordinate system is obtained by extracting three-dimensional information according to the disparity map. During the operation, the lesion area and the center point are extracted by color space conversion, threshold segmentation, contour extraction and fitting, etc. After obtaining the disparity map, the spatial position of the lesion point in the robot base coordinate system is obtained by combining the hand-eye calibration matrix, and the surgical instrument is visually guided to move to the lesion point.

[0027] To achieve the above object, according to one aspect of the present invention, there is provided a binocular endoscope-based abdominal reconstruction and lesion localization system, which is used to perform the binocular endoscope-based abdominal reconstruction and lesion localization method, and comprises:

[0028] Binocular endoscope calibration module, used to obtain the internal parameters of the binocular endoscope, such as focal length, optical center, distortion coefficient, left-right eye conversion matrix, and hand-eye calibration matrix of the binocular endoscope coordinate system relative to the coordinate system of the end of the robotic arm;

[0029] An image acquisition and correction module is used to obtain a binocular endoscope synchronously acquire a left-eye image sequence and a right-eye image sequence, and use the calibrated internal parameter matrix to perform image correction;

[0030] The depth estimation network training module is used to pre-train the deep learning model with the collected data set to obtain the model parameters;

[0031] A disparity map acquisition module, used to acquire a disparity map of a captured image in real time;

[0032] The endoscope motion trajectory calculation module is used to calculate the pose transformation matrix between the current frame and the next frame according to the coarse-to-fine minimization of the photometric error between the two frames to obtain the motion trajectory of the endoscope;

[0033] The abdominal tissue 3D reconstruction module is used to optimize the motion trajectory of the endoscope by using the bundle adjustment method, superimpose the 3D point cloud calculated by the disparity map and remove outliers to complete the 3D reconstruction of the abdominal tissue;

[0034] The lesion recognition and vision-guided surgical arm positioning module is used to identify the lesion area, extract three-dimensional information based on the disparity map, and combine it with the calibrated hand-eye matrix to visually guide the surgical arm to move to the lesion point for positioning.

[0035] To achieve the above-mentioned purpose, according to one aspect of the present invention, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the binocular endoscope-based abdominal reconstruction and lesion localization method when executing the computer program.

[0036] The present invention focuses on obtaining only visual information through binocular laparoscopes, using Zhang Zhengyou's calibration method to obtain the internal parameters of binocular laparoscopes, achieving sub-pixel reprojection accuracy that meets the requirements of medical use, and introducing binocular constraints and data loops to improve the accuracy of the hand-eye calibration algorithm based on binocular endoscopes. The collected left and right eye views are corrected by polar lines, and the depth information is obtained by fine-tuning the binocular unsupervised network based on the parallax attention mechanism, and the pseudo RGBD sequence is obtained by aligning the timestamps. At the same time, the trajectory of the binocular endoscope is obtained by minimizing the photometric error and parallel processing based on the coarse-to-fine pyramid optimization method, so as to achieve fast and dense three-dimensional reconstruction of the collected abdominal image information and accurate positioning of the lesion, provide surgeons or surgical robots with global environmental prior information, better understand the position relationship between lesions and blood vessels, and further reduce intraoperative risks. In addition, the three-dimensional reconstructed point cloud can be fused and registered with the CT or MRI images of preoperative diagnosis, and the invisible anatomical structure under the surface of the patient's organs can be displayed during the operation. It can be used in the fields of AR and VR in the future to train novice doctors, etc., and promote the development of autonomous laparoscopic robotic surgery in the future.

[0037] In general, the above technical solution conceived by the present invention has the following beneficial effects compared with the prior art:

[0038] (1) Introducing binocular constraints and data looping to improve the hand-eye calibration algorithm based on binocular endoscope;

[0039] (2) Unsupervised learning algorithm for binocular images based on the parallax attention mechanism to obtain depth information;

[0040] (3) Coarse-to-fine pyramid optimization to minimize photometric errors and track the endoscope trajectory through parallel processing;

[0041] (4) It can quickly and densely reconstruct the preoperative abdominal environment and locate the lesion with high precision during surgery. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a flow chart of the binocular endoscope-based abdominal cavity reconstruction and lesion localization method provided by the present invention;

[0043] Figure 2 It is a flow chart of an unsupervised binocular depth estimation method based on a parallax attention mechanism provided by the present invention;

[0044] Figure 3 It is the endoscope motion trajectory and three-dimensional reconstruction result provided by the present invention. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0046] Before the system is run, it is only necessary to perform the internal calibration and hand-eye calibration of the binocular endoscope once, and collect a set of left and right eye image data sets of a scene to fine-tune the unsupervised depth estimation network. Before the operation, the reconstructed video collected by the binocular endoscope is uploaded to the computer GPU in real time through the acquisition card. After obtaining the depth map through the trained depth estimation network, a pseudo RGBD frame is created, which is then transmitted to the SLAM algorithm for parallel processing, the binocular endoscope trajectory is estimated, and the dense reconstruction of the abdominal tissue is completed. During the operation, the lesion pixel points are extracted by threshold segmentation, contour extraction and other methods, and the spatial positioning in the robot base coordinate system is performed according to the depth map combined with the hand-eye calibration matrix, and the surgical instrument is visually guided to move to the lesion point.

[0047] See also Figure 1-Figure 3 The present invention provides a binocular endoscope-based abdominal reconstruction and lesion localization method, system and device. The binocular endoscope-based abdominal reconstruction and lesion localization system is used to perform the binocular endoscope-based abdominal reconstruction and lesion localization method, which includes:

[0048] Binocular endoscope calibration module, used to obtain the internal parameters of the binocular endoscope, such as focal length, optical center, distortion coefficient, left-right eye conversion matrix, and hand-eye calibration matrix of the binocular endoscope coordinate system relative to the coordinate system of the end of the robotic arm;

[0049] An image acquisition and correction module is used to obtain a binocular endoscope synchronously acquire a left-eye image sequence and a right-eye image sequence, and use the calibrated internal parameter matrix to perform image correction;

[0050] The depth estimation network training module is used to pre-train the deep learning model with the collected data set to obtain the model parameters;

[0051] A disparity map acquisition module, used to acquire a disparity map of a captured image in real time;

[0052] The endoscope motion trajectory calculation module is used to calculate the pose transformation matrix between the current frame and the next frame according to the coarse-to-fine minimization of the photometric error between the two frames to obtain the motion trajectory of the endoscope;

[0053] The abdominal tissue 3D reconstruction module is used to optimize the motion trajectory of the endoscope by using the bundle adjustment method, superimpose the 3D point cloud calculated by the disparity map and remove outliers to complete the 3D reconstruction of the abdominal tissue;

[0054] The lesion recognition and vision-guided surgical arm positioning module is used to identify the lesion area, extract three-dimensional information based on the disparity map, and combine it with the calibrated hand-eye matrix to visually guide the surgical arm to move to the lesion point for positioning.

[0055] The computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the binocular endoscope-based abdominal cavity reconstruction and lesion localization method when executing the computer program.

[0056] The binocular endoscope-based abdominal reconstruction and lesion localization method includes the following steps:

[0057] S1: Improve the hand-eye calibration of the binocular endoscope internal parameters based on binocular constraints and data loops, obtain the left and right eye images collected by the binocular endoscope and perform corrections based on the obtained internal parameters;

[0058] The left and right eyes of the binocular endoscope are connected to the computer through the USB interface using a video acquisition card. The calibration board is placed on the base platform of the robotic arm. The position of the end of the robotic arm is changed to record multiple sets of chessboard pictures and the corresponding angle data of each joint angle of the robotic arm. At the same time, the focal length, optical center, distortion coefficient and left and right eye conversion matrix of the binocular endoscope are calibrated.

[0059] The specific correction method is as follows: construct the hand-eye calibration equation AX = XB. In order to collect less data so that the iterative solution of X is still accurate, data loop processing is added instead of sequential processing. The conversion matrix of the left eye relative to the right eye is obtained through internal parameter calibration and remains constant, so binocular constraints are added to optimize the calibration results.

[0060]

[0061] in, Represents the transformation matrix from coordinate system j to coordinate system i. A is the product of any two transformation matrices of the robot end relative to the base. B is the product of any two transformation matrices of the chessboard corner points relative to the endoscope. The l subscript represents the left eye coordinate system, and the r subscript represents the right eye coordinate system. It is solved by the Perspective-n-Point (PnP) algorithm. The range of i is the number of calibration images collected, and the range of j is the number of calibration images collected minus 1. The cycle is repeated j times for each i.

[0062] The left and right eyes of the binocular endoscope are connected to the computer through the USB interface using a video capture card to obtain real-time video streams. The left and right eye images collected by the binocular endoscope are corrected using the calibrated internal parameter matrix. The image is cropped at the center, and u0 and v0 representing the optical center pixel position in the camera internal parameter are modified accordingly.

[0063] S2: Collect a set of data sets to train an unsupervised depth estimation network based on the disparity attention mechanism, input the corrected left and right eye images in step S1 into the trained depth estimation network, output the disparity map, use a binocular endoscope to collect a video data set of the abdominal phantom or an environment close to the reconstructed tissue, and process it into images frame by frame.

[0064] Among them, the depth estimation network based on the disparity attention mechanism is obtained by offline training through the following method, which specifically includes the following steps:

[0065] S21: Collect a set of data sets as input;

[0066] The left and right eye images collected by the binocular endoscope are corrected using the calibrated intrinsic parameter matrix; the image is cropped at the center, and u0 and v0 representing the optical center pixel position in the camera intrinsic parameters are modified accordingly.

[0067] S22: Input the data set in step S21 into an hourglass-shaped feature extraction network, first from coarse to fine, and then from fine to coarse to extract feature maps of the left and right eye images respectively;

[0068] S23: Input the feature map obtained in step S22 into the cascaded disparity attention module, and perform matching cost regression in a coarse-to-fine manner;

[0069] Use the 1×1 convolution obtained in step S22 to extract the left and right eye feature maps I respectively left and I right , size is H×W×C, where H is the batch of matrix multiplications, each matrix has W points, and the feature dimension of each point is C;

[0070] Perform geometric product and SoftMax function operations on the feature map, and encode the feature similarity of any two positions along the epipolar line into the disparity attention map M right→left and M left→right In the middle, the recurrent attention map M left→right→left and M right→left→right In the process, the mask is obtained by removing the occluded pixels according to the point pair matching correlation;

[0071] The disparity attention mechanism is used instead of the cost volume to reduce the amount of computation and memory usage. There is no need to set a fixed maximum disparity value. Left-right consistency and cycle consistency are introduced to obtain a reliable matching relationship, calculate left-right consistency, and finally output the disparity map.

[0072]

[0073] Compute cycle consistency:

[0074]

[0075] S24: the regression graph obtained in step S23 is passed through an hourglass-type disparity refinement network;

[0076] The calculated initial disparity map and initial feature map are input, and the features of the left image are used as guidance to provide structural information such as edges, first refining the disparity from coarse to fine and then from fine to coarse.

[0077] The loss function of the unsupervised network includes the photometric loss Smoothness loss and parallax attention loss

[0078]

[0079] Among them, λ p ,λ s ,λ PAM are their respective weights.

[0080] The parallax attention mechanism loss introduces three terms of different scales to regularize the PAM loss to obtain accurate and consistent stereo correlation, namely PAM photometric loss:

[0081]

[0082] PAM smoothness loss:

[0083]

[0084] PAM cycle consistency loss:

[0085]

[0086] Where p is the valid pixels covered by the valid mask, and N is the number of valid pixels. and They are and The effective pixels are and They are and The number of pixels, For example, It is described using feature similarity (i,k) relative to The contribution of (i,j) in s is the identity matrix.

[0087] S25: output disparity map;

[0088] During abdominal reconstruction and lesion localization, based on the model parameters trained by the above-mentioned depth estimation learning network, the above-mentioned corrected left and right eye images are input, and a disparity map is output. The left image is combined with the corresponding output disparity map to obtain a pseudo RGBD frame.

[0089] S3: The images processed by frames in step S2 are subjected to minimization of photometric errors according to a coarse-to-fine pyramid optimization method, and a transformation matrix of the endoscope posture between two consecutive frames is calculated by nonlinear iterative optimization;

[0090] SLAM uses a dense direct method, does not calculate feature points and descriptors, and is suitable for abdominal environments that lack texture. Based on the assumption that the grayscale value of the same spatial point is the same under different camera positions, it focuses on the grayscale value gradient of image pixels and adopts a coarse-to-fine pyramid structure to prevent the movement speed from falling into a local minimum. It uses the Gauss-Newton method for nonlinear iterative optimization, minimizes the photometric error instead of the reprojection error, and directly calculates the camera motion information.

[0091] Based on the pinhole imaging principle, the projection equation can be obtained:

[0092]

[0093] Among them, the endoscope left eye intrinsic parameter matrix is ​​A, the depth of the spatial point P relative to the endoscope imaging plane is Z, and the initial estimated relative pose of the camera motion between the k-1th to the kth frame is the rotation matrix R and the translation vector t, and the corresponding Lie algebra is ξ.

[0094] For all spatial points P, the problem of minimizing the photometric error is constructed:

[0095]

[0096] Among them, I(p i,j ) is the corresponding pixel point p in the previous and next frames i luminosity.

[0097] The Lie group is multiplied by a small quantity on the left, and the perturbation model is used to find the rate of change of the Lie algebra relative to the small quantity. Let It can be deduced that:

[0098]

[0099] in is the partial derivative of grayscale with respect to pixel point, is the partial derivative of the pixel point with respect to the spatial point, is the partial derivative of the Lie algebra corresponding to the Lie group of the transformation matrix of the space point pair. Then the Gauss-Newton method is used to calculate the increment and iterate the solution.

[0100] S4: obtaining the endoscope motion trajectory through the transformation matrix obtained in step S3, performing beam adjustment optimization on the endoscope motion trajectory, superimposing the depth map and removing outliers, so as to realize three-dimensional reconstruction of abdominal tissue;

[0101] As a pure optimization problem, it can still be expressed through a graphical model, where the vertices are the optimization variables of all landmark points p extracted on the motion trajectory together with the camera pose ξ, and the edges are motion and observation constraints.

[0102] Assume that the motion trajectory contains m camera poses and n landmarks, and the poses are expressed by Lie algebra, then the optimization objective is:

[0103]

[0104] Among them, z ij represents the actual pixel position of the jth feature point of the ith camera in the original image, ξ i represents the position of the i-th camera, p i Represents the world coordinate position of the i-th feature point.

[0105] The optimized data was obtained by using the LM (Levenberg-Marquard) algorithm and the Schur elimination method. According to the optimized endoscope motion trajectory, the depth map was superimposed at a certain frame interval to obtain a three-dimensional reconstruction of the abdominal tissue.

[0106] S5: Based on the three-dimensional reconstruction of step S4, the lesion is identified, the three-dimensional information is extracted according to the disparity map, and the surgical arm is visually guided to move to the lesion point to achieve positioning in combination with the calibrated hand-eye matrix;

[0107] A needle is fixed on the designed binocular endoscope clamp. The three-dimensional information is extracted according to the disparity map to obtain the conversion relationship between the tool coordinate system and the endoscope coordinate system. During the operation, the lesion area and center point are extracted through color space conversion, threshold segmentation, contour extraction and fitting, etc. After obtaining the disparity map, the hand-eye calibration matrix is ​​combined to obtain the spatial position of the lesion point in the robot base coordinate system, and the vision guides the movement of the surgical instrument to the lesion point.

[0108] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A binocular endoscope-based abdominal reconstruction and lesion localization method, characterized in that: The following steps are involved: S1: Improve the hand-eye calibration of the binocular endoscope internal parameters based on binocular constraints and data loops, obtain the left and right eye images collected by the binocular endoscope and perform corrections based on the obtained internal parameters; S2: Input the corrected left and right eye images in step S1 into the trained depth estimation network based on the disparity attention mechanism, output the disparity map, use the binocular endoscope to collect the video data set of the abdominal phantom or the environment close to the reconstructed tissue, and process it into images frame by frame; S3: The image obtained in step S2 is subjected to a pyramid optimization method from coarse to fine to minimize the photometric error, and a transformation matrix of the endoscope posture between two consecutive frames is calculated by nonlinear iterative optimization; S4: obtaining the endoscope motion trajectory through the transformation matrix obtained in step S3, performing beam adjustment optimization on the endoscope motion trajectory, superimposing the depth map and removing outliers, so as to realize three-dimensional reconstruction of abdominal tissue; S5: Based on the three-dimensional reconstruction of step S4, the lesion is identified, and the three-dimensional information is extracted according to the disparity map. Combined with the calibrated hand-eye matrix, the surgical arm is visually guided to move to the lesion point to achieve positioning.

2. The binocular endoscope-based abdominal reconstruction and lesion localization method according to claim 1, characterized in that: The method for calibrating the internal parameters of the binocular endoscope in step S1 is specifically as follows: the left and right eyes of the binocular endoscope are respectively connected to the computer through the USB interface using a video acquisition card, the calibration board is placed on the base platform of the robotic arm, the position of the end of the robotic arm is changed, and multiple groups of chessboard images and the corresponding angle data of each joint angle of the robotic arm are recorded; at the same time, the focal length, optical center, distortion coefficient and internal parameters of the left and right eye conversion matrix of the binocular endoscope are calibrated.

3. The binocular endoscope-based abdominal reconstruction and lesion localization method according to claim 1, characterized in that: The correction method in step S1 is specifically as follows: constructing the hand-eye calibration equation AX=XB, adding data loop processing, the transformation matrix of the left eye relative to the right eye is obtained through internal parameter calibration and kept constant, and adding binocular constraints to optimize the calibration result.

4. The binocular endoscope-based abdominal reconstruction and lesion localization method according to claim 1, characterized in that: The depth estimation network based on the disparity attention mechanism in step S2 is obtained by offline training through the following method, which specifically includes the following steps: S21: Collect a set of data sets as input; S22: Input the data set of step S21 into the hourglass-shaped feature extraction network to obtain a feature map; S23: Input the feature map obtained in step S22 into the cascaded disparity attention module, and perform matching cost regression in a coarse-to-fine manner; S24: the regression graph obtained in step S23 is passed through an hourglass-type disparity refinement network; S25: Output the disparity map.

5. The binocular endoscope-based abdominal reconstruction and lesion localization method according to claim 4, characterized in that: The specific steps of step S23 are: use the 1×1 convolution obtained in step S22 to extract the left and right eye feature maps I respectively left and I right , size is H×W×C, where H is the batch of matrix multiplications, each matrix has W points, and the feature dimension of each point is C; Perform geometric product and SoftMax function operations on the feature map, and encode the feature similarity of any two positions along the epipolar line into the disparity attention map M right→left and M left→right In the example, the size is H×W×W, and the mask is obtained by removing the occluded pixels according to the point pair matching correlation; The specific steps of step S24 are: using the disparity attention mechanism instead of the cost volume to reduce the amount of calculation and memory usage, and there is no need to set a fixed maximum disparity value, introducing left-right consistency and cycle consistency to obtain a reliable matching relationship, calculating the left-right consistency, and finally outputting the disparity map.

6. The binocular endoscope-based abdominal reconstruction and lesion localization method according to claim 1, characterized in that: The specific steps of step S3 are: based on the assumption that the grayscale value of the same spatial point is the same under different camera postures, focus on the grayscale value gradient of image pixels, adopt a coarse-to-fine pyramid structure to prevent the fast movement speed from falling into the local minimum, perform nonlinear iterative optimization through the Gauss-Newton method, minimize the photometric error instead of the reprojection error, and directly calculate the camera motion information.

7. The binocular endoscope-based abdominal reconstruction and lesion localization method according to claim 1, characterized in that: The specific steps of step S4 are as follows: Assume that the motion trajectory contains m camera poses and n landmarks, and the poses are expressed by Lie algebra, then the optimization objective is: Among them, z ij represents the actual pixel position of the jth feature point of the ith camera in the original image, ξ i represents the position of the i-th camera, p i Represents the world coordinate system position of the i-th feature point; The LM algorithm and the Schur elimination method are used to sort out and solve the optimized data. According to the optimized endoscope motion trajectory, the depth map is superimposed at a certain frame interval to obtain a three-dimensional reconstruction of the abdominal tissue.

8. The binocular endoscope-based abdominal reconstruction and lesion localization method according to claim 1, characterized in that: The specific steps of step S5 are as follows: a needle is fixed on the designed binocular endoscope clamp, and the conversion relationship between the tool coordinate system and the endoscope coordinate system is obtained by extracting three-dimensional information according to the disparity map. During the operation, the lesion area and the center point are extracted by color space conversion, threshold segmentation, contour extraction and fitting, etc. After obtaining the disparity map, the spatial position of the lesion point in the robot base coordinate system is obtained by combining the hand-eye calibration matrix, and the surgical instrument is visually guided to move to the lesion point.

9. The abdominal cavity reconstruction and lesion localization system based on binocular endoscope is characterized by: A binocular endoscope-based abdominal reconstruction and lesion localization method for performing any one of claims 1 to 8, comprising: Binocular endoscope calibration module, used to obtain the internal parameters of the binocular endoscope, such as focal length, optical center, distortion coefficient, left-right eye conversion matrix, and hand-eye calibration matrix of the binocular endoscope coordinate system relative to the coordinate system of the end of the robotic arm; An image acquisition and correction module is used to obtain a binocular endoscope synchronously acquire a left-eye image sequence and a right-eye image sequence, and use the calibrated internal parameter matrix to perform image correction; The depth estimation network training module is used to pre-train the deep learning model with the collected data set to obtain the model parameters; A disparity map acquisition module, used to acquire a disparity map of a captured image in real time; The endoscope motion trajectory calculation module is used to calculate the pose transformation matrix between the current frame and the next frame according to the coarse-to-fine minimization of the photometric error between the two frames to obtain the motion trajectory of the endoscope; The abdominal tissue 3D reconstruction module is used to optimize the motion trajectory of the endoscope by using the bundle adjustment method, superimpose the 3D point cloud calculated by the disparity map and remove outliers to complete the 3D reconstruction of the abdominal tissue; The lesion recognition and vision-guided surgical arm positioning module is used to identify the lesion area, extract three-dimensional information based on the disparity map, and combine it with the calibrated hand-eye matrix to visually guide the surgical arm to move to the lesion point for positioning.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the binocular endoscope-based abdominal reconstruction and lesion localization method as described in any one of claims 1-8 is implemented.

Images