Bipolar electrosurgical device

Abstract

The application belongs to the technical field of medical image three-dimensional reconstruction, and discloses a laparocavity reconstruction and lesion positioning method based on a binocular endoscope, which comprises internal parameter calibration and correction of the binocular endoscope; a trained depth estimation network is input to obtain a parallax map and process the parallax map into a picture according to frames; a coarse-to-fine pyramid optimization method is used to minimize photometric error, and a nonlinear iterative optimization calculation is used to calculate a transformation matrix of endoscope poses between two continuous frames; the endoscope motion trajectory is optimized by bundle adjustment, a depth map is superimposed, and outliers are removed, so that three-dimensional reconstruction of the laparocavity tissue is realized; a lesion is identified, three-dimensional information is extracted according to the parallax map, a hand-eye matrix is combined, a surgical arm is moved to the lesion point under visual guidance, and positioning is realized. The application realizes fast and dense three-dimensional reconstruction of the preoperative laparocavity environment and precise positioning of the lesion based on visual guidance during the operation, improves the accuracy of endoscope motion trajectory estimation, and is expected to promote the development of surgical robot autonomy.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of medical image three-dimensional reconstruction, in particular to a laparum reconstruction and lesion positioning method, system and device based on binocular endoscopes. BACKGROUND

[0002] Minimally invasive surgery has been widely used in clinical practice in recent decades due to its advantages of small trauma, low infection rate, few complications, light pain and fast recovery, which has promoted a milestone progress in medicine. However, compared with traditional open surgery, it has problems such as limited field of view, inaccurate endoscope positioning and lack of surrounding environment information. The traditional monocular laparoscope commonly used in minimally invasive surgery can only capture two-dimensional scenes, which leads to insufficient estimation of the pushing distance of surgical instruments and the depth of the body cavity, and cannot intuitively and stereoscopically display the relationship between lesions, blood vessels and surrounding organ tissues in the abdominal cavity environment, greatly increasing the difficulty and complexity of surgery and relying more on the experience of doctors.

[0003] With the maturity of binocular vision theory and the development of computer technology, using binocular endoscopes to calculate the depth information of the body cavity environment, stereoscopic positioning of the lesion area, real-time and dense three-dimensional reconstruction of the collected body cavity information by computer, using Simultaneous Localization and Mapping (SLAM) technology to establish an abdominal organ tissue model and track the laparoscope trajectory, more accurately and realistically reflect the abdominal tissue structure and lesion spatial relationship, provide visual depth information for doctors, realize endoscope positioning and navigation, and greatly improve the scientificity and success rate of laparoscopic minimally invasive surgery. And the endoscope cannot observe the information under the surface of the organ. After using SLAM to quickly and densely reconstruct the abdominal cavity environment, it can be fused and registered with the preoperative CT or MRI image, and the patient's anatomical structure can be displayed during the operation. At the same time, it can be combined with AR technology to superimpose additional information in the three-dimensional scene, such as annotations of target lesions, contours of organs, measurements of tumors, etc., which can also achieve the expansion of the current endoscopic field of view. With the development of robot technology, the autonomy of the surgical process is an inevitable trend, and the rapid and dense three-dimensional reconstruction of abdominal tissue and lesion positioning promote the completion of autonomous surgical operations by surgical robots. Therefore, there is an urgent need for a laparum reconstruction and lesion positioning method based on binocular endoscopes. SUMMARY

[0004] In view of the above defects or improvement needs of the prior art, the present application provides a laparum reconstruction and lesion positioning method based on binocular endoscopes.

[0005] To achieve the above-mentioned purpose, according to one aspect of the present application, a laparum reconstruction and lesion positioning method based on binocular endoscopes is provided, comprising the following steps:

[0006] S1: Improve the hand-eye calibration of the binocular endoscope based on the binocular constraint and data cycle, obtain the left and right eye images collected by the binocular endoscope, and correct them according to the obtained internal parameters;

[0007] S2: Input the corrected left and right eye images in step S1 into the depth estimation network based on the disparity attention mechanism, output the disparity map, collect the video data set of the abdominal cavity model or the environment similar to the reconstructed tissue using the binocular endoscope, and process it into pictures according to the frame;

[0008] S3: Minimize the photometric error of the pictures obtained in step S2 according to the coarse-to-fine pyramid optimization method, and non-linearly iteratively optimize the transformation matrix of the endoscope pose between two consecutive frames;

[0009] S4: Obtain the endoscope motion trajectory by the transformation matrix obtained in step S3, optimize the endoscope motion trajectory by bundle adjustment, superimpose the depth map and remove outliers, and realize the three-dimensional reconstruction of the abdominal tissue;

[0010] S5: Based on the three-dimensional reconstruction in step S4, identify the lesion, extract three-dimensional information according to the disparity map, and combine the calibrated hand-eye matrix to guide the movement of the surgical arm to the lesion point to realize positioning.

[0011] As a preferred, the method of calibrating the internal parameters of the binocular endoscope in step S1 is as follows: the left and right eyes of the binocular endoscope are connected to the computer through USB interface using video capture card, the calibration board is placed on the mechanical arm base platform, the end position of the mechanical arm is changed, and multiple groups of checkerboard pictures and corresponding angle data of each joint angle of the mechanical arm are recorded; At the same time, the focal length, optical center, distortion coefficient and internal parameters of the conversion matrix of the left and right eyes of the binocular endoscope are calibrated.

[0012] As a preferred, the correction method in step S1 is as follows: an AX=XB hand-eye calibration equation is constructed, data cycle processing is added, the conversion matrix of the left eye relative to the right eye is obtained by internal parameter calibration and remains constant, and binocular constraint is added to optimize the calibration result.

[0013] As a preferred, the depth estimation network based on the disparity attention mechanism in step S2 is obtained by offline training through the following method, which specifically includes the following steps:

[0014] S21: Collect a group of data sets as input;

[0015] S22: Input the data set in step S21 into the hourglass type feature extraction network to obtain the feature map;

[0016] S23: Input the feature map obtained in step S22 into the cascaded disparity attention module, and use the coarse-to-fine method to perform matching cost regression;

[0017] S24: The regression plot obtained in step S23 is processed by an hourglass-shaped parallax refinement network;

[0018] S25: Output disparity map.

[0019] Preferably, step S23 involves extracting the left and right eye feature maps I using the 1×1 convolution obtained in step S22. left and I right The size is H×W×C, where H is the batch processing of matrix multiplication, each matrix has W points, and the feature dimension of each point is C;

[0020] Perform geometric product and SoftMax function operations on the feature map to encode the feature similarity between any two locations along the epipolar line into the disparity attention map M. right→left and M left→right In the middle, the size is H×W×W, and the mask is obtained by removing occluded pixels according to the point pair matching correlation.

[0021] The parallax attention mechanism is used instead of the cost volume to reduce computation and memory usage. It does not require setting a fixed maximum parallax value. It introduces left-right consistency and cycle consistency to obtain reliable matching relationships, calculates left-right consistency, and finally outputs a parallax map.

[0022] As a preferred option, the specific steps of step S3 are as follows: based on the assumption that the gray value of the same spatial point is the same under different camera poses, focus on the gray value gradient of image pixels, adopt a pyramid structure from coarse to fine to prevent the fast motion speed from getting trapped in local minima, perform nonlinear iterative optimization through the Gauss-Newton method, minimize photometric error instead of reprojection error, and directly calculate camera motion information.

[0023] As a preferred option, step S4 is as follows: Let the motion trajectory include m camera poses and n landmarks, and let the poses be represented by Lie algebras. Then the optimization objective is:

[0024]

[0025] Among them, z ij ξ represents the actual pixel position of the j-th feature point of the i-th camera in the original image. i p represents the pose of the i-th camera. i This represents the world coordinate system position of the i-th feature point. The LM algorithm is used, and the solution is refined using Schur elimination to obtain optimized data. Following the optimized endoscopic motion trajectory, depth maps are superimposed at regular intervals to obtain a 3D reconstruction of the abdominal tissue.

[0026] As a preferred option, the specific steps of step S5 are as follows: a needle is fixed on the designed binocular endoscope clamp; three-dimensional information is extracted based on the disparity map to obtain the transformation relationship between the tool coordinate system and the endoscope coordinate system; during the operation, the lesion area and center point are extracted by methods such as color space conversion, threshold segmentation, and contour extraction fitting; after obtaining the disparity map, the spatial position of the lesion point in the robot base coordinate system is obtained by combining the hand-eye calibration matrix; and the surgical instruments are visually guided to move to the lesion point.

[0027] To achieve the above objectives, according to one aspect of the present invention, a binocular endoscope-based abdominal cavity reconstruction and lesion localization system is provided for performing the binocular endoscope-based abdominal cavity reconstruction and lesion localization method, comprising:

[0028] The binocular endoscope calibration module is used to obtain intrinsic parameters such as focal length, optical center, distortion coefficient, and left-right eye transformation matrix of the binocular endoscope, as well as the hand-eye calibration matrix of the binocular endoscope coordinate system relative to the robotic arm end-effector coordinate system.

[0029] The image acquisition and correction module is used to acquire the left and right eye image sequences simultaneously acquired by binocular endoscopy, and to perform image correction using the calibrated intrinsic parameter matrix.

[0030] The deep estimation network training module is used to pre-train a deep learning model on the collected dataset to obtain the model parameters;

[0031] The disparity map acquisition module is used to acquire the disparity map of the acquired image in real time.

[0032] The endoscope motion trajectory calculation module is used to calculate the pose transformation matrix between two frames based on minimizing photometric error from coarse to fine in the current frame and the next frame, so as to obtain the motion trajectory of the endoscope.

[0033] The abdominal tissue 3D reconstruction module is used to optimize the motion trajectory of the endoscope using the bundle adjustment method, overlay the 3D point cloud calculated by the disparity map and remove outliers to complete the 3D reconstruction of the abdominal tissue.

[0034] The lesion identification and visually guided surgical arm positioning module is used to identify the lesion area, extract three-dimensional information based on the disparity map, and combine it with the calibrated hand-eye matrix to visually guide the surgical arm to move to the lesion point to achieve positioning.

[0035] To achieve the above objectives, according to one aspect of the present invention, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the aforementioned method for abdominal cavity reconstruction and lesion localization based on binocular endoscopy.

[0036] This invention focuses on acquiring visual information through binocular laparoscopy, using the Zhang Zhengyou calibration method to obtain the intrinsic parameters of the binocular laparoscopy, achieving sub-pixel-level reprojection accuracy that meets medical requirements. It also introduces binocular constraints and data looping to improve the accuracy of the hand-eye calibration algorithm based on binocular endoscopes. Epipolar correction is applied to the acquired left and right eye views. Depth information is obtained by fine-tuning a binocular unsupervised network based on a parallax attention mechanism, and a pseudo-RGBD sequence is obtained by aligning timestamps. Simultaneously, a coarse-to-fine pyramid optimization method is used to minimize photometric errors and parallel process the trajectory of the binocular endoscope. This enables rapid and dense 3D reconstruction of the acquired abdominal cavity images and precise lesion localization, providing surgeons or surgical robots with global environmental prior information, better understanding the location and relationship of lesions and blood vessels, and further reducing intraoperative risks. Furthermore, the 3D reconstructed point cloud can be fused and registered with preoperative diagnostic CT or MRI images, revealing anatomical structures invisible beneath the surface of patient organs during surgery. This technology can be subsequently used in AR and VR fields, training novice surgeons, and promoting the development of autonomous laparoscopic robotic surgery in the future.

[0037] In summary, the technical solutions conceived by this invention have the following beneficial effects compared with the prior art:

[0038] (1) Introduce binocular constraints and data loops to improve the hand-eye calibration algorithm based on binocular endoscopes;

[0039] (2) A binocular image unsupervised learning algorithm based on parallax attention mechanism to obtain depth information;

[0040] (3) Pyramid optimization from coarse to fine to minimize photometric error and track endoscope trajectory through parallel processing;

[0041] (4) It can quickly and densely reconstruct the preoperative abdominal environment and accurately locate the lesion during the operation. Attached Figure Description

[0042] Fig. 1 This is a flowchart of the method for abdominal cavity reconstruction and lesion localization based on binocular endoscopy provided by the present invention;

[0043] Fig. 2 This is a flowchart of the unsupervised binocular depth estimation method based on disparity attention mechanism provided by the present invention;

[0044] Fig. 3 This invention provides the endoscope's motion trajectory and three-dimensional reconstruction results. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0046] Before the system runs, only one intrinsic parameter calibration and hand-eye calibration of the binocular endoscope are required, and a set of left and right eye image datasets of the scene are collected to fine-tune the unsupervised depth estimation network. Preoperatively, the reconstructed video acquired by the binocular endoscope is uploaded in real-time to the computer GPU via a capture card. After obtaining the depth map through the trained depth estimation network, pseudo-RGBD frames are created and sequentially fed to the SLAM algorithm for parallel processing to estimate the trajectory of the binocular endoscope and complete the dense reconstruction of the abdominal tissue. Intraoperatively, lesion pixels are extracted using methods such as threshold segmentation and contour extraction. Based on the depth map and the hand-eye calibration matrix, spatial localization in the robot's base coordinate system is performed, and the surgical instruments are visually guided to move to the lesion point.

[0047] Please see Figs. 1-3 The present invention provides a method, system, and device for abdominal cavity reconstruction and lesion localization based on binocular endoscopy. The system for abdominal cavity reconstruction and lesion localization based on binocular endoscopy is used to execute the aforementioned method for abdominal cavity reconstruction and lesion localization based on binocular endoscopy, and includes:

[0048] The binocular endoscope calibration module is used to obtain intrinsic parameters such as focal length, optical center, distortion coefficient, and left-right eye transformation matrix of the binocular endoscope, as well as the hand-eye calibration matrix of the binocular endoscope coordinate system relative to the robotic arm end-effector coordinate system.

[0049] The image acquisition and correction module is used to acquire the left and right eye image sequences simultaneously acquired by binocular endoscopy, and to perform image correction using the calibrated intrinsic parameter matrix.

[0050] The deep estimation network training module is used to pre-train a deep learning model on the collected dataset to obtain the model parameters;

[0051] The disparity map acquisition module is used to acquire the disparity map of the acquired image in real time.

[0052] The endoscope motion trajectory calculation module is used to calculate the pose transformation matrix between two frames based on minimizing photometric error from coarse to fine in the current frame and the next frame, so as to obtain the motion trajectory of the endoscope.

[0053] The abdominal tissue 3D reconstruction module is used to optimize the motion trajectory of the endoscope using the bundle adjustment method, overlay the 3D point cloud calculated by the disparity map and remove outliers to complete the 3D reconstruction of the abdominal tissue.

[0054] The lesion identification and visually guided surgical arm positioning module is used to identify the lesion area, extract three-dimensional information based on the disparity map, and combine it with the calibrated hand-eye matrix to visually guide the surgical arm to move to the lesion point to achieve positioning.

[0055] The computer device includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the described method for abdominal cavity reconstruction and lesion localization based on binocular endoscopy.

[0056] The method for laparoscopic reconstruction and lesion localization based on binocular endoscopy includes the following steps:

[0057] S1: Improve the hand-eye calibration binocular endoscope intrinsic parameters based on binocular constraints and data loops, acquire the left and right eye images acquired by the binocular endoscope and perform correction based on the acquired intrinsic parameters;

[0058] The binocular endoscope is connected to the computer via a USB interface using video capture cards for both the left and right eyes. The calibration board is placed on the robotic arm base platform. The pose of the robotic arm end is changed to record multiple sets of chessboard images and the corresponding angle data of each joint angle of the robotic arm. At the same time, the intrinsic parameters of the binocular endoscope, such as focal length, optical center, distortion coefficient, and left-right eye conversion matrix, are calibrated.

[0059] The specific calibration method is as follows: An AX = XB hand-eye calibration equation is constructed. To ensure accurate iterative solution of X with less data, a data loop processing method is added instead of sequential processing. The transformation matrix between the left and right eyes is obtained through intrinsic parameter calibration and remains constant; therefore, binocular constraints are added to optimize the calibration results.

[0060]

[0061] in, Let A represent the transformation matrix from coordinate system j to coordinate system i. Let A be the product of any two transformation matrices of the robot end effector relative to the base. Let B be the product of any two transformation matrices of the checkerboard corner points relative to the endoscope. The subscript l represents the left eye coordinate system, and the subscript r represents the right eye coordinate system. The transformation matrix is ​​solved using the Perspective-n-Point (PnP) algorithm. The range of i is the number of calibration images acquired, and the range of j is the number of calibration images acquired minus 1. The algorithm iterates j times for each i.

[0062] The binocular endoscope uses separate video capture cards connected to the computer via USB to acquire video streams in real time for both the left and right eyes. The images acquired by the binocular endoscope are corrected using a calibrated intrinsic parameter matrix. The images are then cropped at the center, and the values ​​u0 and v0 representing the optical center pixel positions in the camera's intrinsic parameters are modified accordingly.

[0063] S2: Collect a dataset to train an unsupervised depth estimation network based on the disparity attention mechanism. Input the corrected left and right eye images from step S1 into the trained depth estimation network and output a disparity map. Use a binocular endoscope to collect video datasets of an abdominal phantom or an environment similar to the reconstructed tissue and process them into images frame by frame.

[0064] The depth estimation network based on the parallax attention mechanism is trained offline using the following method, specifically including the following steps:

[0065] S21: Collect a dataset as input;

[0066] The images of the left and right eyes acquired by the binocular endoscope are corrected using the calibrated intrinsic parameter matrix; the images are then cropped at the center, and the u0 and v0 representing the optical center pixel positions in the camera intrinsic parameters are modified accordingly.

[0067] S22: Input the dataset from step S21 into an hourglass-shaped feature extraction network, extracting feature maps from the left and right eye images from coarse to fine and then from fine to coarse.

[0068] S23: Input the feature map obtained in step S22 into the cascaded disparity attention module and perform matching cost regression in a coarse-to-fine manner;

[0069] The 1×1 convolution obtained in step S22 is used to extract the feature maps I of the left and right eyes respectively. left and I right The size is H×W×C, where H is the batch processing of matrix multiplication, each matrix has W points, and the feature dimension of each point is C;

[0070] Perform geometric product and SoftMax function operations on the feature map to encode the feature similarity between any two locations along the epipolar line into the disparity attention map M. right→left and M left→right In the context of recurrent attention graph M left→right→left and M right→left→right In addition, the mask is obtained by removing occluded pixels based on the correlation of point pair matching;

[0071] The parallax attention mechanism is used instead of the cost volume to reduce computation and memory usage. It does not require setting a fixed maximum parallax value. It introduces left-right consistency and cycle consistency to obtain reliable matching relationships, calculates left-right consistency, and finally outputs a parallax map.

[0072]

[0073] Calculate cycle consistency:

[0074]

[0075] S24: The regression plot obtained in step S23 is processed by an hourglass-shaped parallax refinement network;

[0076] Input the calculated initial disparity map and initial feature map, and use the features of the left image as a guide to provide structural information such as edges, first from coarse to fine, and then from fine to coarse to refine the disparity.

[0077] The loss function of this unsupervised network includes photometric loss. Smoothness loss and parallax attention mechanism loss

[0078]

[0079] Where, λ p ,λ s ,λ PAM These are their respective weights.

[0080] The parallax attention mechanism loss introduces three terms at different scales to regularize the PAM loss to obtain accurate and consistent stereo correlation, namely the PAM photometric loss:

[0081]

[0082] PAM smoothness loss:

[0083]

[0084] PAM cycle consistency loss:

[0085]

[0086] Where p represents the effective pixels covered by the effective mask, and N represents the number of effective pixels. and They are and Valid pixels, and They are and The number of pixels, in For example, It is described using feature similarity. In the case of (i,k) relative to The contribution of (i,j) to I s It is an identity matrix.

[0087] S25: Output disparity map;

[0088] During abdominal cavity reconstruction and lesion localization, based on the model parameters trained by the aforementioned depth estimation learning network, the corrected left and right eye images are input, and a disparity map is output. The left image is combined with the corresponding output disparity map to obtain a pseudo-RGBD frame.

[0089] S3: Minimize the photometric error of the images processed frame by frame in step S2 using a coarse-to-fine pyramid optimization method, and calculate the transformation matrix of the endoscope pose between two consecutive frames using nonlinear iterative optimization.

[0090] SLAM uses a dense direct method that does not compute feature points and descriptors, making it suitable for textureless ventral environments. Based on the assumption that the grayscale value of the same spatial point is the same under different camera poses, it focuses on the grayscale gradient of image pixels, employs a coarse-to-fine pyramid structure to prevent the camera from getting trapped in local minima due to high motion speed, and uses the Gauss-Newton method for nonlinear iterative optimization to minimize photometric error rather than reprojection error, directly calculating camera motion information.

[0091] Based on the principle of pinhole imaging, the projection equation can be obtained:

[0092]

[0093] Wherein, the intrinsic parameter matrix of the left eye of the endoscope is A, the depth of the spatial point P relative to the imaging plane of the endoscope is Z, and the initial estimated relative pose of the camera motion between the (k-1)th and the kth frame is the rotation matrix R and the translation vector t, with the corresponding Lie algebra ξ.

[0094] For all spatial points P, construct a problem to minimize the photometric error:

[0095]

[0096] Among them, I(p) i,j ) is the corresponding pixel point p in the two consecutive frames. i luminosity.

[0097] The Lie group is left-multiplied by a small quantity, and the rate of change of the Lie algebra relative to the small quantity is obtained using a perturbation model. Let... It can be deduced that:

[0098]

[0099] in This is the partial derivative of grayscale with respect to the pixel. The partial derivative of a pixel with respect to a point in space. Let be the partial derivative of the Lie algebra corresponding to the transformation matrix of the spatial point. Then, the increment is calculated using the Gauss-Newton method, and the solution is obtained iteratively.

[0100] S4: Obtain the endoscope motion trajectory through the transformation matrix obtained in step S3, optimize the endoscope motion trajectory by beam adjustment, overlay the depth map and remove outliers to achieve three-dimensional reconstruction of abdominal tissues.

[0101] As a pure optimization problem, it can still be expressed by a graph model, where the vertices are the optimization variables of all the landmark points p extracted on the motion trajectory together with the camera pose ξ, and the edges are the motion and observation constraints.

[0102] Suppose the motion trajectory includes m camera poses and n landmarks, and the poses are represented by Lie algebras. Then the optimization objective is:

[0103]

[0104] Among them, z ij ξ represents the actual pixel position of the j-th feature point of the i-th camera in the original image. i p represents the pose of the i-th camera. i This represents the world coordinate system position of the i-th feature point.

[0105] The LM (Levenberg-Marquard) algorithm was used, and the Schur elimination method was used to refine and solve the data. The optimized data was obtained by overlaying depth maps at regular intervals according to the optimized endoscopic motion trajectory to obtain a three-dimensional reconstruction of the abdominal tissue.

[0106] S5: Based on the three-dimensional reconstruction in step S4, identify the lesion, extract three-dimensional information according to the disparity map, and combine it with the calibrated hand-eye matrix to visually guide the surgical arm to move to the lesion point to achieve localization.

[0107] A needle is fixed on the designed binocular endoscope clamp. The three-dimensional information is extracted based on the disparity map to obtain the transformation relationship between the tool coordinate system and the endoscope coordinate system. During the operation, the lesion area and center point are extracted by methods such as color space transformation, threshold segmentation, and contour extraction and fitting. After obtaining the disparity map, the spatial position of the lesion point in the robot base coordinate system is obtained by combining the hand-eye calibration matrix. The surgical instruments are visually guided to move to the lesion point.

[0108] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for laparoscopic reconstruction and lesion localization based on binocular endoscopy, characterized in that, Includes the following steps: S1: Improve the hand-eye calibration binocular endoscope intrinsic parameters based on binocular constraints and data loops, acquire the left and right eye images acquired by the binocular endoscope and perform correction based on the acquired intrinsic parameters; S2: Input the corrected left and right eye images from step S1 into the trained depth estimation network based on the disparity attention mechanism, output the disparity map, use a binocular endoscope to collect video datasets of the abdominal cavity phantom or an environment similar to the reconstructed tissue, and process them into images frame by frame. S3: Minimize the photometric error of the image obtained in step S2 using a pyramid optimization method from coarse to fine, and calculate the transformation matrix of the endoscope pose between two consecutive frames using nonlinear iterative optimization. S4: Obtain the endoscope motion trajectory through the transformation matrix obtained in step S3, optimize the endoscope motion trajectory by beam adjustment, overlay the depth map and remove outliers to achieve three-dimensional reconstruction of abdominal tissues. The specific steps of step S4 are as follows: Suppose the motion trajectory includes m camera poses and n landmarks, and the poses are represented by Lie algebras. Then the optimization objective is: Where, z ij This represents the actual pixel position of the j-th feature point of the i-th camera in the original image. p represents the pose of the i-th camera. i This represents the world coordinate system position of the i-th feature point; The LM algorithm was used and the Schur elimination method was used to solve the problem and obtain optimized data. According to the optimized endoscope motion trajectory, depth maps were superimposed at regular intervals to obtain a three-dimensional reconstruction of the abdominal tissue. S5: Based on the three-dimensional reconstruction in step S4, identify the lesion, extract three-dimensional information according to the disparity map, and combine it with the calibrated hand-eye matrix to visually guide the surgical arm to move to the lesion point to achieve localization.

2. The method for abdominal cavity reconstruction and lesion localization based on binocular endoscopy as described in claim 1, characterized in that, The specific method for calibrating the intrinsic parameters of the binocular endoscope in step S1 is as follows: The left and right eyes of the binocular endoscope are connected to the computer via a USB interface using video capture cards. The calibration board is placed on the robotic arm base platform. The pose of the robotic arm end is changed, and multiple sets of chessboard images and the corresponding angle data of each joint angle of the robotic arm are recorded. At the same time, the intrinsic parameters of the binocular endoscope, such as focal length, optical center, distortion coefficient, and left-right eye conversion matrix, are calibrated.

3. The method for abdominal cavity reconstruction and lesion localization based on binocular endoscopy as described in claim 1, characterized in that, The depth estimation network based on the disparity attention mechanism in step S2 is trained offline using the following method, specifically including the following steps: S21: Collect a dataset as input; S22: Input the dataset from step S21 into an hourglass-shaped feature extraction network to obtain feature maps; S23: Input the feature map obtained in step S22 into the cascaded disparity attention module and perform matching cost regression in a coarse-to-fine manner; S24: The regression plot obtained in step S23 is processed by an hourglass-shaped parallax refinement network; S25: Output disparity map.

4. The method for abdominal cavity reconstruction and lesion localization based on binocular endoscopy as described in claim 3, characterized in that, The specific steps of step S23 are as follows: Use the 1×1 convolution obtained in step S22 to extract the feature maps of the left and right eyes respectively. and The size is H×W×C, where H is the batch processing of matrix multiplication, each matrix has W points, and the feature dimension of each point is C; Perform geometric product and SoftMax function operations on the feature map to encode the feature similarity between any two locations along the epipolar line into the disparity attention map. and In the middle, the size is H×W×W, and the mask is obtained by removing occluded pixels according to the point pair matching correlation. The specific steps of step S24 are as follows: Use the disparity attention mechanism to replace the cost volume, reduce the amount of computation and memory usage, and do not require setting a fixed maximum disparity value. Introduce left-right consistency and cycle consistency to obtain a reliable matching relationship, calculate left-right consistency, and finally output the disparity map.

5. The method for abdominal cavity reconstruction and lesion localization based on binocular endoscopy as described in claim 1, characterized in that, The specific steps of step S3 are as follows: based on the assumption that the gray value of the same spatial point is the same under different camera poses, focus on the gray value gradient of image pixels, adopt a coarse-to-fine pyramid structure to prevent the fast motion speed from getting trapped in local minima, perform nonlinear iterative optimization through the Gauss-Newton method, minimize photometric error instead of reprojection error, and directly calculate camera motion information.

6. The method for abdominal cavity reconstruction and lesion localization based on binocular endoscopy as described in claim 1, characterized in that, The specific steps of step S5 are as follows: A needle is fixed on the designed binocular endoscope clamp. The three-dimensional information is extracted according to the disparity map to obtain the transformation relationship between the tool coordinate system and the endoscope coordinate system. During the operation, the lesion area and center point are extracted by color space conversion, threshold segmentation, contour extraction and fitting methods. After obtaining the disparity map, the spatial position of the lesion point in the robot base coordinate system is obtained by combining the hand-eye calibration matrix. The surgical instruments are visually guided to move to the lesion point.

7. A system for laparoscopic reconstruction and lesion localization based on binocular endoscopy, characterized in that, A method for performing laparoscopic reconstruction and lesion localization based on binocular endoscopy as described in any one of claims 1-6, comprising: The binocular endoscope calibration module is used to obtain the focal length, optical center, distortion coefficient, and left-right eye transformation matrix intrinsic parameters of the binocular endoscope, as well as the hand-eye calibration matrix of the binocular endoscope coordinate system relative to the robotic arm end-effector coordinate system. The image acquisition and correction module is used to acquire the left and right eye image sequences simultaneously acquired by binocular endoscopy, and to perform image correction using the calibrated intrinsic parameter matrix. The deep estimation network training module is used to pre-train a deep learning model on the collected dataset to obtain the model parameters; The disparity map acquisition module is used to acquire the disparity map of the acquired image in real time. The endoscope motion trajectory calculation module is used to calculate the pose transformation matrix between two frames based on minimizing photometric error from coarse to fine in the current frame and the next frame, so as to obtain the motion trajectory of the endoscope. The abdominal tissue 3D reconstruction module is used to optimize the motion trajectory of the endoscope using the bundle adjustment method, overlay the 3D point cloud calculated by the disparity map and remove outliers to complete the 3D reconstruction of the abdominal tissue. The lesion identification and visually guided surgical arm positioning module is used to identify the lesion area, extract three-dimensional information based on the disparity map, and combine it with the calibrated hand-eye matrix to visually guide the surgical arm to move to the lesion point to achieve positioning.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method for abdominal cavity reconstruction and lesion localization based on binocular endoscopy as described in any one of claims 1-6.

Images