Self-supervised deep learning-based binocular three-dimensional image reconstruction and display method capable of tracking lesion position and emphatically displaying lesion image
Through self-supervised deep learning and convolutional neural network training of the MVS framework, the problem of weak texture features of three-dimensional endoscopic images was solved, efficient lesion location tracking and three-dimensional reconstruction were achieved, and the surgical effect and safety were improved.
Patent Information
- Application Number
- CN202510896219.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-30
- Publication Date
- 2025-10-03
AI Technical Summary
The texture features of existing three-dimensional endoscopic images are weak, and the background and lesion targets have similar colors, which are prone to confusion and affect the surgical effect.
A self-supervised deep learning method was used to construct a convolutional neural network based on the MVS framework, which was trained with the ScanNet dataset to achieve real-time lesion location tracking and three-dimensional image reconstruction with binocular endoscopes.
It improves image quality, enhances lesion detection capabilities, assists doctors in navigation and positioning, reduces surgical errors and complication risks, and reduces computing costs and time.
Smart Images
Figure FT_1 
Figure FT_2 
Figure FT_3
Abstract
Description
Technical Field
[0001] The present invention relates to a binocular three-dimensional image reconstruction and display method based on self-supervised deep learning, which can track the location of lesions and focus on displaying lesion images. Background Art
[0002] Compared to traditional endoscopes, 3D endoscopes can provide three-dimensional information and better reflect the actual situation of the scene. On the one hand, they can allow doctors to experience the visual effects of three-dimensional images during surgery, providing a realistic visual experience and helping doctors to operate the endoscope more accurately. On the other hand, 3D endoscopes can integrate 3D measurement functions. They can not only detect the distance between the endoscope tip and the observed tissue, avoiding the risk of the endoscope getting too close or accidentally contacting the tissue to be inspected when moving the endoscope, but also perform accurate 3D measurements of some lesions. Compared to traditional surgical procedures where doctors rely on experience to judge the size of lesions, the results of 3D measurement undoubtedly provide doctors with a quick and reliable reference, reducing their burden and improving the effectiveness of the surgery.
[0003] Currently, 3D reconstruction typically uses a monocular or binocular camera to obtain left and right views of a scene, then uses a binocular matching algorithm to generate a disparity map. This is then used to calculate the depth map, obtaining the 3D coordinates of each point in the image for 3D reconstruction. This method is simple and convenient, but the resulting endoscopic 3D images have weak texture features, and the background and lesion targets are similar in color, making them easily confused. This can affect the doctor's observation of the lesion and complicate the surgical procedure.
[0004] Deep learning, particularly models such as convolutional neural networks (CNNs) and generative adversarial networks (GANs), has brought new breakthroughs in 3D reconstruction. With the advancement of deep learning technology and the increasing availability of large training datasets, a new generation of methods has emerged that can recover the 3D structure of objects from one or more RGB images without the need for complex camera calibration. However, due to the unique use cases and highly miniaturized structure of endoscopes, it is not easy to construct large and accurate supervised learning datasets, making it impossible to train neural networks using supervised learning methods. Summary of the Invention
[0005] In order to overcome the shortcomings of the existing technology, the purpose of the present invention is to provide a method for real-time tracking of lesion positions by binocular endoscopes based on self-supervised deep learning and dynamically displaying lesion images in the form of three-dimensional image reconstruction, which is used to solve the technical problems of the existing technology that the texture features of the three-dimensional images of binocular endoscopes are weak and the background and lesion target colors are similar, which is very easy to be confused, thereby achieving the purpose of improving image quality and enhancing the ability to detect lesions. The method comprises the following steps:
[0006] Step S100 : Using the same image acquisition functional unit as a binocular endoscope to build a binocular endoscope camera system for collecting binocular endoscope images.
[0007] In step S200, a binocular endoscope camera system is used to collect images, and a binocular stereoscopic vision depth calculation formula is used to calculate the depth of each pixel in the image. The binocular stereoscopic vision depth calculation formula is:
[0008]
[0009] in, is the baseline length, is the camera focal length, For parallax.
[0010] Step S300: construct a convolutional neural network based on the MVS framework.
[0011] Step S400: Download the public dataset ScanNet dataset (jointly developed by Stanford University, Princeton University, and Technical University of Munich, suitable for research and development of algorithms such as indoor scene segmentation, recognition, object detection, and scene reconstruction) and perform the first round of training on the convolutional neural network based on the MVS framework.
[0012] In step S500, the binocular image collected in step S200 and the corresponding calculated depth map are used as the data set for the second round of training, and the convolutional neural network based on the MVS framework is trained for the second round.
[0013] In step S600, the image captured by the binocular endoscope software camera system is input into a convolutional neural network based on the MVS framework, and the output is a three-dimensional image and real-time tracking of the trajectory of the binocular endoscope on the three-dimensional image.
[0014] It can be seen that compared with the prior art, the present invention has the following beneficial effects:
[0015] (1) Self-supervised training mode: The self-supervised training mode can achieve similar results to supervised training;
[0016] (2) Assisted navigation and positioning: It can be used to assist doctors in navigation and positioning, especially in minimally invasive surgery. Doctors can more accurately locate and orient the treatment area, reducing errors and the risk of damaging surrounding healthy tissues.
[0017] (3) Improve surgical results: By providing clearer visualization and more accurate diagnosis of lesions, binocular endoscopes can track the location of lesions in real time and dynamically display lesion images in the form of three-dimensional image reconstruction, which helps improve surgical results, reduce the risk of surgical complications, and shorten recovery time.
[0018] (4) Low computational cost and short time: Pre-training the artificial neural network through deep learning can improve the prediction accuracy while reducing the computational cost, thereby reducing the running time of the 3D reconstruction image. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] Figure 1 This is a flowchart of an embodiment of a method for real-time tracking of lesion positions with a binocular endoscope and dynamic display of lesion images in the form of three-dimensional image reconstruction based on self-supervised deep learning of the present invention.
[0020] Figure 2 It is a self-supervised deep learning network framework based on the MVS framework in a method of the present invention for real-time tracking of lesion positions with a binocular endoscope and dynamic display of lesion images in the form of three-dimensional image reconstruction based on self-supervised deep learning.
[0021] Figure 3 It is the network structure of the VGG (a classic convolutional neural network (CNN) model proposed by the Visual Geometry Group of the University of Oxford) module in the self-supervised deep learning network framework based on the MVS framework in the method of the present invention for real-time tracking of lesion positions with a binocular endoscope and dynamic display of lesion images in the form of three-dimensional image reconstruction based on self-supervised deep learning.
[0022] Figure 4 It is the network structure of the double convolution module in the VGG (the classic convolutional neural network CNN model proposed by the Visual Geometry Group of the University of Oxford) module in the self-supervised deep learning network framework based on the MVS framework in the method of the present invention for real-time tracking of lesion positions with a binocular endoscope and dynamically displaying lesion images in the form of three-dimensional image reconstruction based on self-supervised deep learning.
[0023] Figure 5 It is a network structure of three convolution modules in the VGG (a classic convolutional neural network CNN model proposed by the Visual Geometry Group of Oxford University) module in the self-supervised deep learning network framework based on the MVS framework in the method of the present invention for binocular endoscope real-time tracking of lesion positions and dynamic display of lesion images in the form of three-dimensional image reconstruction based on self-supervised deep learning.
[0024] Figure 6 It is the network structure of a U-shaped network module in a self-supervised deep learning network framework based on the MVS framework in a method of the present invention for real-time tracking of lesion positions with a binocular endoscope and dynamic display of lesion images in the form of three-dimensional image reconstruction based on self-supervised deep learning.
[0025] Figure 7It is a structural diagram of a computer system of a server for implementing the method, system, and device embodiments of the present application. DETAILED DESCRIPTION
[0026] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.
[0027] References to "embodiments" herein mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.
[0028] See also Figure 1 The method for real-time tracking of lesion positions by binocular endoscopes and dynamically displaying lesion images in the form of three-dimensional image reconstruction based on self-supervised deep learning in a first embodiment of the present invention includes the following steps:
[0029] Step S100: A binocular endoscope camera system is constructed using the same image acquisition functional units as a binocular endoscope for collecting binocular endoscopic images. It is worth noting that the image acquisition functional units used in the binocular endoscope camera system (including but not limited to the lens, CCD photosensitive unit, and image processing chip) need to be the same as those used in the binocular endoscope camera system.
[0030] In step S200, a binocular endoscope camera system is used to collect images, and a binocular stereoscopic vision depth calculation formula is used to calculate the depth of each pixel in the image. The binocular stereoscopic vision depth calculation formula is:
[0031]
[0032] in, is the baseline length, is the camera focal length, For parallax.
[0033] Step S300: construct a convolutional neural network based on the MVS framework.
[0034] See also Figure 2 The convolutional neural network based on the MVS framework consists of a joint segmentation branch, a depth estimation branch, and a data enhancement branch.
[0035] The joint segmentation branch consists of a pre-trained VGG (a classic convolutional neural network (CNN) model proposed by the Visual Geometry Group of the University of Oxford) module and a non-negative matrix decomposition module.
[0036] See also Figure 3 The pre-trained VGG (a classic convolutional neural network (CNN) model proposed by the Visual Geometry Group at the University of Oxford) module is composed of a double convolution module, a triple convolution module, and a fully connected network layer. The triple convolution module consists of three convolution layers, three activation function layers, and one pooling layer connected alternately. Figure 4 The dual convolution module consists of two convolutional layers, two activation function layers, and one pooling layer connected alternately. Figure 5 .
[0037] The activation function layer uses the nonlinear activation function ReLU activation function for nonlinear activation, and the formula is as follows:
[0038]
[0039] in Indicates the Layer 1 of the network neurons, Indicates the Layer 1 of the network neurons, Indicates the The connection weights of neurons, ReLU represents the activation function, and the formula is as follows:
[0040]
[0041] The non-negative matrix factorization (NMF) is an algorithm in multivariate analysis and linear algebra that decomposes a matrix A into two matrices P and Q. None of the three matrices have negative elements.
[0042] The depth estimation branch is formed by sequentially connecting a convolutional network module, a differentiable homography mapping module, and a U-shaped network module.
[0043] The data enhancement branch is formed by sequentially connecting a random cropping module, a convolutional network module, a differentiable homography mapping module, and a U-shaped network module.
[0044] The convolutional network module is composed of multiple convolutional layers connected in sequence.
[0045] The differentiable homography mapping module has the following formula:
[0046]
[0047] The U-shaped network module is formed by skipping convolutional layers, see Figure 6 .
[0048] Step S400: Download the public dataset ScanNet dataset and other datasets, and perform the first round of training on the convolutional neural network based on the MVS framework.
[0049] In some preferred embodiments, the method comprises the following steps:
[0050] Step S410, calculate the loss function between the output image of the convolutional neural network and the target image, and modify the convolutional neural network according to the loss function. The loss function calculation formula is
[0051]
[0052] They are photometric consistency loss, semantic supervision consistency loss, data enhancement consistency loss, image depth smoothness loss and image structure similarity loss. are weights used to balance the loss function and need to take reasonable values (such as 0.8, 0.1, 0.1, 0.2, 0.0067 respectively).
[0053] The formula for photometric consistency loss is:
[0054]
[0055] is a binary validity mask indicating the valid pixels in the new view.
[0056] The semantic supervision consistency loss formula is:
[0057]
[0058] in and is a binary mask representing the valid pixels from view i to the reference view; argmax is the index of the maximum value of the matrix elements.
[0059] The data enhancement consistency loss formula is
[0060]
[0061] in It is converting The unoccluded mask of is the target image The true depth of It is an enhanced image The predicted depth.
[0062] The image depth smoothness loss formula is:
[0063]
[0064] in is the value of each pixel, is the depth of each pixel.
[0065] The image structure similarity loss is
[0066]
[0067] in is a binary mask for each pixel, and SSIM is an image structural similarity metric function.
[0068] In step S500, the binocular image collected in step S200 and the corresponding calculated depth map are used as the data set for the second round of training, and the convolutional neural network based on the MVS framework is trained for the second round.
[0069] In step S600, the image captured by the binocular endoscope software camera system is input into a convolutional neural network based on the MVS framework, and the output is a three-dimensional image and real-time tracking of the trajectory of the binocular endoscope on the three-dimensional image.
[0070] A second embodiment of the present invention provides a method for real-time tracking of lesion positions with a binocular endoscope and dynamically displaying lesion images in the form of three-dimensional image reconstruction based on self-supervised deep learning. The system comprises: an acquisition module and a training module;
[0071] The acquisition module is configured to acquire the original image through a binocular endoscope camera system;
[0072] The convolutional neural network based on self-supervised deep learning for real-time tracking of lesion positions with binocular endoscopes and dynamic display of lesion images in the form of three-dimensional image reconstruction is trained as follows:
[0073] The training module is configured to obtain a three-dimensional stereo image after three-dimensional reconstruction;
[0074] The original images collected by the binocular endoscope camera system are used as a dataset to construct and train a convolutional neural network based on self-supervised deep learning for binocular endoscope real-time tracking of lesion locations and dynamic display of lesion images in the form of three-dimensional image reconstruction.
[0075] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes and related instructions of the storage device and processing device described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0076] It should be noted that the above-mentioned embodiment provides a binocular endoscope based on self-supervised deep learning for real-time tracking of lesion positions and dynamic display of lesion images in the form of three-dimensional image reconstruction. The system is only illustrated by the division of the above-mentioned functional modules. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the modules or steps in the embodiments of the present invention can be decomposed or combined. For example, the modules of the above-mentioned embodiments can be combined into one module, or further divided into multiple sub-modules to complete all or part of the functions described above. The names of the modules and steps involved in the embodiments of the present invention are only for distinguishing the modules or steps and are not regarded as improper limitations on the present invention.
[0077] An electronic device according to a third embodiment of the present invention comprises: at least one processor and a memory communicatively connected to at least one of the processors; wherein the memory stores instructions executable by the processor, and the instructions are used to be executed by the processor to implement the above-mentioned method based on self-supervised deep learning for binocular endoscope real-time tracking of lesion positions and dynamic display of lesion images in the form of three-dimensional image reconstruction.
[0078] A fourth embodiment of the present invention provides a computer-readable storage medium storing computer instructions, wherein the computer instructions are used to be executed by the computer to implement the above-mentioned method based on self-supervised deep learning for real-time tracking of lesion positions with a binocular endoscope and dynamic display of lesion images in the form of three-dimensional image reconstruction.
[0079] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes and related instructions of the storage device and processing device described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0080] Those skilled in the art should be able to appreciate that, in conjunction with the modules and method steps of each example described in the embodiments disclosed herein, it is possible to implement them with electronic hardware, computer software, or a combination of the two, and the programs corresponding to the software modules and method steps can be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art. In order to clearly illustrate the interchangeability of electronic hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in electronic hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.
[0081] Reference below Figure 7 , which shows a structural diagram of a computer system of a server for implementing the method, system, and device embodiments of the present application. Figure 7 The server shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.
[0082] like Figure 7 As shown, the computer system includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 502 or the program loaded from the storage part 508 into the random access memory (RAM) 503. Various programs and data required for system operation are also stored in the RAM 503. The CPU 501, ROM 502 and RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.
[0083] The following components are connected to the I / O interface 505: an input section 506 including a keyboard, a mouse, and the like; an output section 507 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 508 including a hard disk and the like; and a communication section 509 including a network interface card such as a LAN (Local Area Network) card or a modem. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, and the like, is installed in the drive 510 as needed so that a computer program read therefrom can be installed into the storage section 508 as needed.
[0084] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the central processing unit (CPU) 501, the above-mentioned functions defined in the method of the present application are executed. It should be noted that the computer-readable medium mentioned above in the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code embodied on a computer-readable medium may be transmitted using any suitable medium, including, but not limited to, wireless, wire, optical cable, RF, etc., or any suitable combination thereof.
[0085] Computer program code for performing the operations of the present application may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).
[0086] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.
[0087] The terms "first", "second", etc. are used to distinguish similar objects, rather than to describe or indicate a particular order or sequence.
[0088] The term "comprise" or any other similar term is intended to cover non-exclusive inclusion such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed or inherent to such process, method, article, or apparatus.
[0089] Thus far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art may make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will fall within the scope of protection of the present invention.
Claims
1. A method for real-time tracking of lesion positions with a binocular endoscope and dynamic display of lesion images in the form of three-dimensional image reconstruction based on self-supervised deep learning, characterized in that: The method comprises: Step S100, using the same image acquisition functional unit as a binocular endoscope to build a binocular endoscope camera system for collecting binocular endoscope images; In step S200, a binocular endoscope camera system is used to collect images, and a binocular stereoscopic vision depth calculation formula is used to calculate the depth of each pixel in the image. The binocular stereoscopic vision depth calculation formula is: in, is the baseline length, is the camera focal length, for parallax; Step S300, constructing a convolutional neural network based on the MVS framework; Step S400 , downloading public datasets such as the ScanNet dataset (jointly developed by Stanford University, Princeton University, and Technical University of Munich, suitable for research and development of algorithms such as indoor scene segmentation, recognition, object detection, and scene reconstruction) and performing the first round of training on a convolutional neural network based on the MVS framework; Step S500, using the binocular image acquired in step S200 and the corresponding calculated depth map as secondary training, and performing a second round of training on the convolutional neural network based on the MVS framework; In step S600, the image captured by the binocular endoscope camera system is input into a convolutional neural network based on the MVS framework, and the output is a three-dimensional image and real-time tracking of the trajectory of the binocular endoscope on the three-dimensional image.
2. The method for real-time tracking of lesion positions by binocular endoscopes and dynamic display of lesion images in the form of three-dimensional image reconstruction according to claim 1, characterized in that: The convolutional neural network based on the MVS framework is constructed as follows: Referring to Figure 2, the convolutional neural network based on the MVS framework consists of a joint segmentation branch, a depth estimation branch, and a data enhancement branch; The joint segmentation branch consists of a pre-trained VGG (a classic convolutional neural network (CNN) model proposed by the Visual Geometry Group at the University of Oxford) module and a non-negative matrix factorization module; As shown in Figure 3, the pre-trained VGG module is composed of a double convolution module, a triple convolution module, and a fully connected network layer connected in sequence; the triple convolution module is composed of three convolution layers, three activation function layers, and one pooling layer connected alternately, as shown in Figure 4; the double convolution module is composed of two convolution layers, two activation function layers, and one pooling layer connected alternately, as shown in Figure 5; The activation function layer uses the nonlinear activation function ReLU activation function for nonlinear activation, and the formula is as follows: in Indicates the Layer 1 of the network neurons, Indicates the Layer 1 of the network neurons, Indicates the The connection weights of neurons, ReLU represents the activation function, and the formula is as follows: The non-negative matrix factorization (NMF) is an algorithm in multivariate analysis and linear algebra that decomposes a matrix A into two matrices P and Q; none of these three matrices have negative elements; The depth estimation branch is composed of a convolutional network module, a differentiable homography mapping module, and a U-shaped network module connected in sequence; The data enhancement branch is formed by sequentially connecting a random cropping module, a convolutional network module, a differentiable homography mapping module, and a U-shaped network module; The convolutional network module is composed of multiple convolutional layers connected in sequence; The differentiable homography mapping module has the following formula: The U-shaped network module is formed by jump connections of convolutional layers, see Figure 6.
3. The method for real-time tracking of lesion positions by binocular endoscopes and dynamic display of lesion images in the form of three-dimensional image reconstruction according to claim 1, characterized in that: The loss function of the convolutional neural network output image and the target image is calculated as follows: They are photometric consistency loss, semantic supervision consistency loss, data enhancement consistency loss, image depth smoothness loss and image structure similarity loss; are weights used to balance the loss function and need to take reasonable values (such as 0.8, 0.1, 0.1, 0.2, 0.0067 respectively); The formula for photometric consistency loss is: in is a binary validity mask indicating the valid pixels in the new view; The semantic supervision consistency loss formula is: in and is a binary mask representing the valid pixels from view i to the reference view; argmax is the index of the maximum value of the matrix elements; The data enhancement consistency loss formula is in It is converting The unoccluded mask of is the target image The true depth of It is an enhanced image The predicted depth of The image depth smoothness loss formula is: in is the value of each pixel, is the depth of each pixel; The image structure similarity loss is in is a binary mask for each pixel, and SSIM is an image structural similarity metric function.