Deep learning-based method for tracking focus position in real time by monocular endoscope and dynamically displaying focus image in form of three-dimensional image reconstruction

Abstract

The invention relates to a method for tracking a focus position in real time by a monocular endoscope and dynamically displaying a focus image in a three-dimensional image reconstruction form based on deep learning. The method comprises the following steps: 1, constructing a binocular endoscope camera system; 2, constructing a convolutional neural network based on a Mnas network; 3, using a ScanNet data set and other data sets to carry out a first round of training on the convolutional neural network based on the Mnas network; 4, using the collected monocular video and the corresponding calculation depth map as a training data set, and performing a second round of training on the convolutional neural network based on the Mnas network; and 5, inputting an image acquired by a monocular endoscope software camera system into the convolutional neural network based on the Mnas network in a video streaming media mode, outputting a three-dimensional image, and tracking the track of the monocular endoscope on the three-dimensional image in real time. The invention not only provides a three-dimensional reconstruction method which is simple and convenient to operate, but also is convenient for doctors to find and more specifically observe lesions, and the operation efficiency of the doctors is improved.

Description

Technical Field

[0001] The present invention relates to a method for real-time tracking of lesion positions with a monocular endoscope and dynamic display of lesion images in the form of three-dimensional image reconstruction based on deep learning. Background Art

[0002] With the continuous development of medical technology, endoscopes, as detection instruments that integrate optics, ergonomics, precision machinery, modern electronics, mathematics, and software, are becoming increasingly widely used. The limited field of view and lack of depth perception of endoscopes pose challenges to doctors in locating lesions during surgery, greatly affecting the accuracy and safety of surgical operations. In recent years, three-dimensional stereoscopic vision technology has gradually penetrated various fields, and the demand for stereoscopic visualization images in clinical medicine has become an important part of computer vision. Three-dimensional laparoscopes developed based on this technology have been successful in clinical trials, significantly reducing surgical complications, alleviating patient pain, improving the level of minimally invasive surgery, and promoting medical development.

[0003] Compared to traditional endoscopes, 3D endoscopes can provide three-dimensional information and better reflect the actual situation of the scene. On the one hand, it allows doctors to experience the visual effects of 3D images during surgery, providing a realistic visual experience and helping doctors to operate the endoscope more accurately. On the other hand, 3D endoscopes can integrate three-dimensional measurement functions. Not only can they detect the distance between the front end of the endoscope and the observed tissue, avoiding the risk of the endoscope getting too close or accidentally contacting the tissue to be inspected when moving the endoscope, but they can also perform accurate three-dimensional measurements of some lesions. Compared to traditional surgical procedures where doctors rely on experience to judge the size of lesions, the results of three-dimensional measurement undoubtedly provide doctors with a quick and reliable reference, reducing their burden and improving the effectiveness of the surgery.

[0004] Currently, 3D reconstruction typically uses a monocular or binocular camera to obtain left and right views of a scene, then uses a binocular matching algorithm to generate a disparity map. This is then used to calculate the depth map, obtaining the 3D coordinates of each point in the image for 3D reconstruction. This method is simple and convenient, but the resulting endoscopic 3D images have weak texture features, and the background and lesion targets are similar in color, making them easily confused. This can affect the doctor's observation of the lesion and complicate the surgical procedure.

[0005] The emergence of deep learning, particularly models such as convolutional neural networks (CNNs) and generative adversarial networks (GANs), has brought new breakthroughs in 3D reconstruction. With the development of deep learning technology and the increasing availability of large training datasets, a new generation of methods has emerged that can recover the 3D structure of objects from one or more RGB images without the need for complex camera calibration. Summary of the Invention

[0006] In order to overcome the shortcomings of the existing technology, the purpose of the present invention is to provide a method based on deep learning for real-time tracking of lesion positions with a monocular endoscope and dynamically displaying lesion images in the form of three-dimensional image reconstruction, which is used to solve the technical problems of the existing technology that the three-dimensional image texture features of the monocular endoscope are weak, the background and the lesion target color are similar, and confusion is very easy to occur, thereby achieving the purpose of improving image quality and enhancing the ability to detect lesions.

[0007] The present invention achieves the above-mentioned object through the following technical solution: a method for real-time tracking of lesion location with a monocular endoscope based on deep learning and dynamically displaying lesion images in the form of three-dimensional image reconstruction, the method comprising the following steps:

[0008] Step S100: A binocular endoscope camera system is constructed using the same image acquisition functional unit as a monocular endoscope to acquire binocular images of the endoscope. It is worth noting that the image acquisition functional unit (including but not limited to the lens, CCD photosensitive unit, and image processing chip) used in the binocular endoscope camera system needs to be the same as that of the monocular endoscope camera system.

[0009] In step S200, a binocular endoscope camera system is used to collect images and monocular videos, and a binocular stereoscopic depth calculation formula is used to calculate the depth of each pixel on the image. The binocular stereoscopic depth calculation formula is:

[0010]

[0011] in, is the baseline length, is the camera focal length, For parallax.

[0012] Step S300: Build a convolutional neural network based on the Mnas network (proposed by Google's Tan Mingxing in 2019, which further optimizes convolutional neural networks). The Mnas network weights are initialized using network structure parameters trained on the ImageNet dataset (created by Stanford University Professor Fei-Fei Li's team).

[0013] Step S400: Download the public dataset ScanNet dataset (jointly developed by Stanford University, Princeton University, and Technical University of Munich, suitable for research and development of algorithms such as indoor scene segmentation, recognition, object detection, and scene reconstruction) and perform the first round of training on the convolutional neural network based on the Mnas network.

[0014] In step S500, the monocular video collected in step S200 and the corresponding calculated depth map are used as secondary training to perform a second round of training on the convolutional neural network based on the Mnas network.

[0015] In step S600, the image captured by the monocular endoscope software camera system is input into the convolutional neural network based on the Mnas network in the form of video streaming media, and the output is a three-dimensional image and real-time tracking of the trajectory of the monocular endoscope on the three-dimensional image.

[0016] It can be seen that compared with the prior art, the present invention has the following beneficial effects:

[0017] (1) Monocular reconstruction: It expands the application scope of endoscopic three-dimensional reconstruction. In particular, many fields where binocular endoscopes cannot be used can use this model to implement three-dimensional reconstruction and achieve the effect of auxiliary medical treatment;

[0018] (2) Assisted navigation and positioning: This design can also be used to assist doctors in navigation and positioning, especially in minimally invasive surgery. Doctors can more accurately locate and orient the treatment area, reduce errors, and reduce the risk of damaging surrounding healthy tissues.

[0019] (3) Improve surgical results: By providing clearer visualization and more accurate diagnosis of lesions, the monocular endoscope can track the location of lesions in real time and dynamically display the lesion images in the form of three-dimensional image reconstruction, which can help improve the effectiveness of surgery, reduce the risk of surgical complications, and shorten recovery time;

[0020] (4) Low computational cost and short time: Pre-training the artificial neural network through deep learning can improve the prediction accuracy while reducing the computational cost, thereby reducing the running time of the 3D reconstruction image. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a flowchart of an embodiment of a method of the present invention for real-time tracking of lesion positions with a monocular endoscope based on deep learning and dynamically displaying lesion images in the form of three-dimensional image reconstruction.

[0022] Figure 2 This is a network structure diagram of a convolutional neural network in a method of the present invention for real-time tracking of lesion positions with a monocular endoscope and dynamic display of lesion images in the form of three-dimensional image reconstruction based on deep learning.

[0023] Figure 3 This is a network structure diagram of the convolutional neural network submodule Mnas network in a method of the present invention based on deep learning for real-time tracking of lesion positions with a monocular endoscope and dynamic display of lesion images in the form of three-dimensional image reconstruction.

[0024] Figure 4This is a method based on deep learning for real-time tracking of lesion positions with a monocular endoscope and dynamic display of lesion images in the form of three-dimensional image reconstruction, and a network structure diagram of the MB convolution module in the convolutional neural network Mnas network.

[0025] Figure 5 It is a structural diagram of a computer system of a server for implementing the method, system, and device embodiments of the present application. DETAILED DESCRIPTION

[0026] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0027] References to "embodiments" herein mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0028] See also Figure 1 The method for real-time tracking of lesion position with a monocular endoscope and dynamically displaying lesion images in the form of three-dimensional image reconstruction based on deep learning in the first embodiment of the present invention includes the following steps:

[0029] Step S100: A binocular endoscope camera system is constructed using the same image acquisition functional unit as a monocular endoscope to acquire binocular images of the endoscope. It is worth noting that the image acquisition functional unit (including but not limited to the lens, CCD photosensitive unit, and image processing chip) used in the binocular endoscope camera system needs to be the same as that of the monocular endoscope camera system.

[0030] In step S200, a binocular endoscope camera system is used to collect images and monocular videos, and a binocular stereoscopic depth calculation formula is used to calculate the depth of each pixel on the image. The binocular stereoscopic depth calculation formula is:

[0031]

[0032] in, is the baseline length, is the camera focal length, For parallax.

[0033] Step S300, constructing a convolutional neural network based on the Mnas network.

[0034] See also Figure 2 The convolutional neural network based on the Mnas network is composed of a Mnas network, a gated recurrent unit (GRU), and a fully connected neural network (MLP) connected in sequence, wherein the weights of the Mnas network are initialized using the network structure parameters trained based on the ImageNet dataset.

[0035] The Mnas network is a model that focuses on both inference speed and inference accuracy. Figure 3 , which is composed of a convolutional layer, a separable convolutional layer, 6 layers of moving inverted bottleneck convolutional layers, and a pooling layer. For the specific structure of the moving inverted bottleneck convolutional layer, see Figure 4 ,Depend on Convolutional layer, depthwise separable convolutional layer, The convolutional layers are connected sequentially.

[0036] Step S400: Download the public dataset ScanNet dataset and other datasets, and perform the first round of training on the convolutional neural network based on the Mnas network.

[0037] Calculate the loss function between the output image of the convolutional neural network and the target image, and modify the convolutional neural network according to the loss function. The loss function calculation formula is:

[0038]

[0039] in is the true value of the target image, The predicted value is The probability of For the target image The signed distance function.

[0040] In step S500, the monocular video collected in step S200 and the corresponding calculated depth map are used as secondary training to perform a second round of training on the convolutional neural network based on the Mnas network.

[0041] In step S600, the image captured by the monocular endoscope software camera system is input into the convolutional neural network based on the Mnas network in the form of video streaming media, and the output is a three-dimensional image and real-time tracking of the trajectory of the monocular endoscope on the three-dimensional image.

[0042] A second embodiment of the present invention provides a method for real-time tracking of lesion positions with a monocular endoscope and dynamically displaying lesion images in the form of three-dimensional image reconstruction based on deep learning. The system includes: an acquisition module and a training module;

[0043] The acquisition module is configured to acquire original 2D images through a monocular endoscope camera system and synthesize a video stream;

[0044] The convolutional neural network based on the Mnas network is trained by:

[0045] The training module is configured to obtain a three-dimensional image after three-dimensional reconstruction;

[0046] The video stream synthesized from the original 2D images acquired by the monocular endoscope camera system is used as a data set to construct and train a convolutional neural network based on the Mnas network.

[0047] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes and related instructions of the storage device and processing device described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0048] It should be noted that the above-mentioned embodiment provides a monocular endoscopic imaging system based on deep learning for real-time tracking of lesion positions through a monocular endoscope and dynamic display of lesion images in the form of three-dimensional image reconstruction. It is only illustrated by the division of the above-mentioned functional modules. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the modules or steps in the embodiments of the present invention can be further decomposed or combined. For example, the modules in the above-mentioned embodiments can be combined into one module, or further divided into multiple sub-modules to complete all or part of the functions described above. The names of the modules and steps involved in the embodiments of the present invention are only for the purpose of distinguishing the modules or steps and are not to be regarded as improper limitations of the present invention.

[0049] An electronic device according to a third embodiment of the present invention comprises: at least one processor and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the processor, and the instructions are used to be executed by the processor to implement the above-mentioned method based on deep learning for real-time tracking of lesion positions with a monocular endoscope and dynamically displaying lesion images in the form of three-dimensional image reconstruction.

[0050] A fourth embodiment of the present invention provides a computer-readable storage medium storing computer instructions, wherein the computer instructions are used to be executed by the computer to implement the above-mentioned method for real-time tracking of lesion positions with a monocular endoscope based on deep learning and dynamically displaying lesion images in the form of three-dimensional image reconstruction.

[0051] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes and related instructions of the storage device and processing device described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0052] Those skilled in the art should be able to appreciate that, in conjunction with the modules and method steps of each example described in the embodiments disclosed herein, it is possible to implement them with electronic hardware, computer software, or a combination of the two, and the programs corresponding to the software modules and method steps can be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art. In order to clearly illustrate the interchangeability of electronic hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in electronic hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0053] Reference below Figure 5 , which shows a structural diagram of a computer system of a server for implementing the method, system, and device embodiments of the present application. Figure 5 The server shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0054] like Figure 5 As shown, the computer system includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 502 or the program loaded from the storage part 508 into the random access memory (RAM) 503. Various programs and data required for system operation are also stored in the RAM 503. The CPU 501, ROM 502 and RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0055] The following components are connected to the I / O interface 505: an input section 506 including a keyboard, a mouse, and the like; an output section 507 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 508 including a hard disk and the like; and a communication section 509 including a network interface card such as a LAN (Local Area Network) card or a modem. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, and the like, is installed in the drive 510 as needed so that a computer program read therefrom can be installed into the storage section 508 as needed.

[0056] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the central processing unit (CPU) 501, the above-mentioned functions defined in the method of the present application are executed. It should be noted that the computer-readable medium mentioned above in the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code embodied on a computer-readable medium may be transmitted using any suitable medium, including, but not limited to, wireless, wire, optical cable, RF, etc., or any suitable combination thereof.

[0057] Computer program code for performing the operations of the present application may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0058] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0059] The terms "first", "second", etc. are used to distinguish similar objects, rather than to describe or indicate a particular order or sequence.

[0060] The term "comprise" or any other similar term is intended to cover non-exclusive inclusion such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0061] Thus far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art may make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will fall within the scope of protection of the present invention.

Claims

1. A method based on deep learning for real-time tracking of lesion location with a monocular endoscope and dynamic display of lesion images in the form of three-dimensional image reconstruction, characterized in that: The method comprises: Step S100: A binocular endoscope camera system is constructed using the same image acquisition functional unit as a monocular endoscope to acquire binocular images of the endoscope. It is worth noting that the image acquisition functional unit (including but not limited to a lens, a CCD photosensitive unit, and an image processing chip) used in the binocular endoscope camera system needs to be the same as that of the monocular endoscope camera system. In step S200, a binocular endoscope camera system is used to collect images and monocular videos, and a binocular stereoscopic depth calculation formula is used to calculate the depth of each pixel on the image. The binocular stereoscopic depth calculation formula is: in, is the baseline length, is the camera focal length, for parallax; Step S300: Construct a convolutional neural network based on the Mnas network; wherein the weights of the Mnas network are initialized using the network structure parameters trained based on the ImageNet dataset (created by the team led by Professor Fei-Fei Li of Stanford University); Step S400: Download the public dataset ScanNet (jointly developed by Stanford University, Princeton University, and Technical University of Munich, suitable for research and development of algorithms such as indoor scene segmentation, recognition, object detection, and scene reconstruction) and perform the first round of training on the convolutional neural network based on the Mnas network. Step S500, using the monocular video collected in step S200 and the corresponding calculated depth map as secondary training, performing a second round of training on the convolutional neural network based on the Mnas network; In step S600, the image captured by the monocular endoscope software camera system is input into the convolutional neural network based on the Mnas network in the form of video streaming media, and the output is a three-dimensional image and real-time tracking of the trajectory of the monocular endoscope on the three-dimensional image.

2. The method according to claim 1, wherein the method is characterized in that: The convolutional neural network based on the Mnas network is constructed by: Referring to FIG2 , the convolutional neural network based on the Mnas network is composed of a Mnas network, a gated recurrent unit (GRU), and a fully connected neural network (MLP) connected in sequence, wherein the weights of the Mnas network are initialized using the network structure parameters trained based on the ImageNet dataset; The Mnas network is a model that focuses on both inference speed and inference accuracy. See Figure 3. It is composed of a convolutional layer, a separable convolutional layer, a 6-layer moving inverted bottleneck convolutional layer, and a pooling layer connected in sequence. The specific structure of the moving inverted bottleneck convolutional layer is shown in Figure 4. Convolutional layer, depthwise separable convolutional layer, The convolutional layers are connected sequentially.

3. The method according to claim 1, wherein the method is characterized in that: The loss function of the convolutional neural network output image and the target image is calculated as follows: in is the true value of the target image, The predicted value is The probability of For the target image The signed distance function.

Images