Endoscopic localization methods and electronic devices based on visual odometry networks

Abstract

This invention provides an endoscope localization method and electronic device based on a visual odometry network. The method includes: acquiring the pose information of a real endoscope when acquiring a first frame image; inputting the current frame image acquired by the real endoscope and the previous frame image acquired into a pre-trained visual odometry network to obtain relative pose estimation information of the real endoscope when acquiring the current frame image and the previous frame image; wherein the style of the current frame image and the previous frame image input into the visual odometry network is consistent with the style of the images used to pre-train the visual odometry network; superimposing the pose information of the real endoscope when acquiring the first frame image with all the cumulative relative pose estimation information from acquiring the first frame image to acquiring the current frame image to obtain the pose information of the real endoscope when acquiring the current frame image, and locating the real endoscope based on the pose information. This method can quickly and continuously locate the endoscope.

Description

Technical Field

[0001] This invention relates to the field of endoscopic positioning technology, and more particularly to an endoscopic positioning method and electronic device based on a visual odometry network. Background Technology

[0002] An endoscope is a diagnostic instrument that integrates traditional optics, ergonomics, precision mechanics, modern electronics, mathematics, and software. It includes an image sensor, optical lens, light source, and mechanical devices. It can be inserted into the stomach through the mouth or other natural orifices. Endoscopes can visualize lesions that are not visible to X-rays, making them a commonly used technique in medical examinations.

[0003] Currently, commonly used methods for endoscopic localization include: (1) Extracting the depth of the endoscopic image using the Shape from Shading (SFS) method, identifying the deeper parts as airways. After extracting the airways, comparing the model reconstructed from preoperative CT, the current image is mapped to the airway branch where the camera is located, or the endoscopic motion is estimated based on the change in the deepest part of the airway in adjacent images. This method can be achieved at airway bifurcation points, but it is difficult to provide continuous endoscopic localization information when there is no airway or only one airway in the field of view. (2) Extracting feature points of the endoscopic image using the Structure From Motion (SFM) method, matching the feature points one by one for two adjacent frames, and calculating the Perspective-n-Point (PnP) based on this to estimate the endoscopic pose. This method cannot solve the Perspective-n-Point (PnP) when there are few or no feature points in the endoscopic image, resulting in the problem of lost endoscopic localization. (3) 2D / 3D registration method: The 2D images captured by the endoscope are registered onto the virtual model reconstructed before the operation to obtain the position of the endoscope in the model. This method is based on an iterative optimization algorithm, so obtaining the localization of each frame requires a long computation time. However, the pose of the endoscope changes rapidly during actual examination, and excessive computation time can easily lead to localization loss. Summary of the Invention

[0004] This invention provides an endoscope positioning method and electronic device based on a visual odometry network to solve the defects of existing technologies that cannot provide continuous positioning information and are prone to positioning loss, thereby achieving rapid and accurate positioning of the endoscope and obtaining continuous pose information.

[0005] This invention provides an endoscopic localization method based on a visual odometry network, comprising:

[0006] Obtain the pose information of the real endoscope when acquiring the first frame image;

[0007] The current frame image and the previous frame image acquired by a real endoscope are input into a pre-trained visual odometry network to obtain relative pose estimation information when the current frame image and the previous frame image are acquired by the real endoscope; wherein, the style of the current frame image and the previous frame image input into the visual odometry network is consistent with the style of the images used to pre-train the visual odometry network.

[0008] The pose information of the real endoscope when acquiring the first frame image is superimposed with all the relative pose estimation information accumulated from acquiring the first frame image to acquiring the current frame image to obtain the pose information of the real endoscope when acquiring the current frame image, and the real endoscope is positioned according to the pose information.

[0009] According to the present invention, an endoscope localization method based on a visual odometry network is provided, wherein the visual odometry network is trained in the following manner:

[0010] A virtual model is established, and virtual images are acquired in the virtual model using a virtual endoscope, and the virtual pose information of the virtual endoscope corresponding to the acquisition of the virtual images is obtained, wherein the virtual endoscope is determined based on the real endoscope;

[0011] The virtual image is input into the initial visual odometry network, and the initial visual odometry network outputs the relative pose estimation information of the virtual endoscope when acquiring two adjacent frames of virtual images;

[0012] The virtual pose information is used as the training ground value, and the relative pose ground value information of the virtual endoscope when acquiring the virtual images of two adjacent frames is obtained based on the virtual pose information.

[0013] The loss function is obtained by weighted summing of translation and rotation losses between the relative pose estimation information and the relative pose ground truth information of the virtual endoscope;

[0014] The loss function is optimized, and the parameters of the initial visual odometry network are updated until convergence is achieved to obtain the visual odometry network.

[0015] According to the present invention, an endoscope localization method based on a visual odometry network is provided, wherein the style of the current frame image and the previous frame image input to the visual odometry network are consistent with the style of the virtual image pre-trained on the visual odometry network.

[0016] A pre-trained recurrent generative adversarial network is used to transform the current frame image and the previous frame image in a realistic style into a virtual style, or

[0017] By converting the virtual image into a realistic virtual image.

[0018] According to the present invention, an endoscope localization method based on a visual odometry network is provided, wherein the visual odometry network is trained in the following manner:

[0019] Acquire real endoscopic images captured by a real endoscope;

[0020] The real endoscope images are used as training data and input into the initial visual odometry network. The initial visual odometry network includes a pose network and a depth network. The pose network is used to estimate the relative pose information of the real endoscope when acquiring two adjacent frames of real endoscope images. The depth network is used to estimate the depth images of the two adjacent frames of real endoscope images respectively.

[0021] Based on the relative pose information of the real endoscope corresponding to the two adjacent real endoscope images estimated by the pose network and the depth images of the two adjacent virtual images, the previous real endoscope image is warped to obtain the next real endoscope warped image.

[0022] Based on the visual relationship between the subsequent frame of the real endoscope warped image obtained through image warping and the subsequent frame of the real endoscope image. Figure One Consistency is restored to its original state;

[0023] Geometric consistency loss is obtained by calculating the scale consistency of the depth images of two adjacent real endoscopic images estimated by the deep network.

[0024] The reconstruction loss and geometric consistency loss are weighted and summed to obtain the loss function. The loss function is then optimized, and the parameters of the initial visual odometry network are updated until convergence, thus obtaining the visual odometry network.

[0025] According to the present invention, an endoscope localization method based on a visual odometry network, before superimposing the pose information of the real endoscope when acquiring the first frame image with all the relative pose estimation information accumulated from acquiring the first frame image to acquiring the current frame image, to obtain the pose information of the real endoscope acquiring the current frame image, further includes:

[0026] The relative pose estimation information obtained from the current frame image and the previous frame image acquired by the real endoscope is scaled to obtain the unit of the relative pose estimation information.

[0027] According to the present invention, an endoscope localization method based on a visual odometry network is provided to obtain the pose information of a real endoscope when acquiring the first frame image, including:

[0028] The first frame image captured by a real endoscope is acquired, and a pre-trained recurrent generative adversarial network is used to convert the style of the first frame image into a virtual style.

[0029] A target virtual model is established, and a target virtual image is acquired in the target virtual model through a virtual endoscope, and the target pose information of the virtual endoscope is obtained when the target virtual image is acquired, wherein the virtual endoscope is determined based on the real endoscope;

[0030] The image retrieval network is trained using the target virtual image and the target pose information;

[0031] A pre-trained image retrieval network is used to retrieve the first frame image converted to a virtual style, thereby obtaining the pose information of the real endoscope when acquiring the first frame image.

[0032] According to the present invention, an endoscope localization method based on a visual odometry network is provided to obtain the pose information of a real endoscope when acquiring the first frame image, including:

[0033] The first frame image captured by a real endoscope is acquired, and a pre-trained recurrent generative adversarial network is used to convert the style of the first frame image into a virtual style.

[0034] A target virtual model is established, a target virtual image is acquired by a virtual endoscope in the target virtual model, and the target pose information corresponding to the target virtual endoscope when the target virtual image is acquired is acquired, wherein the virtual endoscope is determined based on the real endoscope;

[0035] The first frame image, converted to a virtual style, is registered with the target virtual image using an image registration method. The target pose information corresponding to the target virtual image with the highest similarity to the first frame image is the pose information of the real endoscope when acquiring the first frame image.

[0036] An endoscope localization method based on a visual odometry network according to the present invention further includes:

[0037] The pose information of the current frame image acquired by the real endoscope is corrected by using a registration method based on an iterative optimization algorithm and running in parallel with the visual odometry.

[0038] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the endoscopic positioning method based on visual odometry network as described above.

[0039] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the endoscopic localization method based on a visual odometry network as described above.

[0040] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the endoscopic positioning method based on a visual odometry network as described above.

[0041] The endoscopic localization method based on a visual odometry network provided by this invention, given the initial pose of the actual endoscope, can obtain the relative pose of the actual endoscope from two consecutive image frames acquired by the endoscope using a pre-trained visual odometry network. This allows for the rapid and continuous acquisition of the current pose information of the actual endoscope by superimposing the relative pose with the initial pose. The visual odometry network in this method can be directly used for different patients after training, eliminating the need for preoperative training, thus saving time and effort. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0043] Figure 1 This is a flowchart illustrating the endoscopic localization method based on a visual odometry network provided by the present invention.

[0044] Figure 2 This is a test result diagram of the visual odometry network in the endoscopic positioning method based on visual odometry network provided by the present invention;

[0045] Figure 3 This is one of the flowcharts illustrating the training method for the visual odometry network provided by the present invention;

[0046] Figure 4 This is a schematic diagram of the structure of a Cycle GAN (Generative Adversarial Network).

[0047] Figure 5 This is the second flowchart illustrating the training method for the visual odometry network provided by this invention.

[0048] Figure 6 This is one of the flowcharts provided by the present invention for obtaining the pose information of a real endoscope when acquiring the first frame image;

[0049] Figure 7This is the second schematic diagram of the process for obtaining the pose information of a real endoscope when acquiring the first frame image, provided by the present invention.

[0050] Figure 8 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0052] The following is combined with Figures 1-7 The endoscopic localization method based on visual odometry networks of the present invention is described as follows: Figure 1 As shown, the method includes:

[0053] S101: Obtain the pose information of the real endoscope when acquiring the first frame image.

[0054] In this embodiment of the invention, the endoscopic positioning method can be used in natural cavities of the human body such as the respiratory tract, biliary tract, and ventricles. The method first requires obtaining the pose information of the real endoscope when it acquires the first frame image, i.e., the initial pose information of the real endoscope. The pose information is a 6-DOF spatial coordinate system, including 3D translation and 3D Euler angles.

[0055] The initial pose information can be obtained through image registration or image retrieval methods, but is not limited to these two methods.

[0056] S102: Input the current frame image acquired by the real endoscope and the previous frame image acquired into the pre-trained visual odometry network to obtain the relative pose estimation information of the current frame image and the previous frame image acquired by the real endoscope; wherein, the style of the current frame image and the previous frame image input into the visual odometry network is consistent with the style of the images used to pre-train the visual odometry network.

[0057] Specifically, a pre-trained visual odometry network is used to acquire pose change information between the current frame and the previous frame of an image captured by a real endoscope. To ensure the accuracy of the pose change information acquired by the pre-trained visual odometry network, the style of the images input to the visual odometry network needs to be consistent with the style of the images used to pre-train the visual odometry network. The style of the images refers to the texture, color, and visual patterns at different spatial scales in the images.

[0058] S103: The pose information of the real endoscope when acquiring the first frame image is superimposed with all the relative pose estimation information accumulated from acquiring the first frame image to acquiring the current frame image to obtain the pose information of the real endoscope acquiring the current frame image, and the real endoscope is positioned according to the pose information.

[0059] As the real endoscope moves, the aforementioned steps can be used to obtain all pose change information accumulated from the acquisition of the first frame image to the acquisition of the current frame image. By superimposing all pose change information with the initial pose information of the real endoscope, the pose information of the real endoscope in the current frame image is obtained, and the real endoscope is positioned based on the pose information of the real endoscope in the current frame image.

[0060] For example, a real endoscope acquires the pose information P of the current frame image. t for: Wherein, the initial pose information is P1, P 1,2 To obtain pose change information between acquiring the first frame image and acquiring the second frame image using a real endoscope, P t-1,t To obtain pose change information between acquiring the (t-1)th frame image and acquiring the tth frame image (the current frame image) using a real endoscope. This represents the superposition of poses.

[0061] A virtual model was built to acquire virtual images, and the aforementioned visual odometry network was tested to obtain the test results. For example... Figure 2 As shown, Figure 2 The solid line in the figure represents the actual trajectory of the virtual endoscope, while the dashed line represents the trajectory of the virtual endoscope estimated using the method described above. This trajectory includes 289 images and poses, and the unit is mm.

[0062] An endoscope localization method based on a visual odometry network provided by the present invention, such as... Figure 3 As shown, the visual odometry network is trained in the following manner:

[0063] S301: Establish a virtual model, acquire virtual images through a virtual endoscope in the virtual model, and obtain the virtual pose information of the virtual endoscope corresponding to the acquisition of the virtual images, wherein the virtual endoscope is determined based on the real endoscope.

[0064] Specifically, the visual odometry network is composed of convolutional neural networks (CNNs), with a typical encoder-decoder structure. The encoder structure is the same as the FlowNetC encoder structure, but with pre-trained weights. FlowNet is a relatively mature optical flow extraction network that estimates the optical flow between two input frames. Optical flow is an approximation of the motion field and is of great reference value for camera pose estimation.

[0065] In practice, obtaining the pose of the endoscope during a real endoscopic examination is quite difficult. Therefore, we need to establish a virtual model. Using a virtual endoscope, we acquire a large number of virtual images and corresponding virtual pose information of the virtual endoscope to train and supervise the visual odometry network. This improves the robustness of the visual odometry network. The virtual images are virtual endoscope images that retain only geometric structural features and have no surface texture. Various virtual models can be used, such as virtual models for the respiratory tract, virtual models for the biliary tract, etc., and corresponding virtual models can be established according to the specific needs.

[0066] To improve the positioning accuracy of the visual odometry network for real endoscopes after training, the virtual endoscope mentioned above needs to be determined based on the real endoscope. Therefore, the internal parameters of the virtual endoscope need to be consistent with those of the real endoscope.

[0067] Example: Use MATLAB software to perform checkerboard calibration on a real endoscope to obtain the endoscope's intrinsic parameters.

[0068] The internal references of a real endoscope are:

[0069]

[0070] Image pixel count:

[0071] Width * Length = width × height

[0072] make:

[0073] Average focal length:

[0074] The x-coordinate of the window center is: wcx = -2 × (cx - width / 2) / width

[0075] Window center y-axis coordinate: wcy = 2 × (cy - height / 2) / height

[0076] At this point, when designing a virtual endoscope, the parameters of the virtual endoscope are:

[0077] Field of view:

[0078] ViewAngle=180 / π*(2.0*atan2(heght / 2.0,focal_length))

[0079] Window size:

[0080] WindeowSize = [width, height]

[0081] Center of window:

[0082] WindowCenter = [wcx, wcy]

[0083] S302: Input the virtual image into the initial visual odometry network, and the initial visual odometry network outputs the relative pose estimation information of the virtual endoscope when acquiring two adjacent frames of virtual images.

[0084] The virtual images obtained by the above method are input into the visual odometry network for supervised training. The output of the visual odometry network is the relative pose estimation information of the virtual endoscope when acquiring two adjacent frames of virtual images.

[0085] S303: Using the virtual pose information as training ground value, obtain the relative pose ground value information of the virtual endoscope when acquiring the virtual images of two adjacent frames based on the virtual pose information.

[0086] Using the virtual pose information as the training ground value, the relative pose ground value information of the virtual endoscope when acquiring two adjacent frames of images can be obtained by calculating the virtual pose information. At this time, the relative pose ground value information and phase pose estimation information of the virtual endoscope when acquiring two adjacent frames of images are obtained.

[0087] S304: The loss function is obtained by weighted summing of translation loss and rotation loss between the relative pose estimation information and the relative pose truth information of the virtual endoscope.

[0088] Specifically, the pose information is a 6-dimensional vector, consisting of 3-dimensional translation and 3-dimensional Euler angles. The translational and rotational losses are calculated separately between the estimated relative pose information of the virtual endoscope and the true relative pose information. The weighted sum of the translational and rotational losses yields the final loss function.

[0089] L(z t-m ,z t ) = L t (z t-m ,z t )+ωL r (z t-m ,z t )

[0090] Among them, L t For translation loss: These are the translation vectors in the true relative pose information and the relative pose estimation information, respectively; L r For rotational losses: These are the rotation vectors in the true relative pose information and the relative pose estimation information, respectively; ω is a hyperparameter used to adjust the proportions of rotation loss and displacement loss.

[0091] S305: Optimize the loss function and update the parameters of the initial visual odometry network until convergence is achieved, thereby obtaining the visual odometry network.

[0092] During training, the loss function obtained from the above steps is continuously optimized to update the parameters of the visual odometry network until convergence, thus determining the final visual odometry network used for localization estimation of real endoscopes. The visual odometry network determined by this training method can quickly locate real endoscopes and output information with clear scale when performing localization estimation.

[0093] For example, during training, two consecutive frames of virtual endoscope images and virtual pose information of the endoscope are input. ω can be 100. The Adam optimizer can be used to optimize the loss function. The initial learning rate during training can be 1e-5. The training is conducted for a total of 200 rounds, and the learning rate is gradually reduced as the number of training rounds increases.

[0094] According to the present invention, an endoscope localization method based on a visual odometry network is provided, wherein the style of the current frame image and the previous frame image input to the visual odometry network are consistent with the style of the virtual image pre-trained on the visual odometry network.

[0095] A pre-trained recurrent generative adversarial network is used to transform the current frame image and the previous frame image in a realistic style into a virtual style, or

[0096] By converting the virtual image into a realistic virtual image.

[0097] Specifically, there are two methods to ensure that the style of the current frame image and the previous frame image input to the visual odometry network is consistent with the style of the virtual image used to pre-train the visual odometry network. One method is to convert the style of the virtual image into a real-style virtual image before inputting it into the visual odometry network during training. In this way, the trained visual odometry network can directly input images acquired by a real endoscope when it is used. The other method is to directly input the virtual image into the visual odometry network for training. In this way, the trained visual odometry network can first convert the images acquired by a real endoscope into virtual-style images before inputting them into the visual odometry network when it is used.

[0098] Pre-trained recurrent generative adversarial networks can be used for style transfer, such as... Figure 4 As shown, the Cycle GAN consists of two interconnected GANs. The generator Gb aims to transform an image from domain A to domain B; the generator Ga aims to transform an image from domain B to domain A; the discriminator DA aims to distinguish whether an image belongs to domain A or is a fake domain A image transformed from domain B; and the discriminator DB aims to distinguish whether an image belongs to domain B or is a fake domain B image transformed from domain A. In this way, the generator and discriminator compete against each other, aiming to minimize the loss during training, and are co-trained to obtain a deterministic Cycle GAN.

[0099] The method for determining the loss function using a cyclic generative adversarial network is as follows:

[0100] The generative adversarial loss adds constraints to each GAN pair to optimize the discriminator D towards higher accuracy, and the generator G towards lower accuracy by deceiving the discriminator D. For the generator G· and the discriminator D·, the generative adversarial loss is:

[0101]

[0102] Where y is a variable representing a certain frame of the image, and p represents the probability distribution. The · symbol represents the expected value, and is used to omit the image or depth. y~p(data) means that the sample follows a distribution in the domain X or Y.

[0103] Cycle consistency loss refers to the difference between an image in domain A, converted to a image in domain B by generator Gb, and then converted back to a image in domain A by generator Ga, and the original image in domain A. To encourage the generator to recover the structure of the original image, thus ensuring that the generator retains important features of the original input image, cycle consistency loss is defined as follows:

[0104]

[0105] To further assist the generator in learning the response probability distribution, source image information is preserved, and an identity loss is added. Taking an image in domain A as an example, the image in domain A is input into G. a In this case, we want the output to remain unchanged:

[0106]

[0107] However, the information contained in real endoscopic images is not entirely consistent with that in virtual endoscopic images. For example, information such as texture and light source is not included in virtual endoscopic images. Since Cycle GAN requires that the images from both domains be able to be transformed and restored, Gb needs to encode the texture information from the real endoscope into the transformed image to ensure that this information can be restored. This texture encoding will cause unnecessary information to be generated during the domain transformation. Our desired style transfer network can retain only structural information while discarding other information; that is, the transformation constraints between the two domain images are relaxed: Gb removes texture, while Ga generates texture.

[0108] To address this issue, we remove the cycle invariance loss and identity loss from the original Cycle GAN loss function, which covers the transition from domain A to domain B (i.e., from real images to virtual images). Furthermore, to further ensure that structural information is preserved during style transfer, we add a structural similarity loss for both the domain A image and the pseudo-B image after the transfer.

[0109]

[0110] SSIM is a structural similarity measure.

[0111] Based on the above, the loss function is obtained:

[0112]

[0113] Where λ, θ, η, and α are hyperparameters for adjusting the weights of each loss.

[0114] For example, during training, the hyperparameters λ, θ, η, and α can be set to 10, 1, 5, and 1, respectively. The Adam optimizer is used to optimize the loss function during training, with an initial learning rate of 0.002. Training is conducted for 200 epochs, and the training input consists of unpaired real and virtual endoscope images.

[0115] An endoscope localization method based on a visual odometry network provided by the present invention, such as... Figure 5 As shown, the visual odometry network is trained in the following manner:

[0116] S501: Acquire real endoscopic images captured by a real endoscope.

[0117] S502: Input the real endoscope image as training data into the initial visual odometry network. The initial visual odometry network includes a pose network and a depth network. The pose network is used to estimate the relative pose information of the real endoscope when acquiring two adjacent frames of real endoscope images. The depth network is used to estimate the depth images of the two adjacent frames of real endoscope images respectively.

[0118] Specifically, the initial visual odometry network is a visual odometry network based on SfmLearner, which can include a pose network and a depth network. The pose network is used to estimate the relative pose information of the real endoscope when acquiring two adjacent frames of real endoscope images, and the depth network is used to estimate the depth images of two adjacent frames of real endoscope images respectively. This structure of visual odometry network can perform unsupervised learning.

[0119] S503: Based on the relative pose information of the real endoscope corresponding to the two adjacent real endoscope images estimated by the pose network and the depth image of the two adjacent virtual images, the previous real endoscope image is warped to obtain the next real endoscope warped image.

[0120] Specifically, for two consecutive input real endoscopic images x t-n x t Depth estimation networks can estimate the depth of two consecutive frames. and Pose-mapping networks can estimate the relative translation vector t of the endoscope's relative pose when acquiring two consecutive frames of images. t-n,t and relative rotation vector R t-n,t Given the endoscope's intrinsic parameter K, the pixel point in homogeneous coordinates. Warp to the pixel using the following formula

[0121]

[0122] because Typically, the coordinates are non-integer, requiring hyperbolic sampling to obtain integer pixel coordinates, ultimately yielding the result from x. t-n Warped image

[0123] S504: Based on the visual relationship between the next frame of the real endoscopic warped image obtained through image warping and the next frame of the real endoscopic image. Figure One The integrity was restored to its original state.

[0124] From x t-n Warped image Should be with x t Consistency, by view Figure OneConsistency is lost during reconstruction:

[0125]

[0126] Where w(·) is from x t-n The space it occupies is warped to x t The operator in the space, w(x) t-n ) is composed of x t-n and through relative translation vector t t-n,t and relative rotation vector R t-n,t Depth image obtained by reprojection Represents a pixel in image x. Warping refers to manipulating the image to deform the pixels in the image.

[0127] S505: The geometric consistency loss is obtained by calculating the scale consistency of the depth images of the two adjacent real endoscopic images estimated by the deep network.

[0128] Specifically, for depth images generated by deep networks and If they correspond to the same 3D scene, then the corresponding depth scale should be consistent.

[0129] Depth image and Inconsistency z diff Defined as:

[0130]

[0131] in It is by and through relative translation vector t t-n,t and relative rotation vector R t-n,t Depth image obtained by reprojection. From The depth image obtained from sampling. Here, the calculation is performed. and The error, rather than and The error is because The result of reprojection is not in an integer coordinate system, so it needs to be... The same coordinate system is sampled to calculate the difference between the two. Therefore, the geometric consistency loss is:

[0132]

[0133] in, This represents a pixel in the image z.

[0134] S506: The reconstruction loss and geometric consistency loss are weighted and summed to obtain a loss function. The loss function is optimized, and the parameters of the initial visual odometry network are updated until convergence is achieved to obtain the visual odometry network.

[0135] Based on the above steps, the loss function is: L = α1L rec +α2L gc α1 and α2 are hyperparameters used to adjust the proportions of rotational loss and displacement loss.

[0136] In summary, this visual odometry network training method has lower requirements for training data compared to the previous method. It only requires endoscopic images and does not need to collect pose information. Furthermore, since the training images are also captured by real endoscopes, image style transfer is not required. However, the visual odometry network trained using this method has an ambiguous measurement of the motion scale of the real endoscope.

[0137] According to the present invention, an endoscope localization method based on a visual odometry network, before superimposing the pose information of the real endoscope when acquiring the first frame image with all the relative pose estimation information accumulated from acquiring the first frame image to acquiring the current frame image, to obtain the pose information of the real endoscope acquiring the current frame image, further includes:

[0138] The relative pose estimation information obtained from the current frame image and the previous frame image acquired by the real endoscope is scaled to obtain the unit of the relative pose estimation information.

[0139] In this embodiment, the visual odometry network trained by unsupervised learning in the previous embodiment obtains scale-ambiguous relative pose estimation information when locating a real endoscope by acquiring two consecutive frames of images. Therefore, it is necessary to scale the relative pose estimation information of the real endoscope to obtain the unit of the relative pose estimation information.

[0140] The specific calibration methods include the following two, and at least one of the following two methods can be used when performing calibration:

[0141] (1) When the real endoscope enters the cavity, the visible range of the real endoscope is segmented according to a depth threshold. The diameter of the area above the threshold is compared with the depth at the same diameter of the depth peak in the cavity in the preoperative virtual model to obtain the scale of the real endoscope. For example, if the depth threshold is set to 5, the depth portion above the threshold in the depth image extracted by the real endoscope is segmented into a circle with a diameter of 10 pixels. For the virtual model established for the main airway, assuming the real endoscope is in the center of the main airway, the corresponding depth map is contoured, and a circle with a diameter of 10 pixels around the peak can be found. The depth corresponding to this contour line is 1 cm, so the scale of the depth network is 1 / 5 = 0.2 cm.

[0142] (2) Based on the visual odometry network in the above embodiments, its pose network and depth network have the same fuzzy scale. When the endoscope is actually inserted, the relative pose estimation information of the pose network can be compared with the robot control signal for calibration. For example, if the robot control signal controls the endoscope to be inserted 1cm, and the relative translation vector obtained by the pose network is a translation of 2 in the direction of insertion, then the scale is 1 / 2 = 0.5cm.

[0143] An endoscope localization method based on a visual odometry network provided by the present invention, such as... Figure 6 As shown, the pose information of the real endoscope when acquiring the first frame image is obtained, including:

[0144] S601: Acquire the first frame image captured by a real endoscope, and use a pre-trained recurrent generative adversarial network to convert the style of the first frame image into a virtual style.

[0145] Specifically, this method involves obtaining the pose information of a real endoscope when acquiring the first frame image using image retrieval. First, the first frame image acquired by the real endoscope needs to be obtained and then converted into a virtual style.

[0146] S602: Establish a target virtual model, acquire a target virtual image in the target virtual model through a virtual endoscope, and obtain the target pose information of the virtual endoscope when acquiring the target virtual image, wherein the virtual endoscope is determined based on the real endoscope.

[0147] Secondly, this method requires the establishment of a target virtual model. This target virtual model can be a model established for each individual patient before surgery, specifically based on the patient's preoperative CT scan, or it can be a virtual model that can be used for every patient with the same case. For example, a virtual model can be established for the respiratory tract, which can be used for all patients with respiratory problems without the need to establish a model for each patient before surgery in order to obtain the initial pose.

[0148] Virtual endoscopes need to be determined based on real endoscopes. The specific determination method has been introduced in the above embodiments and will not be repeated here.

[0149] S603: Train an image retrieval network using the target virtual image and the target pose information.

[0150] Specifically, the image retrieval network is trained using the target virtual image and target pose information acquired by the target virtual endoscope in the previous step.

[0151] S604: Using a pre-trained image retrieval network, the first frame image converted to a virtual style is retrieved to obtain the pose information of the real endoscope when acquiring the first frame image.

[0152] Specifically, the image retrieval network trained in the above steps is used to retrieve the first frame image converted to a virtual style in step 501, and to retrieve the target virtual image with the highest similarity to the first frame image in the virtual style, thereby obtaining the pose information of the real endoscope when acquiring the first frame image.

[0153] An endoscope localization method based on a visual odometry network provided by the present invention, such as... Figure 7 As shown, obtaining the pose information of the real endoscope when acquiring the first frame image includes:

[0154] S701: Acquire the first frame image captured by a real endoscope, and use a pre-trained recurrent generative adversarial network to convert the style of the first frame image into a virtual style.

[0155] Specifically, this method involves obtaining the pose information of a real endoscope when acquiring the first frame image using image registration. First, the first frame image acquired by the real endoscope needs to be obtained and then converted into a virtual style.

[0156] S702: Establish a target virtual model, acquire a target virtual image captured by a virtual endoscope in the target virtual model, and acquire the target pose information corresponding to the target virtual endoscope when acquiring the target virtual image, wherein the virtual endoscope is determined based on the real endoscope.

[0157] Secondly, this method requires the establishment of a target virtual model, which is a model established for each individual patient before surgery. Specifically, it can be established based on the patient's preoperative CT scan, or it can be a virtual model that can be used for every patient with the same case. For example, a virtual model can be established for the respiratory tract, which can be used for all patients with respiratory problems without the need to establish a model for each patient before surgery in order to obtain the initial pose.

[0158] Virtual endoscopes need to be determined based on real endoscopes. The specific determination method has been introduced in the above embodiments and will not be repeated here.

[0159] S703: The first frame image converted to a virtual style is registered with the target virtual image using an image registration method. The target pose information corresponding to the target virtual image with the highest similarity to the first frame image is the pose information of the real endoscope when acquiring the first frame image.

[0160] During the registration process, the target with the highest similarity to the first frame image of the virtual style is found as the target virtual image. The pose information corresponding to the target virtual image is the target pose information, which is the pose information of the real endoscope when acquiring the first frame image.

[0161] Example: The first frame image acquired by a real endoscope converted to a virtual style Solve for the initial camera pose P0 such that the virtual endoscope image at that position and The highest similarity. That is, to solve:

[0162]

[0163] P′ t As a variable, x(P′) t ) for virtual endoscopy at P′ t The image is in pose, and SIM(·) is the image similarity measure. The Powell optimizer is used, and the initial value is set to a fixed position. For example, in bronchoscopy navigation, if we want to start the positioning from the carina, then the initial value is set to the coordinates of the carina. After giving the initial value calibration signal, the pose information of the real endoscope when acquiring the first frame image is solved from this initial value.

[0164] An endoscope localization method based on a visual odometry network according to the present invention further includes:

[0165] The pose information of the current frame image acquired by the real endoscope is corrected by using a registration method based on an iterative optimization algorithm and running in parallel with the visual odometry.

[0166] Specifically, registration methods based on iterative algorithms include two types: image registration and point cloud registration. Corrections can reduce the cumulative error in the obtained real endoscope localization.

[0167] Image registration methods include:

[0168] The k-th frame image acquired by a real endoscope is used as the current correction image, and the pose information P of the k-th frame image acquired by the real endoscope through the visual odometry is obtained. k , where k≤t;

[0169] Based on image similarity measures, using pose information P k Optimize the solution using the initial values ​​to obtain the corrected pose of the current corrected image:

[0170]

[0171] in To convert the k-th frame image to a virtual style, x(P′) t ) for virtual endoscopy at P′ t The image is in pose, and SIM(·) is the image similarity measure. A Powell optimizer is used, starting from the initial value P′. t =P k Begin iterative solution. Understandably, SIM(·) can be replaced with SSIM (structural similarity), NCC (normalized cross-correlation), or other similarity measures of the design.

[0172] Correct pose It can correct the pose information of the endoscope when acquiring the k-th frame image, and it can also correct the current pose information of the endoscope. For example, when the visual odometry network estimates the relative pose information P of the real endoscope. t+4,t+5 P needs to be solved t+5 When using the above calculation After correction, the pose information of the actual endoscope when acquiring the (t+5)th frame image is obtained:

[0173] Point cloud registration methods include:

[0174] The k-th frame image acquired by the real endoscope is used as the current corrected image, where k ≤ t;

[0175] Based on the k-th frame image x acquired using a real endoscope k Generate depth image Given the actual endoscopic parameters, through depth images It can be converted into the corresponding point cloud.

[0176] A virtual model of the target body is established, which can be a model based on the patient's preoperative CT scan. The point cloud Z-axis corresponding to the k-th frame image acquired at the k-th frame positioning pose of the virtual endoscope within the virtual model of the target body is obtained. k In this context, the virtual endoscope moves along with the real endoscope in the virtual target model. The positioning pose of the virtual endoscope in the k-th frame of the virtual target model is obtained by mapping the positioning pose of the real endoscope when acquiring the k-th frame image to the virtual target model.

[0177] Solving Z using the ICP algorithm k arrive Relative pose between Using relative pose Correct the pose information p obtained from the real endoscope when acquiring the k-th frame image. k And the pose information of the k-th frame image is acquired using a modified real endoscope. The pose of the k-th frame image acquired by the real endoscope and the current pose are corrected.

[0178] For example, using the above calculation By making corrections, we can obtain the pose information of the actual endoscope when acquiring the (t+5)th frame image:

[0179] Understandably, other methods can be used to correct the pose information of the current frame image obtained from the real endoscope, such as the RANSAC algorithm.

[0180] The endoscopic localization method based on a visual odometry network provided by this invention, given the initial pose of the actual endoscope, can obtain the relative pose of the actual endoscope from two consecutive image frames acquired by the endoscope using a pre-trained visual odometry network. This allows for the rapid and continuous acquisition of the current pose information of the actual endoscope by superimposing the relative pose with the initial pose. The visual odometry network in this method can be directly used for different patients after training, eliminating the need for preoperative training, thus saving time and effort.

[0181] Figure 8 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 8As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 can call logic instructions in the memory 830 to execute an endoscope localization method based on a visual odometry network. This method includes: acquiring pose information of a real endoscope when acquiring a first frame image; inputting the current frame image acquired by the real endoscope and the previous frame image acquired into a pre-trained visual odometry network to obtain relative pose estimation information of the real endoscope when acquiring the current frame image and the previous frame image; wherein the style of the current frame image and the previous frame image input into the visual odometry network is consistent with the style of the images used to pre-train the visual odometry network; and superimposing the pose information of the real endoscope when acquiring the first frame image with all accumulated relative pose estimation information from acquiring the first frame image to acquiring the current frame image to obtain the pose information of the real endoscope when acquiring the current frame image.

[0182] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0183] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the endoscope localization method based on the visual odometry network provided by the above methods. The method includes: acquiring the pose information of a real endoscope when acquiring a first frame image; inputting the current frame image acquired by the real endoscope and the previous frame image acquired into a pre-trained visual odometry network to obtain relative pose estimation information of the real endoscope when acquiring the current frame image and the previous frame image; wherein the style of the current frame image and the previous frame image input into the visual odometry network is consistent with the style of the images used to pre-train the visual odometry network; superimposing the pose information of the real endoscope when acquiring the first frame image with all the relative pose estimation information accumulated from acquiring the first frame image to acquiring the current frame image to obtain the pose information of the real endoscope when acquiring the current frame image.

[0184] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the endoscope localization method based on a visual odometry network provided by the above methods. The method includes: acquiring pose information of a real endoscope when acquiring a first frame image; inputting the current frame image acquired by the real endoscope and the previous frame image acquired into a pre-trained visual odometry network to obtain relative pose estimation information of the real endoscope when acquiring the current frame image and the previous frame image; wherein the style of the current frame image and the previous frame image input into the visual odometry network is consistent with the style of the images used to pre-train the visual odometry network; and superimposing the pose information of the real endoscope when acquiring the first frame image with all the relative pose estimation information accumulated from acquiring the first frame image to acquiring the current frame image to obtain the pose information of the real endoscope when acquiring the current frame image.

[0185] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0186] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0187] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An endoscopic localization method based on a visual odometry network, characterized in that, include: The first frame image captured by a real endoscope is acquired, and a pre-trained recurrent generative adversarial network is used to convert the style of the first frame image into a virtual style. A target virtual model is established, and a target virtual image is acquired in the target virtual model through a virtual endoscope, and the target pose information of the virtual endoscope is obtained when the target virtual image is acquired, wherein the virtual endoscope is determined based on the real endoscope; Based on the target virtual image and the target pose information, determine the pose information of the real endoscope when acquiring the first frame image; The current frame image and the previous frame image acquired by a real endoscope are input into a pre-trained visual odometry network to obtain relative pose estimation information when the current frame image and the previous frame image are acquired by the real endoscope; wherein, the style of the current frame image and the previous frame image input into the visual odometry network is consistent with the style of the images used to pre-train the visual odometry network. The pose information of the real endoscope when acquiring the first frame image is superimposed with all the relative pose estimation information accumulated from acquiring the first frame image to acquiring the current frame image to obtain the pose information of the real endoscope when acquiring the current frame image, and the real endoscope is positioned according to the pose information.

2. The endoscopic localization method based on visual odometry network according to claim 1, characterized in that, The visual odometry network was trained in the following manner: A virtual model is established, and virtual images are acquired in the virtual model using a virtual endoscope, and the virtual pose information of the virtual endoscope corresponding to the acquisition of the virtual images is obtained, wherein the virtual endoscope is determined based on the real endoscope; The virtual image is input into the initial visual odometry network, and the initial visual odometry network outputs the relative pose estimation information of the virtual endoscope when acquiring two adjacent frames of virtual images; The virtual pose information is used as the training ground value, and the relative pose ground value information of the virtual endoscope when acquiring the virtual images of two adjacent frames is obtained based on the virtual pose information. The loss function is obtained by weighted summing of translation and rotation losses between the relative pose estimation information and the relative pose truth information of the virtual endoscope; The loss function is optimized, and the parameters of the initial visual odometry network are updated until convergence is achieved to obtain the visual odometry network.

3. The endoscopic localization method based on visual odometry network according to claim 2, characterized in that, The method for ensuring that the style of the current frame image and the previous frame image input to the visual odometry network are consistent with the style of the virtual images used to pre-train the visual odometry network is as follows: A pre-trained recurrent generative adversarial network is used to convert the current frame image and the previous frame image in a realistic style into a virtual style, or to convert the virtual image into a virtual image in a realistic style.

4. The endoscopic localization method based on visual odometry network according to claim 2, characterized in that, The visual odometry network was trained in the following manner: Acquire real endoscopic images captured by a real endoscope; The real endoscope images are used as training data and input into the initial visual odometry network. The initial visual odometry network includes a pose network and a depth network. The pose network is used to estimate the relative pose information of the real endoscope when acquiring two adjacent frames of real endoscope images. The depth network is used to estimate the depth images of the two adjacent frames of real endoscope images respectively. Based on the relative pose information of the real endoscope corresponding to the two adjacent real endoscope images estimated by the pose network and the depth image of the two adjacent virtual images, the previous real endoscope image is warped to obtain the next real endoscope warped image. The reconstruction loss is derived from the view consistency between the warped image of the next frame of the real endoscope obtained through image warping and the real endoscope image of the next frame. Geometric consistency loss is obtained by calculating the scale consistency of the depth images of two adjacent real endoscopic images estimated by the deep network. The reconstruction loss and geometric consistency loss are weighted and summed to obtain the loss function. The loss function is then optimized, and the parameters of the initial visual odometry network are updated until convergence, thus obtaining the visual odometry network.

5. The endoscopic localization method based on visual odometry network according to claim 4, characterized in that, Before obtaining the pose information of the current frame image acquired by the real endoscope by superimposing the pose information of the real endoscope when acquiring the first frame image with all the relative pose estimation information accumulated from the acquisition of the first frame image to the acquisition of the current frame image, the process also includes: The relative pose estimation information obtained from the current frame image and the previous frame image acquired by the real endoscope is scaled to obtain the unit of the relative pose estimation information.

6. The endoscopic localization method based on visual odometry network according to claim 1, characterized in that, The step of determining the pose information of the real endoscope when acquiring the first frame image based on the target virtual image and the target pose information includes: The image retrieval network is trained using the target virtual image and the target pose information; A pre-trained image retrieval network is used to retrieve the first frame image converted to a virtual style, thereby obtaining the pose information of the real endoscope when acquiring the first frame image.

7. The endoscopic localization method based on visual odometry network according to claim 1, characterized in that, The step of determining the pose information of the real endoscope when acquiring the first frame image based on the target virtual image and the target pose information includes: The first frame image, converted to a virtual style, is registered with the target virtual image using an image registration method. The target pose information corresponding to the target virtual image with the highest similarity to the first frame image is the pose information of the real endoscope when acquiring the first frame image.

8. The endoscopic localization method based on a visual odometry network according to any one of claims 1 to 7, characterized in that, Also includes: A registration method based on an iterative optimization algorithm and a visual odometry parallel operation were used to correct the pose information of the current frame image acquired by the real endoscope.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the endoscopic localization method based on a visual odometry network as described in any one of claims 1 to 8.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the endoscopic localization method based on a visual odometry network as described in any one of claims 1 to 8.

Images