Endoscope positioning method, electronic device, and non-transitory computer-readable storage medium

Abstract

The present application provides an endoscope positioning method, an electronic device and a non-transitory computer readable storage medium, comprising: obtaining a depth image of a t-th frame image collected by a real endoscope based on a depth extraction network, obtaining a depth image d of a t-n-th target virtual image collected by a virtual endoscope t‑n , or obtaining a depth image of a t-n-th frame image collected by the real endoscope based on the depth extraction network, inputting the depth image and the depth image d t‑n into a depth registration network to obtain relative pose estimation information of the real endoscope, superimposing the relative pose estimation information and pose estimation information when the real endoscope collects the t-n-th frame image to obtain pose estimation information of the t-th frame image collected by the real endoscope. The method can quickly, accurately and continuously obtain the current pose information of the real endoscope.

Description

Technical Field

[0001] This invention relates to the field of endoscope positioning technology, and more particularly to an endoscope positioning method, electronic device, and non-transitory computer-readable storage medium. Background Technology

[0002] An endoscope is a diagnostic instrument that integrates traditional optics, ergonomics, precision mechanics, modern electronics, mathematics, and software. It includes an image sensor, optical lens, light source, and mechanical devices. It can be inserted into the stomach through the mouth or other natural orifices. Endoscopes can visualize lesions that are not visible to X-rays, making them a commonly used technique in medical examinations.

[0003] Currently, commonly used methods for endoscopic localization include: (1) Extracting the depth of the endoscopic image using the Shape from Shading (SFS) method, identifying the deeper parts as airways. After extracting the airways, comparing the model reconstructed from preoperative CT, the current image is mapped to the airway branch where the camera is located, or the endoscopic motion is estimated based on the change in the deepest part of the airway in adjacent images. This method can be achieved at airway bifurcation points, but it is difficult to provide continuous endoscopic localization information when there is no airway or only one airway in the field of view. (2) Extracting feature points of the endoscopic image using the Structure From Motion (SFM) method, matching the feature points one by one for two adjacent frames, and calculating the Perspective-n-Point (PnP) based on this to estimate the endoscopic pose. This method cannot solve the Perspective-n-Point (PnP) when there are few or no feature points in the endoscopic image, resulting in the problem of lost endoscopic localization. (3) 2D / 3D registration method: The 2D images captured by the endoscope are registered onto the virtual model reconstructed before the operation to obtain the position of the endoscope in the model. This method is based on an iterative optimization algorithm, so obtaining the localization of each frame requires a long computation time. However, the pose of the endoscope changes rapidly during actual examination, and excessive computation time can easily lead to localization loss. Summary of the Invention

[0004] This invention provides an endoscope positioning method, an electronic device, and a non-transitory computer-readable storage medium to address the shortcomings of existing technologies that cannot provide continuous positioning information and are prone to positioning loss, thereby achieving rapid and accurate positioning of the endoscope and obtaining continuous pose information.

[0005] This invention provides an endoscope positioning method, comprising:

[0006] Depth image of the t-th frame acquired by a real endoscope is obtained using a pre-trained depth extraction network.

[0007] Obtain the depth image d of the target virtual image at frame tn, acquired by the virtual endoscope at the pose of frame tn within the target virtual model. t-n Alternatively, the depth image of the nth frame of the real endoscope can be obtained based on the pre-trained depth extraction network. The virtual endoscope is determined based on the real endoscope;

[0008] The depth image and the depth image d t-n Or the depth image and the depth image Inputting the pre-trained deep registration network, the relative pose estimation information of the real endoscope when acquiring the t-th frame image and the tn-th frame image is obtained.

[0009] The relative pose estimation information Pose estimation information compared with the nth frame image acquired by the real endoscope By superimposing the images, the pose estimation information of the t-th frame image acquired by the real endoscope is obtained. And based on the pose estimation information The actual endoscope is positioned.

[0010] According to an endoscope localization method provided by the present invention, the depth extraction network is a depth extraction network based on a recurrent generative adversarial network and a pre-trained depth registration network. The recurrent generative adversarial network includes a first generator, a first discriminator, a second generator, and a second discriminator. The first generator is used to convert a depth image into a realistic endoscope image, and the second generator is used to convert a realistic endoscope image into a depth image.

[0011] The deep extraction network based on the recurrent generative adversarial network and the deep registration network is trained in the following manner:

[0012] A virtual model is established, and the depth image of the virtual image captured by the virtual endoscope in the virtual model is obtained, and the virtual pose information of the virtual endoscope corresponding to the virtual image is obtained when the virtual image is captured;

[0013] Acquire preset real endoscopic images;

[0014] The preset real endoscope image, the depth image of the virtual image, and the virtual pose information are used as training data to perform weakly supervised training on the initial depth extraction network.

[0015] The loss function is obtained by weighted summation of the cycle consistency loss, identity loss, generative adversarial loss, reconstruction loss, and geometric consistency loss constrained by the initial deep extraction network;

[0016] The loss function is optimized, and the parameters of the initial depth extraction network based on the recurrent generative adversarial network and the depth registration network are updated until a preset number of rounds are reached to obtain the depth extraction network based on the recurrent generative adversarial network and the depth registration network.

[0017] According to the endoscope localization method provided by the present invention, the depth extraction network is a depth extraction network based on SfMLearner or a depth extraction network based on recurrent generative adversarial network.

[0018] In the depth image and the depth image d t-n Or the depth image and the depth image Before inputting the pre-trained deep registration network, the method further includes:

[0019] For the depth image and the depth image Scale calibration is performed to obtain the depth image. and the depth image The unit.

[0020] According to the endoscopic localization method provided by the present invention, the depth registration network is trained in the following manner:

[0021] A virtual model is established, the depth image of the virtual image acquired by the virtual endoscope in the virtual model is obtained, and the virtual pose information corresponding to the virtual image acquired by the virtual endoscope is obtained.

[0022] The depth image of the virtual image is input into the initial depth registration network, and the initial depth registration network outputs the relative pose estimation information of the virtual endoscope when acquiring two adjacent frames of virtual images;

[0023] The virtual pose information is used as the training ground value, and the virtual relative pose information when the virtual endoscope acquires the virtual images of two adjacent frames is obtained based on the virtual pose information.

[0024] The loss function is obtained by weighted summing of translation and rotation losses between the relative pose estimation information and the virtual relative pose information.

[0025] The loss function is optimized, and the parameters of the initial depth registration network are updated until convergence is achieved to obtain the depth registration network.

[0026] An endoscope positioning method provided by the present invention further includes:

[0027] The registration method based on the iterative optimization algorithm is used in parallel with the deep registration network. The pose estimation information of the real endoscope is corrected based on the corrected pose obtained by the registration method based on the iterative optimization algorithm, thereby eliminating the cumulative error.

[0028] According to the present invention, an endoscope positioning method is provided, comprising a method for obtaining a corrected pose based on a registration method using an iterative optimization algorithm, including:

[0029] The k-th frame image acquired by a real endoscope is used as the current corrected image, and the depth image of the k-th frame image is obtained through the depth extraction network. Where k≤t;

[0030] Obtain pose estimation information of the k-th frame image acquired by the real endoscope based on the depth registration network.

[0031] Using the current corrected image or the depth image Or the current corrected image and the depth image Semantic segmentation is performed on the cavity images in the actual endoscopic field of view;

[0032] Based on image similarity measures and semantic segmentation similarity measures, pose estimation information is used. The initial values ​​are optimized to obtain the corrected pose of the current corrected image.

[0033] The pose estimation information when the real endoscope acquires the k-th frame image Replace with the corrected pose

[0034] According to the present invention, an endoscope positioning method is provided, comprising a method for obtaining a corrected pose based on a registration method using an iterative optimization algorithm, including:

[0035] The k-th frame image acquired by a real endoscope is used as the current corrected image, and the depth image of the k-th frame image is obtained through the depth extraction network. Where k≤t;

[0036] The depth image d of the k-th frame virtual image of the target virtual model acquired by the virtual endoscope at the k-th frame positioning pose is obtained. k ;

[0037] The depth image Convert to the corresponding point cloud The depth image dk Convert to point cloud image Y k ;

[0038] Solving Y using the ICP algorithm k arrive Relative pose between

[0039] Using the relative pose Correcting the pose estimation information when the real endoscope acquires the k-th frame image

[0040] An endoscope positioning method provided by the present invention further includes:

[0041] The feature information of the t-th frame image acquired by a real endoscope is extracted using the RGB image feature extraction method. The feature information of the t-th frame image is then combined with the depth image. The pre-trained deep registration network is then input together;

[0042] The RGB image feature extraction method is used to extract feature information of the nth frame image acquired by a real endoscope or to extract feature information of the nth frame target virtual image acquired by a virtual endoscope. The feature information of the nth frame target virtual image is extracted after texture mapping of the nth frame target virtual image.

[0043] The feature information of the target virtual image in the nth frame and the depth image d are used. t-n Alternatively, the feature information of the nth frame image and the depth image can be combined. Input the pre-trained deep registration network.

[0044] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the endoscope positioning method as described above.

[0045] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the endoscope positioning method as described above.

[0046] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the endoscope positioning method as described above.

[0047] The endoscopic positioning method provided by this invention, by knowing the initial pose of the actual endoscope, can quickly, accurately, and continuously obtain the current pose information of the actual endoscope using a pre-trained depth extraction network and a depth registration network. The depth extraction network and depth registration network in this method can be directly used for different patients after training, without the need for preoperative training, which is convenient and saves time. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0049] Figure 1 This is one of the flowcharts illustrating the endoscope positioning method provided by the present invention;

[0050] Figure 2 This is a schematic diagram of the deep extraction network structure provided by the present invention;

[0051] Figure 3 This is a flowchart illustrating the training method for the deep extraction network provided by the present invention;

[0052] Figure 4(a) is a schematic diagram of the deep extraction network generator architecture provided by the present invention;

[0053] Figure 4(b) is a schematic diagram of the ResNet block architecture of the deep extraction network provided by the present invention.

[0054] Figure 4(c) is a schematic diagram of the deep extraction network discriminator architecture provided by the present invention.

[0055] Figure 5 This is a flowchart illustrating the training method for the deep registration network provided by the present invention.

[0056] Figure 6 This is a schematic diagram of the deep registration network architecture provided by the present invention;

[0057] Figure 7 This is one of the flowcharts illustrating the method for obtaining a corrected pose using a registration method based on an iterative optimization algorithm provided by the present invention.

[0058] Figure 8 This is the second flowchart illustrating the method for obtaining corrected pose using the registration method based on iterative optimization algorithm provided by this invention.

[0059] Figure 9 This is the second flowchart of the endoscope positioning method provided by the present invention;

[0060] Figure 10 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0062] The following is combined Figures 1-9 The endoscope positioning method of the present invention is described as follows: Figure 1 As shown, the method includes:

[0063] S101: Obtain the depth image of the t-th frame of a real endoscope image based on a pre-trained depth extraction network.

[0064] In this embodiment of the invention, the endoscopic localization method can be used in natural cavities of the human body such as the respiratory tract, biliary tract, and ventricles. The method first requires acquiring the depth image of the current frame (frame t) captured by a real endoscope. A depth image, also known as a range image, is an image that uses the distance (depth) from the image acquisition device to various points in the scene as pixel values. It directly reflects the geometry of the visible surfaces of objects. A depth image can be converted into point cloud data through coordinate transformation, and point cloud data with regularity and necessary information can also be converted back into depth image data.

[0065] S102: Obtain the depth image d of the target virtual image at frame tn, acquired by the virtual endoscope at the positioning pose of frame tn in the target virtual model. t-n Alternatively, the depth image of the nth frame of the real endoscope can be obtained based on the pre-trained depth extraction network. The virtual endoscope is determined based on the real endoscope.

[0066] Specifically, the depth image d of the target virtual image in frame tn, acquired by the virtual endoscope at the positioning pose of frame tn in the target virtual model, is obtained. t-n Alternatively, obtain the depth image of the nth frame of the image acquired by the real endoscope. In the target virtual model, the virtual endoscope moves along with the real endoscope. The virtual endoscope's pose at frame tn in the target virtual model is obtained by mapping the pose of the real endoscope when acquiring the image at frame tn to the pose in the target virtual model. Here, n ≤ 10, meaning the images within the first ten frames of the current frame, to ensure that frames tn and t have many similar feature points. The value of n in this method is not fixed. For example, when the current frame is the 8th frame, tn can be equal to 7 (the 7th frame), in which case n = 1; or it can be equal to 3 (the 3rd frame), in which case n = 5. When the current frame is the 9th frame, tn can be equal to 7 (the 7th frame), in which case n = 2.

[0067] Virtual endoscopy needs to be based on real endoscopy for determination, therefore the internal parameters of virtual endoscopy need to be consistent with those of real endoscopy.

[0068] Example: Use MATLAB software to perform checkerboard calibration on a real endoscope to obtain the endoscope's intrinsic parameters.

[0069] The internal references of a real endoscope are:

[0070]

[0071] Image pixel count:

[0072] Width * Length = width × height

[0073] make:

[0074] Average focal length:

[0075] The x-coordinate of the window center is: wcx = -2 × (cx - width / 2) / width

[0076] Window center y-axis coordinate: wcy = 2 × (cy - height / 2) / height

[0077] At this point, when designing a virtual endoscope, the parameters of the virtual endoscope are:

[0078] Field of view:

[0079] ViewAngle=180 / π*(2.0*atan2(height / 2.0,focal_length))

[0080] Window size:

[0081] WindowSize = [width, height]

[0082] Center of window:

[0083] WindowCenter = [wcx, wcy]

[0084] S103: Transfer the depth image and the depth image d t-n Or the depth image and the depth image Inputting the pre-trained deep registration network, the relative pose estimation information of the real endoscope when acquiring the t-th frame image and the tn-th frame image is obtained.

[0085] Specifically, this can be achieved by using depth images and depth image d t-n Inputting the pre-trained deep registration network, we obtain the relative pose estimation information of the t-th frame image acquired by the real endoscope and the tn-th frame image acquired. Depth images can also be used and depth images Inputting the pre-trained deep registration network, the relative pose estimation information of the real endoscope when acquiring the t-th frame image and the tn-th frame image is obtained.

[0086] S104: Transfer the relative pose estimation information Pose estimation information compared with the nth frame image acquired by the real endoscope By superimposing the images, the pose estimation information of the t-th frame image acquired by the real endoscope is obtained. And based on the pose estimation information The actual endoscope is positioned.

[0087] Specifically, the obtained relative pose estimation information Pose estimation information compared with the nth frame image acquired by the real endoscope By superimposing the images, the pose estimation information of the t-th frame image acquired by the real endoscope can be obtained. Based on this pose estimation information The actual endoscope is positioned.

[0088] pose information of the actual initial position of the endoscope This information can be obtained during the initialization of the deep registration network.

[0089] In one embodiment, such as Figure 2As shown, the depth extraction network is a depth extraction network based on a recurrent generative adversarial network and a pre-trained depth registration network. The recurrent generative adversarial network includes a first generator, a first discriminator, a second generator, and a second discriminator. The first generator is used to convert the depth image into a realistic endoscopic image, and the second generator is used to convert the realistic endoscopic image into a depth image.

[0090] like Figure 3 As shown, the deep extraction network based on the recurrent generative adversarial network and the deep registration network is trained in the following manner:

[0091] S301: Establish a virtual model, obtain the depth image of the virtual image captured by the virtual endoscope in the virtual model, and obtain the virtual pose information of the virtual endoscope when the virtual image is captured.

[0092] Specifically, before training the aforementioned deep extraction network, a deep registration network needs to be trained first. This deep extraction network then applies the trained deep registration network. Image style refers to the texture, color, and visual patterns at different spatial scales within an image.

[0093] In practice, obtaining the pose of the endoscope during a real endoscopic examination is quite difficult. Therefore, we need to build a virtual model. By using a virtual endoscope to obtain a large amount of depth images and virtual pose information, we can train and supervise the depth extraction network, thereby improving the robustness of the depth extraction network. There can be various virtual models, such as virtual models for the respiratory tract, virtual models for the biliary tract, etc., and corresponding virtual models can be built according to the needs of use.

[0094] S302: Obtain a preset real endoscope image.

[0095] The target body corresponding to the preset real endoscope image is consistent with the target body corresponding to the virtual model. For example, if the virtual model is a virtual model of the respiratory tract, then the preset real endoscope image is also an image of the respiratory tract.

[0096] S303: The preset real endoscope image, the depth image of the virtual image, and the virtual pose information are used as training data to perform weakly supervised training on the initial depth extraction network.

[0097] Specifically, the depth images and virtual pose information obtained in the above steps are used as training data to perform weakly supervised training on the initial depth extraction network.

[0098] S304: The loss function is obtained by weighted summation of the cycle consistency loss, identity loss, generative adversarial loss, reconstruction loss, and geometric consistency loss constrained by the initial deep extraction network.

[0099] For details, please refer to Figure 2 Cycle GAN, a generative adversarial network, includes a first generator G. image First discriminator G image Second generator G depth Second discriminator D dept Let Z and X be the depth image domain and the endoscope image domain, respectively.

[0100] Cyclic consistency loss:

[0101] For an endoscopic image x∈X, the depth extraction algorithm aims to learn a mapping G. dept : X→Z, from x t Generate its corresponding depth image Next, map G image Z→X will Reconstructing to domain X completes the cycle; the cycle consistency loss refers to... Reconstructed to domain X and x t The difference is significant. The transformation from the Z domain to the X domain is similar. In this reconstruction loop, the network model... image G depth Apply cycle consistency loss:

[0102]

[0103] Where y is a variable representing a certain frame of the image, and p represents the probability distribution. It expresses expectation.

[0104] Loss of identity:

[0105] To add constraints to the learning of mappings, we propose identity loss:

[0106]

[0107] Generate adversarial loss:

[0108] While the generator completes the mapping loop, the discriminator D... image D depth The algorithm learns to distinguish whether the input endoscope image and depth image are real or fake; the generator aims to fool the discriminator by generating images that the discriminator considers real. Therefore, a generative adversarial loss is introduced, which can be LS-GAN loss.

[0109]

[0110] The dot (·) is used to omit the image or depth. y~p(data) represents that the samples follow a distribution in the domain X or Y.

[0111] Reconstruction losses:

[0112] To enable the network to learn depth image estimation at a given scale, the motion trajectory of a virtual endoscope can be acquired from a virtual model, recording the virtual endoscope pose and corresponding depth image at each moment. The acquired virtual endoscope pose and corresponding depth image are then used to apply visual input between image frames acquired by the generated real endoscope. Figure 1 Consistency constraints are added based on Perspective-n-point (PnP) to the adversarial loss, along with image perspective constraints. Figure 1 Causes sexual harm.

[0113] Depth image z t-n and z t The inputs are respectively fed into generator G. image The generated endoscopic images can be obtained. and Since the virtual pose information at times tn and t is also recorded during data acquisition, the virtual relative pose p from time tn to time t can be calculated. t-n,t =(t x ,t y ,t z Given the camera intrinsic parameters K, the pixel points in homogeneous coordinates are... It can be warped to the following formula

[0114]

[0115] in, R is the camera translation vector from time tn to time t; R is the camera rotation matrix from time tn to time t. t-n,t Calculated by the following formula:

[0116]

[0117] Among them, α1=sinθ, α2=sinφ, α3=sinψ, β1=cosθ, β2=cosφ, β3=cosψ.

[0118] Use n ≤ 5 during training. Too large an n cannot guarantee sufficient common viewing area between two images.

[0119] because Typically, the coordinates are non-integer, requiring hyperbolic sampling to obtain integer pixel coordinates, ultimately yielding the result from... warped image and Should be with Consistent, therefore by sight Figure 1 Consistency is lost during reconstruction:

[0120]

[0121] Where w(·) is the warp to Spatial operators, It is by and through relative translation vector t t-n,t and relative rotation vector R t-n,t The depth image obtained by reprojection; G represents a pixel in image x. Therefore, G image They are encouraged to learn an unbiased estimate from a depth image to the corresponding endoscopic image. Due to the constraint of cycle consistency, G... depth They will also be encouraged to learn unbiased estimation from endoscopic images to depth images, i.e., to generate depth images with the same scale as the input depth map.

[0122] To further constrain the generator G depth Learning X→Z, from the perspective of Figure 1 Coherence was also added to x t-n and x t and the generated depth map and Although the relative pose of the endoscope cannot be captured at this time, a pre-trained depth registration network with a depth-based pose estimation algorithm can be used to... and The relative pose of the endoscope can be calculated. A pre-trained pose estimation network is loaded during training to estimate the relative motion p of the endoscope. t-n,t At this point, an ideal depth image estimate should include information about the endoscope's motion captured by the pose estimation network, thus obtaining the depth image from the view. Figure 1 Reconstruction loss obtained through homogeneity:

[0123]

[0124] Therefore, the total view is obtained. Figure 1 Loss of reconstructive function:

[0125]

[0126] Geometric consistency loss:

[0127] For the generated depth map and If they correspond to the same 3D scene, then their depth information should be consistent. Depth map and Inconsistency z diff Defined as:

[0128]

[0129] in, It is by The relative pose p of the virtual endoscope calculated by a pre-trained deep registration network. t-n,t Depth image obtained by reprojection. From The depth map obtained from sampling. Here, the calculation is performed. and The error, rather than and The error is because The result of reprojection is not in an integer coordinate system, so it needs to be... Sample to the same coordinate system to calculate the difference between the two.

[0130] Geometric consistency loss is defined as:

[0131]

[0132] in, This represents a pixel in the image z.

[0133] In summary, the total loss function for training a deep extraction network is:

[0134]

[0135] Where β, γ, δ, θ1, θ2, and η are hyperparameters for adjusting the weights of each loss.

[0136] S305: Optimize the loss function and update the parameters of the initial depth extraction network based on the recurrent generative adversarial network and the depth registration network until a preset number of rounds are reached, so as to obtain the depth extraction network based on the recurrent generative adversarial network and the depth registration network.

[0137] like Figure 4(a) , 4(b) Figure 4(c) shows a schematic diagram of the deep extraction network architecture, where (a) is the generator, (b) is a ResNet block in the deep registration network, and (c) is the discriminator. The tensor dimension shown in this figure is based on an input image of size 1×256×256; Res(256, 256) represents a ResNet block with 256 input and output channels; IN represents the InstanceNorm layer, and Leaky ReLU represents the Leaky ReLU activation function.

[0138] For example, the depth extraction network can be trained using 7 preset real endoscope videos and 8 videos acquired via virtual endoscopes, including multiple preset real endoscope images, 2187 depth images, and corresponding virtual endoscope poses. In the depth extraction network architecture, the generator is a conventional encoder-decoder architecture, where the bottleneck layer consists of six ResNet blocks, and the discriminator consists of five convolutional layers. The Adam optimizer is used for 100 epochs of training. At the start of training, the learning rate is set to 0.001 and θ1 = θ2 = η = 0 to avoid imposing consistency constraints on early depth maps with poor generation results. After 10 epochs, θ1, θ2, and η are set to 0.3, 5, and 5, respectively. β, γ, and δ are set to 10, 5, and 1, respectively, throughout the training process.

[0139] During training, the loss function obtained from the above steps is continuously optimized to update the parameters of the depth extraction network until a predetermined number of epochs is determined, which can be 50-300 epochs, or even 100-200 epochs. This trained depth extraction network, compared to depth extraction networks like SfMLearner, can generate depth images with clearer contours. Compared to using only depth extraction networks like Cycle GAN, it ensures that the structure of the input image is not altered. It can generate scale-stable and predictable depth images (the scale is essentially the same as the training data scale).

[0140] In one embodiment, the deep extraction network is a deep extraction network based on SfMLearner or a deep extraction network based on recurrent generative adversarial networks.

[0141] In the depth image and the depth image d t-n Or the depth image and the depth image Before inputting the pre-trained deep registration network, the method further includes:

[0142] For the depth image and the depth image Scale calibration is performed to obtain the depth image. and the depth image The unit.

[0143] Specifically, for deep extraction networks based on SfMLearner:

[0144] Simultaneously train a depth estimation network and a pose network. The depth estimation network estimates the depth information z from an input endoscope image, while the pose network estimates the camera relative pose T and R between two input endoscope images.

[0145] For two consecutive input endoscopic images x t-n x t Depth estimation networks can estimate the depth of two frames. and Pose-based networks can estimate the relative motion t of the camera. t-n,t and R t-n,t .

[0146] Given the camera intrinsic parameter K, the pixel points in homogeneous coordinates It can be warped to the following formula

[0147]

[0148] because Typically, the coordinates are non-integer, requiring hyperbolic sampling to obtain integer pixel coordinates, ultimately yielding the result from... warped image Should be with Consistent. From the perspective of Figure 1 Consistency is lost during reconstruction:

[0149]

[0150] Where w(·) is the warp to Spatial operators, It is by and through relative translation vector t t-n,t and relative rotation vector R t-n,t The depth image obtained by reprojection; Let x represent a pixel in the image. Warping refers to manipulating the image to deform the pixels within it. Through this loss function, the pose network and depth estimation network can achieve self-supervision, thus completing the network training.

[0151] To ensure scale stability of the depth images generated by the network, a geometric consistency loss is added. For the generated depth images... and If they correspond to the same 3D scene, then their depth information should be consistent. Depth map and Inconsistency z diff Defined as:

[0152]

[0153] in It is by The relative motion of the real endoscope calculated by the pose network Depth map obtained by reprojection. From The depth map obtained from sampling. Here, the calculation is performed. and The error, rather than and The error is because The result of reprojection is not in an integer coordinate system, so it needs to be... Sample to the same coordinate system to calculate the difference between the two.

[0154] Geometric consistency loss is defined as:

[0155]

[0156] in, This represents a pixel in the image z.

[0157] In summary, the loss function is: L = aL rec +bL gc , where a and b are hyperparameters for adjusting the weights of each loss.

[0158] Specifically, for deep extraction networks based on Cycle GAN, the loss function can include the following loss:

[0159] For an endoscopic image x∈X, the depth extraction algorithm aims to learn a mapping G. depth : X→Z, generate the corresponding depth map from x. Next, map G image Z→X will The network model is reconstructed to domain X, thus completing the loop. The transformation from domain Z to domain X is similar. In this reconstruction loop, the network model... image G depth Apply cycle consistency loss:

[0160]

[0161] Where p represents the probability distribution. It expresses expectation.

[0162] To add constraints to the learning of the mapping, other loss functions include identity loss:

[0163]

[0164] While the generator completes the mapping loop, the discriminator D... image D dept The algorithm learns to distinguish whether the input endoscope image and depth image are real or fake; the generator aims to fool the discriminator by generating images that the discriminator considers real. Therefore, a generative adversarial loss is introduced, specifically the LS-GAN loss.

[0165]

[0166] The dot (·) is used to omit the image or depth. y~p(data) represents that the samples follow a distribution in the domain X or Y.

[0167] It is difficult to guarantee the generation of scale-stable depth images using only Cycle GAN, so geometric consistency loss can also be considered.

[0168] The depth images obtained by the two depth extraction networks described above are blurry and unitless, therefore calibration is required. When calibrating the dimensions, specific calibration methods include the following two, and at least one of these two methods can be used during calibration:

[0169] (1) When the real endoscope enters the cavity, the visible range of the real endoscope is segmented according to a depth threshold. The diameter of the area above the threshold is compared with the depth at the same diameter of the depth peak in the cavity in the preoperative virtual model to obtain the scale of the real endoscope. For example, if the depth threshold is set to 5, the depth portion above the threshold in the depth image 0 extracted by the real endoscope is segmented into a circle with a diameter of 10 pixels. For the virtual model established for the main airway, assuming the real endoscope is in the center of the main airway, the corresponding depth map is contoured, and a circle with a diameter of 10 pixels around the peak can be found. The depth corresponding to this contour line is 1 cm, so the scale of the depth network is 1 / 5 = 0.2 cm.

[0170] (2) Based on the depth extraction network in the above embodiments, its pose network and depth network have the same fuzzy scale. When the endoscope is actually inserted, the relative pose estimation information of the pose network can be compared with the robot control signal for calibration. For example, if the robot control signal controls the endoscope to be inserted 1cm, and the relative translation vector obtained by the pose network is a translation of 2 in the direction of insertion, then the scale is 1 / 2 = 0.5cm.

[0171] In one embodiment, such as Figure 5 As shown, the deep registration network is trained in the following manner:

[0172] S501: Establish a virtual model, obtain the depth image of the virtual image captured by the virtual endoscope in the virtual model, and obtain the virtual pose information corresponding to the virtual image captured by the virtual endoscope.

[0173] Specifically, the depth registration network is a deep neural network in the form of an encoder-decoder. The network input consists of two frames of depth information. The encoder adopts the structure of the FlowNetC encoder (the optical flow extracted by FlowNet is a simulation of the motion field), and the decoder uses several layers of CNN (Convolutional Neural Network) to finally transform the encoded information into 6DOF (i.e., 3D translation and 3D Euler angle) pose parameters for output.

[0174] When training a deep registration network, a virtual model needs to be established first. A large amount of depth images and virtual pose information are obtained through a virtual endoscope to supervise the training of the deep registration network and improve its robustness.

[0175] S502: Input the depth image of the virtual image into the initial depth registration network, and the initial depth registration network outputs the relative pose estimation information of the virtual endoscope when acquiring two adjacent frames of virtual images.

[0176] Specifically, the depth image of the virtual image obtained in the above steps is input into the initial depth registration network for weakly supervised training. The output of the initial depth registration network can obtain the relative pose estimation information of the virtual endoscope when acquiring two adjacent frames of virtual images.

[0177] S503: Using the virtual pose information as the training ground value, obtain the virtual relative pose information when the virtual endoscope acquires the virtual images of two adjacent frames based on the virtual pose information.

[0178] Meanwhile, the virtual pose information is used as the training ground value. By calculating the virtual pose information, the virtual relative pose information when the virtual endoscope acquires the virtual images of two adjacent frames can be obtained. At this time, the relative pose ground value information and relative pose estimation information when the virtual endoscope acquires the images of two adjacent frames are obtained.

[0179] S504: The loss function is obtained by weighted summation of translation loss and rotation loss between the relative pose estimation information and the virtual relative pose information.

[0180] Specifically, the translational and rotational losses between the relative pose estimation information of the virtual endoscope and the real relative pose are calculated separately, and the translational and rotational losses are weighted and summed to obtain the final loss function:

[0181] L(z t-m ,z t ) = L t (z t-m ,z t )+ωL r (z t-m ,zt )

[0182] Among them, L t For translation loss: These are the translation vectors in the true relative pose information and the relative pose estimation information, respectively; L r For rotational losses: These are the rotation vectors in the true relative pose information and the relative pose estimation information, respectively; ω is a hyperparameter used to adjust the proportions of rotation loss and displacement loss.

[0183] like Figure 6 The diagram shown is a schematic of the architecture for deep registration.

[0184] The pose estimation network was trained using 37 segments of virtual endoscope pose and depth images acquired via virtual endoscope trajectories, comprising 11,904 frames. The network employed a pre-trained FlowNetC encoder to regress the pose vector using three convolutional blocks. The network was trained using the Adam optimizer with an initial learning rate of 1e-5 and a training time of 300 epochs. ω was set to 100.

[0185] S505: Optimize the loss function and update the parameters of the initial depth registration network until convergence is achieved, thereby obtaining the depth registration network.

[0186] This depth extraction network learns the endoscope pose transformation parameters between two input depth images using deep learning methods, thereby updating the endoscope pose transformation for each input endoscope image. This depth registration network is based on depth registration rather than image intensity, eliminating the need for additional rendering requirements on virtual images acquired by the virtual endoscope in the simulator. The deep learning algorithm directly estimates the pose transformation, enabling fast, real-time operation and providing real-time localization results.

[0187] In one embodiment, it also includes:

[0188] The registration method based on the iterative optimization algorithm is used in parallel with the deep registration network. The pose estimation information of the real endoscope is corrected based on the corrected pose obtained by the registration method based on the iterative optimization algorithm, thereby eliminating the cumulative error.

[0189] Specifically, the registration method based on iterative optimization algorithm is relatively slow. Running it in parallel with the deep registration network for pose correction can delay the correction of the pose estimation information of the real endoscope, so that the accumulated error will not continue to increase and the positioning accuracy will be improved.

[0190] In one embodiment, such as Figure 7As shown, the method for obtaining the corrected pose based on the registration method using an iterative optimization algorithm includes:

[0191] S701: Obtain the k-th frame image acquired by a real endoscope as the current corrected image, and obtain the depth image of the k-th frame image through the depth extraction network. Where k≤t.

[0192] Specifically, this correction method is slower than the network used to estimate the actual endoscope pose. Therefore, it does not perform frame-by-frame correction during parallel correction. The k-th frame (k≤t) is used as the current correction image, meaning that the actual endoscope pose estimation information corresponding to the image frame used as the correction image has already been estimated.

[0193] S702: Obtain pose estimation information of the k-th frame image acquired by the real endoscope based on the depth registration network.

[0194] Specifically, since k ≤ t, when correcting the k-th frame image, the pose estimation information of the k-th frame image... It has already been estimated and can be obtained directly.

[0195] S703: Utilize the current corrected image or the depth image Or the current corrected image and the depth image Semantic segmentation is performed on the cavity images in the actual endoscopic field of view.

[0196] Experiments revealed that because a similarity metric is used during registration, when an image contains both a deep cavity and several shallow cavities, the deeper cavity, due to its greater depth, is prioritized for alignment during registration optimization, potentially neglecting the registration of shallower cavities. This leads to the ignoring of structural information in the shallower cavities. To address this issue, cavity image segmentation is performed using depth images before registration. The registration process then needs to align not only to cavities of similar depth but also to those with similar cavity structures.

[0197] Here, segmentation refers to dividing the image of all cavities within the detection field of view into regions, i.e., partitioning. For the input endoscopic image x t Depth images can be used Or an RGB image x t Or RGBD image (x t and The cavity can be segmented. Segmentation methods can include segmenting the depth image using a depth threshold, or training a network to learn cavity segmentation for RGB or RGBD images.

[0198] S704: Based on image similarity measures and semantic segmentation similarity measures, using pose estimation information The initial values ​​are optimized to obtain the corrected pose of the current corrected image.

[0199] Specifically, this method is a correction method based on image registration, where the segmentation process is denoted as Seg(·), and the corrected pose of the real endoscope at time k is... The corresponding airway segmentation result is Given the camera pose at time t-1, starting from the initial pose value Start optimization solution The optimization process is described as follows:

[0200]

[0201] Where SIM1(·) is the image similarity measure, SIM2(·) is the segmentation similarity measure, and P′ t As a variable. Seg(P′) t Let P' be the virtual pose of the virtual endoscope. t The segmentation result is obtained from the corresponding image or depth map. The Powell algorithm is also used as the optimization strategy. For example, when k = t, the latest calculated pose estimation information is used. Using these initial values ​​for optimization can improve the convergence of the algorithm and reduce the number of iterations.

[0202] This method can compensate for the inaccurate calculation caused by using only image similarity measures, such as NCC (Normalized Cross Correlation), which focuses on the deep cavity part of the two depth maps and ignores the features of the shallow cavity when there are two cavities with one deep and one shallow cavity.

[0203] S705: Obtain the pose estimation information of the real endoscope when acquiring the k-th frame image. Replace with the corrected pose

[0204] After obtaining the corrected pose, the pose estimation information from the k-th frame image acquired by the real endoscope is used. Replace with corrected pose At this point, the pose of the endoscope when acquiring the t-th frame image along the actual endoscopic trajectory is corrected. In one embodiment, such as... Figure 8 As shown, the method for obtaining the corrected pose based on the registration method using an iterative optimization algorithm includes:

[0205] S801: Obtain the k-th frame image acquired by a real endoscope as the current corrected image, and obtain the depth image of the k-th frame image through the depth extraction network. Where k≤t.

[0206] Specifically, this correction method is slower than the network used to estimate the true endoscope pose information. Therefore, when performing parallel correction, it is not performed frame-by-frame. Instead, the k-th frame (k≤t) is used as the current correction image.

[0207] S802: Obtain the depth image d of the k-th frame target virtual image captured by the virtual endoscope at the k-th frame positioning pose in the target virtual model. k .

[0208] Specifically, the virtual endoscope moves along with the real endoscope in the target virtual model. The positioning pose of the virtual endoscope in the target virtual model at frame k is obtained by mapping the positioning pose of the real endoscope when acquiring the image at frame k to the positioning pose in the target virtual model.

[0209] S803: Transfer the depth image Convert to the corresponding point cloud The depth image d k Convert to point cloud image Y k .

[0210] S804: Solving Y using the ICP algorithm k arrive Relative pose between

[0211] S805: Adopt the aforementioned relative pose Correcting the pose estimation information when the real endoscope acquires the k-th frame image

[0212] Specifically, the relative pose is adopted. Correcting the pose estimation information when the real endoscope acquires the k-th frame image At this point, the pose of the k-th frame image acquired on the actual endoscopic trajectory is corrected.

[0213] In one embodiment, the method further includes:

[0214] S901: The feature information of the t-th frame image acquired by the real endoscope is extracted using the RGB image feature extraction method. The feature information of the t-th frame image and the depth image are then combined. The pre-trained deep registration network is then input together;

[0215] S902: Use the RGB image feature extraction method to extract feature information of the nth frame image acquired by a real endoscope or extract feature information of the nth frame target virtual image acquired by a virtual endoscope, wherein the feature information of the nth frame target virtual image is extracted after texture mapping of the nth frame target virtual image;

[0216] S903: Combine the feature information of the target virtual image in the nth frame with the depth image d t-n Alternatively, the feature information of the nth frame image and the depth image can be combined. Input the pre-trained deep registration network.

[0217] Current algorithms only use RGB image information or only depth information. Although depth-based localization techniques have been shown to be more robust, in practical applications, when there is only one cavity in the field of view, there will be a circular depth peak region in the depth image, making it difficult to estimate the rotation and translation of the endoscope.

[0218] Therefore, RGB feature extraction is integrated into the relative pose calculation for real-time localization. Specifically, features such as cavity texture extracted from RGB images can be used to extract features from two frames of endoscopic images using feature descriptors (such as SIFT, ORB) or pre-trained feature extraction networks. These features, along with the depth image, are then used as input to a depth registration network. This can compensate for the difficulty in estimating the endoscope pose when the depth map structure is simple, and assist in estimating the motion of the real endoscope. In this case, it is necessary to acquire virtual endoscope images, depth images, and the corresponding virtual endoscope poses to train the depth extraction network.

[0219] During data acquisition, texture mapping is required for virtual endoscope images. This texture mapping needs to closely approximate the texture of images acquired by a real endoscope.

[0220] The endoscopic positioning method provided by this invention, by knowing the initial pose of the actual endoscope, can quickly and continuously obtain the current pose information of the actual endoscope using a pre-trained depth extraction network and a depth registration network. The depth extraction network and depth registration network in this method can be directly used for different patients after training, without the need for preoperative training, which is convenient and time-saving.

[0221] Figure 10 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 10As shown, the electronic device may include a processor 1010, a communication interface 1020, a memory 1030, and a communication bus 1040. The processor 1010, communication interface 1020, and memory 1030 communicate with each other via the communication bus 1040. The processor 1010 can call logical instructions in the memory 1030 to execute an endoscope positioning method. This method includes: acquiring the depth image of the current frame (i.e., the t-th frame) of a real endoscope image based on a pre-trained depth extraction network. Obtain the depth image d of the target virtual image at frame tn, acquired by the virtual endoscope at the pose of frame tn within the target virtual model. t-n Alternatively, the depth image of the nth frame of the real endoscope can be obtained based on the pre-trained depth extraction network. The virtual endoscope is determined based on the real endoscope; the depth image is used to... and the depth image d t-n Or the depth image and the depth image Inputting the pre-trained deep registration network, the relative pose estimation information of the real endoscope when acquiring the t-th frame image and the tn-th frame image is obtained. The relative pose estimation information Pose estimation information compared with the nth frame image acquired by the real endoscope By superimposing the images, the pose estimation information of the t-th frame image acquired by the real endoscope is obtained. And based on the pose estimation information The real endoscope is positioned, wherein the pose information of the initial position of the real endoscope is... It is known.

[0222] Furthermore, the logical instructions in the aforementioned memory 1030 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0223] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the endoscope localization method provided by the above methods. The method includes: acquiring the depth image of the current frame, i.e., the t-th frame image, acquired by a real endoscope based on a pre-trained depth extraction network. Obtain the depth image d of the target virtual image at frame tn, acquired by the virtual endoscope at the pose of frame tn within the target virtual model. t-n Alternatively, the depth image of the nth frame of the real endoscope can be obtained based on the pre-trained depth extraction network. The virtual endoscope is determined based on the real endoscope; the depth image is used to... and the depth image d t-n Or the depth image and the depth image Inputting the pre-trained deep registration network, the relative pose estimation information of the real endoscope when acquiring the t-th frame image and the tn-th frame image is obtained. The relative pose estimation information Pose estimation information compared with the nth frame image acquired by the real endoscope By superimposing the images, the pose estimation information of the t-th frame image acquired by the real endoscope is obtained. And based on the pose estimation information The real endoscope is positioned, wherein the pose information of the initial position of the real endoscope is... It is known.

[0224] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the endoscope localization method provided by the methods described above, the method comprising: acquiring a depth image of the current frame, i.e., the t-th frame image, acquired by a real endoscope based on a pre-trained depth extraction network. Obtain the depth image d of the target virtual image at frame tn, acquired by the virtual endoscope at the pose of frame tn within the target virtual model. t-n Alternatively, the depth image of the nth frame of the real endoscope can be obtained based on the pre-trained depth extraction network. The virtual endoscope is determined based on the real endoscope; the depth image is used to... and the depth image d t-n Or the depth image and the depth image Inputting the pre-trained deep registration network, the relative pose estimation information of the real endoscope when acquiring the t-th frame image and the tn-th frame image is obtained. The relative pose estimation information Pose estimation information compared with the nth frame image acquired by the real endoscope By superimposing the images, the pose estimation information of the t-th frame image acquired by the real endoscope is obtained. And based on the pose estimation information The real endoscope is positioned, wherein the pose information of the initial position of the real endoscope is... It is known.

[0225] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0226] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0227] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An endoscope positioning method, characterized in that, include: Acquiring the first data captured by a real endoscope using a pre-trained deep extraction network. Depth image of a frame ; Acquiring a virtual endoscope in the target virtual model The first frame acquired at the pose localization position Depth image of the frame target virtual image Or, based on the pre-trained depth extraction network, the first image captured by the real endoscope is obtained. Depth image of a frame The virtual endoscope is determined based on the real endoscope. The depth image and the depth image Or the depth image and the depth image Inputting the pre-trained deep registration network, the first image captured by the real endoscope is obtained. Frame Image and Acquisition Relative pose estimation information for frame images ; The relative pose estimation information The first image acquired by the real endoscope Pose estimation information for frame images Superimposed to obtain the first image captured by the real endoscope. Pose estimation information of frame images And based on the pose estimation information Position the actual endoscope; The depth extraction network is a depth extraction network based on a recurrent generative adversarial network and a pre-trained depth registration network. The recurrent generative adversarial network includes a first generator, a first discriminator, a second generator, and a second discriminator. The first generator is used to convert the depth image into a realistic endoscopic image, and the second generator is used to convert the realistic endoscopic image into a depth image. The deep extraction network based on the recurrent generative adversarial network and the deep registration network is trained in the following manner: A virtual model is established, and the depth image of the virtual image captured by the virtual endoscope in the virtual model is obtained, and the virtual pose information of the virtual endoscope corresponding to the virtual image is obtained when the virtual image is captured; Acquire preset real endoscopic images; The preset real endoscope image, the depth image of the virtual image, and the virtual pose information are used as training data to perform weakly supervised training on the initial depth extraction network. The loss function is obtained by weighted summation of the cycle consistency loss, identity loss, generative adversarial loss, reconstruction loss, and geometric consistency loss constrained by the initial deep extraction network; The loss function is optimized, and the parameters of the initial depth extraction network based on the recurrent generative adversarial network and the depth registration network are updated until a preset number of rounds are reached to obtain the depth extraction network based on the recurrent generative adversarial network and the depth registration network. The deep registration network was trained in the following manner: A virtual model is established, the depth image of the virtual image acquired by the virtual endoscope in the virtual model is obtained, and the virtual pose information corresponding to the virtual image acquired by the virtual endoscope is obtained. The depth image of the virtual image is input into the initial depth registration network, and the initial depth registration network outputs the relative pose estimation information of the virtual endoscope when acquiring two adjacent frames of virtual images; The virtual pose information is used as the training ground value, and the virtual relative pose information when the virtual endoscope acquires the virtual images of two adjacent frames is obtained based on the virtual pose information. The loss function is obtained by weighted summing of translation and rotation losses between the relative pose estimation information and the virtual relative pose information; The loss function is optimized, and the parameters of the initial depth registration network are updated until convergence is achieved to obtain the depth registration network. The registration method based on the iterative optimization algorithm is used in parallel with the deep registration network. The pose estimation information of the real endoscope is corrected based on the corrected pose obtained by the registration method based on the iterative optimization algorithm, thereby eliminating the cumulative error.

2. The endoscope positioning method according to claim 1, characterized in that, The deep extraction network is either a deep extraction network based on SfMLearner or a deep extraction network based on a recurrent generative adversarial network. In the depth image and the depth image Or the depth image and the depth image Before inputting the pre-trained deep registration network, the method further includes: For the depth image and the depth image Scale calibration is performed to obtain the depth image. and the depth image The unit.

3. The endoscope positioning method according to claim 1, characterized in that, Methods for obtaining corrected poses based on registration methods using iterative optimization algorithms include: The first time to obtain real endoscopic images The frame image is used as the current corrected image, and the first frame is obtained through the depth extraction network. Depth image of a frame ,in ; Obtain the real endoscopic images acquired based on the depth registration network. Pose estimation information of frame images ; Using the current corrected image or the depth image or the current corrected image and the depth image Semantic segmentation is performed on the cavity images in the actual endoscopic field of view; Based on image similarity measures and semantic segmentation similarity measures, pose estimation information is used. The initial values ​​are optimized to obtain the corrected pose of the current corrected image. ; The real endoscope was used to collect the first... Pose estimation information for frame images Replace with the corrected pose .

4. The endoscope positioning method according to claim 1, characterized in that, Methods for obtaining corrected poses based on registration methods using iterative optimization algorithms include: The first time to obtain real endoscopic images The frame image is used as the current corrected image, and the first frame is obtained through the depth extraction network. Depth image of a frame ,in ; Obtain the virtual endoscope in the target virtual model. The first frame acquired at the pose localization position Depth image of the frame target virtual image ; The depth image Convert to the corresponding point cloud The depth image Convert to point cloud image ; Solving using the ICP algorithm arrive Relative pose between ; Using the relative pose Correcting the real endoscope acquisition of the first Pose estimation information for frame images .

5. The endoscope positioning method according to any one of claims 1 to 2, characterized in that, Also includes: The first image was extracted using the RGB image feature extraction method from real endoscope images. The feature information of the frame image will be used to... Feature information of the frame image and the depth image The pre-trained deep registration network is then input together; The first image was extracted using the RGB image feature extraction method from real endoscope images. Feature information of frame images or extraction of the first frame acquired by virtual endoscopy Feature information of the target virtual image in the frame, wherein the first The feature information of the target virtual image in the frame is based on the first... The texture is extracted from the virtual image of the target frame after texture mapping. The first Feature information of the frame target virtual image and the depth image , or the first Feature information of the frame image and the depth image Input the pre-trained deep registration network.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the endoscope positioning method as described in any one of claims 1 to 5.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the endoscope positioning method as described in any one of claims 1 to 5.

Images