Continuous frame endoscope positioning method and device

Abstract

The continuous frame endoscope positioning method and device can effectively estimate accurate positioning information from the original endoscope image, effectively eliminate the cumulative error of multi-frame endoscope motion estimation, provide a new idea of ​​feature extraction for applications such as endoscope image positioning, lesion location detection, and three-dimensional reconstruction, and solve the current problem of cumulative error in endoscope continuous frame positioning. The method includes: (1) data acquisition; (2) multi-channel visual feature aggregation and image depth information network estimation; (3) adaptive constraint repositioning optimization: a method based on adaptive constraint repositioning is used to optimize the position information of the continuous frame endoscope. In the process of removing the cumulative error of multi-frame endoscope image positioning, the minimum inscribed circle radius of the cavity surface is measured as the continuous frame adaptive error threshold. When the error is greater than the threshold, the frame is repositioned as the initial frame to obtain the optimized result of endoscope positioning; (4) endoscope positioning information is obtained.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision, and in particular to a continuous-frame endoscope positioning method and a continuous-frame endoscope positioning device. Background Art

[0002] In recent years, as computer vision research has extended to assisting clinical medicine, endoscopic positioning has found wide applications in lesion location surveying, endoscopic perception feedback, and three-dimensional reconstruction. It can assist clinical surgeons in precisely controlling the motion process of endoscopic surgery, thereby ensuring the success of the operation. It has also become an effective way to improve the clinical use value of computer vision. Therefore, endoscopic positioning technology has become a key technology for obtaining real-time motion and surface morphology during endoscopic surgery. However, due to the complex operating environment during clinical surgery, the accumulation of endoscopic positioning errors in consecutive frames, and the uneven movement speed during the operation, the errors in the obtained endoscopic positioning results are relatively large.

[0003] During minimally invasive surgery, the surgeon's understanding of the surgical instrument's direction and the distance to the lesion is based on repeated comparisons of the endoscope's movements with the on-screen display during actual surgery. Currently, common endoscopic positioning methods, such as magnetic and infrared optical positioning systems, rely on additional hardware. However, the ferromagnetic surgical instruments used during surgery and the surgeon's manipulation can interfere with positioning results, hindering accurate positioning. Currently, widely used endoscopic positioning methods suffer from significant errors, and the accumulated errors can easily deviate from the localized scope of the cavity. These error factors can severely impact subsequent surgical assistance applications. Summary of the Invention

[0004] In order to overcome the defects of the existing technology, the technical problem to be solved by the present invention is to provide a continuous frame endoscope positioning method, which can effectively estimate precise positioning information from the original endoscope image, effectively eliminate the cumulative error of multi-frame endoscope motion estimation, and provide a new idea for feature extraction for applications such as endoscopic image positioning, lesion position detection, and three-dimensional reconstruction, solving the problem of cumulative error in current endoscope continuous frame positioning.

[0005] The technical solution of the present invention is: this continuous frame endoscope positioning method comprises the following steps:

[0006] (1) Data acquisition: When three samples are input, the 2-norm distance between the positive and negative sample reference images is:

[0007]

[0008] Among them, Net(·) is a three-branch fusion residual network, P1, P2, P3 are three sample inputs;

[0009] (2) Multi-channel visual feature aggregation and image depth information network estimation: In order to output multi-branch aggregated features from the model, the multi-level fusion feature blocks are passed through a series of fully connected layers to generate two outputs, representing the regression offsets of the positive sample and the non-sample to the reference image respectively. The two intermediate values ​​use the softmax function to create a ratio metric, and the feature vector loss function L fv Expressed as:

[0010]

[0011]

[0012]

[0013] in, Represent the feature vectors of positive samples and negative samples to reference samples respectively, γ is a hyperparameter used to adjust the distance between feature vectors. The distance optimization takes the minimum value, The maximum value is taken for distance optimization. During the retraining process, the average value of the distance between the feature vector of each frame of the endoscopic image in the current segment is estimated in segments, and the minimum value is taken as the location of the endoscope area.

[0014] The main path of the network is composed of two independent residual result networks with the same structure, each consisting of 4 consecutive convolutional layers, each convolutional layer consists of 256 3×3 filters, and finally outputs two sets of 196×1×1 vectors to the fully connected layer. During training, the original image is compared with the corresponding depth image. As a pair of samples, it is input into two independent networks, and the resulting vector is passed through a fully connected layer, which produces an estimate of the location of the original image in the depth image; finally, the localization result consists of 6 components, including 3 position offsets [p x ,p y ,p z ], and 3 rotation offsets [α, β, γ]. The network is trained using Adam optimization. By calculating the first-order moment estimate and the second-order moment estimate of the gradient, independent adaptive learning rates are designed for different parameters. Then, the network is constructed to minimize the estimated offset and depth offset to estimate the precise position information of the single-frame endoscope.

[0015] (3) Adaptive constraint repositioning optimization: A method based on adaptive constraint repositioning is used to optimize the position information of the continuous frame endoscope. In the process of removing the cumulative error of multi-frame endoscopic image positioning, the minimum inscribed circle radius of the cavity surface is measured as the adaptive error threshold of the continuous frame. When the error is greater than the threshold, the frame is used as the initial frame for repositioning to obtain the optimized result of endoscope positioning;

[0016] (4) Obtain endoscope positioning information.

[0017] Based on endoscopic images acquired during clinical surgery, this paper defines an endoscopic positioning method based on the motion of consecutive frames of endoscopic images. By jointly estimating the planar information of the endoscopic image and the three-dimensional information of the surgical space, precise positioning information can be effectively estimated from the original endoscopic image. Using an adaptive constrained repositioning optimization method, the cumulative error in multi-frame endoscopic motion estimation can be effectively eliminated. This paper provides a new feature extraction method for applications such as endoscopic image positioning, lesion location detection, and three-dimensional reconstruction. It solves the current problem of cumulative error in endoscopic consecutive frame positioning.

[0018] Also provided is a continuous frame endoscope positioning device comprising:

[0019] The data acquisition module is configured to take three samples as input, and the 2-norm distance between the positive and negative sample reference images is:

[0020]

[0021] Among them, Net(·) is a three-branch fusion residual network, P1, P2, and P3 are three sample inputs; the multi-channel visual feature aggregation and image depth information network estimation module is configured to output multi-branch aggregation features from the model. The multi-level fusion feature block passes through a series of fully connected layers to generate two outputs, representing the regression offsets of the positive sample and the non-sample to the reference image. The two intermediate values ​​use the softmax function to create a ratio metric, and the feature vector loss function L fv Expressed as:

[0022]

[0023]

[0024]

[0025] in, Represent the feature vectors of positive samples and negative samples to reference samples respectively, γ is a hyperparameter used to adjust the distance between feature vectors. The distance optimization takes the minimum value, The maximum value is taken for distance optimization. During the retraining process, the average value of the distance between the feature vector of each frame of the endoscopic image in the current segment is estimated in segments, and the minimum value is taken as the location of the endoscope area.

[0026] The main path of the network is composed of two independent residual result networks with the same structure, each consisting of 4 consecutive convolutional layers, each convolutional layer consists of 256 3×3 filters, and finally outputs two sets of 196×1×1 vectors to the fully connected layer. During training, the original image is compared with the corresponding depth image. As a pair of samples, it is input into two independent networks, and the resulting vector is passed through a fully connected layer, which produces an estimate of the location of the original image in the depth image; finally, the localization result consists of 6 components, including 3 position offsets [p x ,p y ,p z ], and 3 rotation offsets [α, β, γ]. The network is trained using Adam optimization. By calculating the first-order moment estimate and the second-order moment estimate of the gradient, independent adaptive learning rates are designed for different parameters. Then, the network is constructed to minimize the estimated offset and depth offset to estimate the precise position information of the single-frame endoscope.

[0027] The adaptive constraint repositioning optimization module is configured to optimize the position information of the continuous frame endoscope using an adaptive constraint repositioning method. In the process of removing the cumulative error of multi-frame endoscope image positioning, the minimum inscribed circle radius of the lumen surface is measured as the adaptive error threshold of the continuous frame. When the error is greater than the threshold, the frame is used as the initial frame for repositioning to obtain the optimized result of endoscope positioning;

[0028] An information acquisition module is configured to obtain endoscope positioning information. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 1 is a flow chart of a continuous frame endoscope positioning method according to the present invention.

[0030] Figure 2 It is a flowchart of a specific embodiment of the continuous frame endoscope positioning method according to the present invention.

[0031] Figure 3 1 is a flow chart of step (2) of the continuous frame endoscope positioning method according to the present invention.

[0032] Figure 4 3 is a flow chart of step (3) of the continuous frame endoscope positioning method according to the present invention. DETAILED DESCRIPTION

[0033] like Figure 1 As shown, this continuous frame endoscope positioning method includes the following steps:

[0034] (1) Data acquisition: When three samples are input, the 2-norm distance between the positive and negative sample reference images is:

[0035]

[0036] Among them, Net(·) is a three-branch fusion residual network, P1, P2, P3 are three sample inputs;

[0037] (2) Multi-channel visual feature aggregation and image depth information network estimation: In order to output multi-branch aggregated features from the model, the multi-level fusion feature blocks are passed through a series of fully connected layers to generate two outputs, representing the regression offsets of the positive sample and the non-sample to the reference image respectively. The two intermediate values ​​use the softmax function to create a ratio metric, and the feature vector loss function L fv Expressed as:

[0038]

[0039]

[0040]

[0041] in, Represents the characteristic orientation of positive samples and negative samples to reference samples.

[0042] , γ is a hyperparameter used to adjust the distance between feature vectors. The distance optimization takes the minimum value, The maximum value is taken for distance optimization. During the retraining process, the average value of the distance between the feature vector of each frame of the endoscopic image in the current segment is estimated in segments, and the minimum value is taken as the location of the endoscope area.

[0043] The main path of the network is composed of two independent residual result networks with the same structure, each consisting of 4 consecutive convolutional layers, each convolutional layer consists of 256 3×3 filters, and finally outputs two sets of 196×1×1 vectors to the fully connected layer. During training, the original image is compared with the corresponding depth image. As a pair of samples, it is input into two independent networks, and the resulting vector is passed through a fully connected layer, which produces an estimate of the location of the original image in the depth image; finally, the localization result consists of 6 components, including 3 position offsets [p x ,p y ,p z ], and 3 rotation offsets [α, β, γ]. The network is trained using Adam optimization. By calculating the first-order moment estimate and the second-order moment estimate of the gradient, independent adaptive learning rates are designed for different parameters. Then, the network is constructed to minimize the estimated offset and depth offset to estimate the precise position information of the single-frame endoscope.

[0044] (3) Adaptive constraint repositioning optimization: A method based on adaptive constraint repositioning is used to optimize the position information of the continuous frame endoscope. In the process of removing the cumulative error of multi-frame endoscopic image positioning, the minimum inscribed circle radius of the cavity surface is measured as the adaptive error threshold of the continuous frame. When the error is greater than the threshold, the frame is used as the initial frame for repositioning to obtain the optimized result of endoscope positioning;

[0045] (4) Obtain endoscope positioning information.

[0046] Based on endoscopic images acquired during clinical surgery, this paper defines an endoscopic positioning method based on the motion of consecutive frames of endoscopic images. By jointly estimating the planar information of the endoscopic image and the three-dimensional information of the surgical space, precise positioning information can be effectively estimated from the original endoscopic image. Using an adaptive constrained repositioning optimization method, the cumulative error in multi-frame endoscopic motion estimation can be effectively eliminated. This paper provides a new feature extraction method for applications such as endoscopic image positioning, lesion location detection, and three-dimensional reconstruction. It solves the current problem of cumulative error in endoscopic consecutive frame positioning.

[0047] Preferably, in the step (1), the residual network results are used in the three input branches to generate the feature vector of the input image, and the input image size is a three-channel color image of 512*512; the image has a similar topological structure to the image during virtual endoscopy, and the visual feature information of the image is introduced into the structural characteristics of the network coding as prior knowledge to improve the weight of the feature vector in the structural information encoding process; a decoding network is connected after the convolution layer, which consists of 5 convolution layers and 4 upsampling layers; the branch outputs of the 512*512 structural information graph are connected in series and fed to the linear fully connected layer to generate a 128-dimensional feature descriptor vector.

[0048] Preferably, in the step (2), in the depth information positioning task, the position of each frame depth estimation result mapped in the segmentation result is regarded as a separate endoscope positioning coordinate, and the loss function L used by the depth information optimization network is im Expressed as:

[0049]

[0050] Where Δd t Indicates the actual displacement between frames, Denotes the estimated displacement between frames, Δd r Indicates the actual rotation angle between frames, Represents the estimated rotation angle of the inter-frame rotation. In order to link the position and rotation errors, δ is used to represent the weight of adjusting the displacement and rotation.

[0051] Those skilled in the art will appreciate that all or part of the steps in the above-described embodiment method can be accomplished by instructing the relevant hardware through a program. The program can be stored in a computer-readable storage medium. When executed, the program includes the steps of the above-described embodiment method. The storage medium can be ROM / RAM, a magnetic disk, an optical disk, a memory card, etc. Therefore, corresponding to the method of the present invention, the present invention also includes a continuous frame endoscope positioning device, which is generally represented in the form of functional modules corresponding to the steps of the method. The device includes:

[0052] The data acquisition module is configured to take three samples as input, and the 2-norm distance between the positive and negative sample reference images is:

[0053]

[0054] Among them, Net(·) is a three-branch fusion residual network, P1, P2, and P3 are three sample inputs; the multi-channel visual feature aggregation and image depth information network estimation module is configured to output multi-branch aggregation features from the model. The multi-level fusion feature block passes through a series of fully connected layers to generate two outputs, representing the regression offsets of the positive sample and the non-sample to the reference image. The two intermediate values ​​use the softmax function to create a ratio metric, and the feature vector loss function L fv Expressed as:

[0055]

[0056]

[0057]

[0058] in, Represent the feature vectors of positive samples and negative samples to reference samples respectively, γ is a hyperparameter used to adjust the distance between feature vectors. The distance optimization takes the minimum value, The maximum value is taken for distance optimization. During the retraining process, the average value of the distance between the feature vector of each frame of the endoscopic image in the current segment is estimated in segments, and the minimum value is taken as the location of the endoscope area.

[0059] The main path of the network is composed of two independent residual result networks with the same structure, each consisting of 4 consecutive convolutional layers, each convolutional layer consists of 256 3×3 filters, and finally outputs two sets of 196×1×1 vectors to the fully connected layer. During training, the original image is compared with the corresponding depth image. As a pair of samples, it is input into two independent networks, and the resulting vector is passed through a fully connected layer, which produces an estimate of the location of the original image in the depth image; finally, the localization result consists of 6 components, including 3 position offsets [p x ,p y ,p z ], and 3 rotation offsets [α, β, γ]. The network is trained using Adam optimization. By calculating the first-order moment estimate and the second-order moment estimate of the gradient, independent adaptive learning rates are designed for different parameters. Then, the network is constructed to minimize the estimated offset and depth offset to estimate the precise position information of the single-frame endoscope.

[0060] The adaptive constraint repositioning optimization module is configured to optimize the position information of the continuous frame endoscope using an adaptive constraint repositioning method. In the process of removing the cumulative error of multi-frame endoscope image positioning, the minimum inscribed circle radius of the lumen surface is measured as the adaptive error threshold of the continuous frame. When the error is greater than the threshold, the frame is used as the initial frame for repositioning to obtain the optimized result of endoscope positioning;

[0061] An information acquisition module is configured to obtain endoscope positioning information.

[0062] Preferably, in the data acquisition module, the residual network results are used in the three input branches to generate the feature vector of the input image, and the input image size is a three-channel color image of 512*512; the image has a similar topological structure to the image during virtual endoscopy, and the visual feature information of the image is introduced into the structural characteristics of the network coding as prior knowledge to improve the weight of the feature vector in the structural information encoding process; a decoding network is connected after the convolution layer, which consists of 5 convolution layers and 4 upsampling layers; the branch outputs of the 512*512 structural information graph are connected in series and fed to the linear fully connected layer to generate a 128-dimensional feature descriptor vector.

[0063] Preferably, in the multi-channel visual feature aggregation and image depth information network estimation module, in the depth information positioning task, the position of each frame depth estimation result mapped in the segmentation result is regarded as a separate endoscope positioning coordinate, and the loss function L used by the depth information optimization network is im Expressed as:

[0064]

[0065] Where Δd t Indicates the actual displacement between frames, Denotes the estimated displacement between frames, Δd r Indicates the actual rotation angle between frames, Represents the estimated rotation angle of the inter-frame rotation. In order to link the position and rotation errors, δ is used to represent the weight of adjusting the displacement and rotation.

[0066] Compared with the existing technology, the present invention has the following beneficial effects: Based on endoscopic images obtained during clinical surgery, the present invention defines an endoscopic positioning method according to the motion of continuous frame endoscopic images. By jointly estimating the planar information of the endoscopic image and the three-dimensional information of the surgical space, precise positioning information can be effectively estimated from the original endoscopic image. The adaptive constrained repositioning optimization method can effectively eliminate the cumulative error of multi-frame endoscopic motion estimation. The present invention provides a new feature extraction method for applications such as endoscopic image positioning, lesion location detection, and three-dimensional reconstruction. It solves the problem of cumulative error in the current endoscopic continuous frame positioning.

[0067] The above description is merely a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention are still within the scope of protection of the technical solution of the present invention.

Claims

1. A continuous frame endoscope positioning method, characterized in that: It includes the following steps: (1) Data acquisition: When three samples are input, the 2-norm distance between the positive and negative sample reference images is: Among them, Net(·) is a three-branch fusion residual network, P1, P2, P3 are three sample inputs; (2) Multi-channel visual feature aggregation and image depth information network estimation: In order to output multi-branch aggregated features from the model, the multi-level fusion feature blocks are passed through a series of fully connected layers to generate two outputs, representing the regression offsets of the positive sample and the non-sample to the reference image respectively. The two intermediate values ​​use the softmax function to create a ratio metric, and the feature vector loss function L fv Expressed as: in, Represent the feature vectors of positive samples and negative samples to reference samples respectively, γ is a hyperparameter used to adjust the distance between feature vectors. The distance optimization takes the minimum value, The maximum value is taken for distance optimization. During the retraining process, the average value of the distance between the feature vector of each frame of the endoscopic image in the current segment is estimated in segments, and the minimum value is taken as the location of the endoscope area. The main path of the network is composed of two independent residual result networks with the same structure, each consisting of 4 consecutive convolutional layers, each convolutional layer consists of 256 3×3 filters, and finally outputs two sets of 196×1×1 vectors to the fully connected layer. During training, the original image is compared with the corresponding depth image. As a pair of samples, it is input into two independent networks, and the resulting vector is passed through a fully connected layer, which produces an estimate of the location of the original image in the depth image; finally, the localization result consists of 6 components, including 3 position offsets [p x ,p y ,p z ], and 3 rotation offsets [α, β, γ]. The network is trained using Adam optimization. By calculating the first-order moment estimate and the second-order moment estimate of the gradient, independent adaptive learning rates are designed for different parameters. Then, the network is constructed to minimize the estimated offset and depth offset to estimate the precise position information of the single-frame endoscope. (3) Adaptive constraint repositioning optimization: A method based on adaptive constraint repositioning is used to optimize the position information of the continuous frame endoscope. In the process of removing the cumulative error of multi-frame endoscopic image positioning, the minimum inscribed circle radius of the cavity surface is measured as the adaptive error threshold of the continuous frame. When the error is greater than the threshold, the frame is used as the initial frame for repositioning to obtain the optimized result of endoscope positioning; (4) Obtain endoscope positioning information.

2. The continuous frame endoscope positioning method according to claim 1, characterized in that: In the step (1), the residual network results are used in three input branches to generate the feature vector of the input image, and the input image size is a three-channel color image of 512*512; the image has a similar topological structure to the image during virtual endoscopy, and the visual feature information of the image is introduced into the structural characteristics of the network coding as prior knowledge to improve the weight of the feature vector in the structural information encoding process; a decoding network is connected after the convolution layer, which consists of 5 convolution layers and 4 upsampling layers; the branch outputs of the 512*512 structural information graph are connected in series and fed into a linear fully connected layer to generate a 128-dimensional feature descriptor vector.

3. The continuous frame endoscope positioning method according to claim 2, characterized in that: In the step (2), in the depth information positioning task, the position of each frame depth estimation result mapped in the segmentation result is regarded as a separate endoscope positioning coordinate, and the loss function L used by the depth information optimization network is im Expressed as: Where Δd t Indicates the actual displacement between frames, Denotes the estimated displacement between frames, Δd r Indicates the actual rotation angle between frames, Represents the estimated rotation angle of the inter-frame rotation. In order to link the position and rotation errors, δ is used to represent the weight of adjusting the displacement and rotation.

4. Continuous frame endoscope positioning device, characterized in that: It includes: The data acquisition module is configured to take three samples as input, and the 2-norm distance between the positive and negative sample reference images is: Among them, Net(·) is a three-branch fusion residual network, P1, P2, and P3 are three sample inputs; the multi-channel visual feature aggregation and image depth information network estimation module is configured to output multi-branch aggregation features from the model. The multi-level fusion feature block passes through a series of fully connected layers to generate two outputs, representing the regression offsets of the positive sample and the non-sample to the reference image. The two intermediate values ​​use the softmax function to create a ratio metric, and the feature vector loss function L fv Expressed as: in, Represent the feature vectors of positive samples and negative samples to reference samples respectively, γ is a hyperparameter used to adjust the distance between feature vectors. The distance optimization takes the minimum value, The maximum value is taken for distance optimization. During the retraining process, the average value of the distance between the feature vector of each frame of the endoscopic image in the current segment is estimated in segments, and the minimum value is taken as the location of the endoscope area. The main path of the network is composed of two independent residual result networks with the same structure, each consisting of 4 consecutive convolutional layers, each convolutional layer consists of 256 3×3 filters, and finally outputs two sets of 196×1×1 vectors to the fully connected layer. During training, the original image is compared with the corresponding depth image. As a pair of samples, it is input into two independent networks, and the resulting vector is passed through a fully connected layer, which produces an estimate of the location of the original image in the depth image; finally, the localization result consists of 6 components, including 3 position offsets [p x ,p y ,p z ], and 3 rotation offsets [α, β, γ]. The network is trained using Adam optimization. By calculating the first-order moment estimate and the second-order moment estimate of the gradient, independent adaptive learning rates are designed for different parameters. Then, the network is constructed to minimize the estimated offset and depth offset to estimate the precise position information of the single-frame endoscope. The adaptive constraint repositioning optimization module is configured to optimize the position information of the continuous frame endoscope using an adaptive constraint repositioning method. In the process of removing the cumulative error of multi-frame endoscope image positioning, the minimum inscribed circle radius of the lumen surface is measured as the adaptive error threshold of the continuous frame. When the error is greater than the threshold, the frame is used as the initial frame for repositioning to obtain the optimized result of endoscope positioning; An information acquisition module is configured to obtain endoscope positioning information.

5. The continuous frame endoscope positioning device according to claim 4, characterized in that: In the data acquisition module, the residual network results are used in three input branches to generate feature vectors of the input image, and the input image size is a three-channel color image of 512*512. The image has a similar topological structure to the image during virtual endoscopy, and the visual feature information of the image is introduced into the structural characteristics of the network coding as prior knowledge to improve the weight of the feature vector in the structural information encoding process. A decoding network is connected after the convolution layer, which consists of 5 convolution layers and 4 upsampling layers. The branch outputs of the 512*512 structural information map are connected in series and fed into a linear fully connected layer to generate a 128-dimensional feature descriptor vector.

6. The continuous frame endoscope positioning device according to claim 5, characterized in that: In the multi-channel visual feature aggregation and image depth information network estimation module, in the depth information positioning task, the position of each frame depth estimation result mapped in the segmentation result is regarded as a separate endoscope positioning coordinate, and the loss function L used by the depth information optimization network is im Expressed as: Where Δd t Indicates the actual displacement between frames, Denotes the estimated displacement between frames, Δd r Indicates the actual rotation angle between frames, Represents the estimated rotation angle of the inter-frame rotation. In order to link the position and rotation errors, δ is used to represent the weight of adjusting the displacement and rotation.

Images