Endoscope video positioning method based on reflection removal

Abstract

The invention discloses an endoscope video positioning method based on reflection removal. The method comprises the following steps: S1, obtaining a monocular video of a moving endoscope in an intestinal tract; s2, performing fuzzy detection on the single-frame picture, screening key frames and fuzzy pictures, and classifying the pictures into key frames and non-key frames; s3, segmenting the video frame, extracting a reflective area, creating a mask, expanding by using an elliptic structure operator, deblurring non-key frames, filling the reflective area by using STTN according to information of adjacent time frames and adjacent areas, generating textures of the corresponding areas, and repairing reflective blocks in the monocular video; compared with a non-pure vision three-dimensional mapping scheme, the method does not need to obtain postures or depths as input through complex sensing equipment such as a laser radar or a depth or binocular camera, the cost of video acquisition equipment is lower, the method can adapt to violent changes of illumination, and the method is suitable for large-scale popularization and application. And meanwhile, the required calculation amount is reduced.

Description

Technical Field

[0001] The present invention relates to the field of medical technology, and in particular to an endoscope video positioning method based on reflection removal. Background Art

[0002] The intestine is an important organ for immunity and detoxification. Unhealthy eating habits lead to an increasing incidence of intestinal diseases year by year. Since the intestinal endoscopy diagnosis and treatment system has high requirements for operators, doctors need to undergo a lot of training to adapt to the hand-eye coordination of surgical instruments. Due to the long learning curve and large individual differences, patients often have to endure great pain due to contact and collision with the intestinal wall during the diagnosis and treatment process. When faced with a narrow and tortuous intestinal environment, the doctor's operation of the endoscope, the location of the lesion, and the accurate judgment of the lesion size are highly dependent on the doctor's experience.

[0003] In order to reduce the collision of the endoscope system with the intestine and reduce the pain during the diagnosis process, it is necessary to plan a reasonable endoscope movement plan according to the internal structure and size of the intestine. Therefore, the SLAM method is often used to build an internal map of the intestine and estimate the position and posture of the endoscope. In the complex intestinal structure, SLAM technology can prevent the endoscope from getting lost and improve the safety and efficiency of the operation. At the same time, it can also reduce the repeated movement and unnecessary contact of the endoscope in the intestine, reducing the patient's discomfort and potential tissue damage.

[0004] There is a lack of natural light sources in endoscopy scenes, and due to the particularity of the intestinal cavity, the image sequences taken by the endoscope contain a large number of reflective areas, making it difficult to perform correct depth estimation. Similarly, the reflective area also affects the extraction and matching of feature points. Removing reflections can reduce erroneous feature matching and improve camera tracking stability.

[0005] For example, the existing Chinese patent technology CN118172266A places a cosine intensity mask on the surface of the imaging sensor for the endoscope, performs imaging according to the conventional process, uses the Fourier domain to realize the splicing and reconstruction of the light field information, uses a low-rank matrix approximation algorithm to remove the highlight information in the light field image components at different angles, and uses a neural network algorithm to fuse images at different angles, and outputs the image after removing the highlight, thereby improving the image quality of the endoscope; CN115830122A uses a method based on adaptive constraint repositioning to optimize the position information of the continuous frame endoscope. In the process of removing the accumulated error of multi-frame endoscope image positioning, the minimum inscribed circle radius of the cavity surface is measured as the adaptive error threshold of the continuous frame. When the error is greater than the threshold, the frame is repositioned as the initial frame to obtain the optimization result of the endoscope positioning. It can effectively eliminate the cumulative error of multi-frame endoscopic motion estimation; CN115919461A, acquires images through a binocular endoscope and uses binocular SLAM for mapping; for the initial key frame image, binocular matching is performed first, and then it is aligned with the preoperative three-dimensional model through the landmark method to establish a world coordinate system; during the movement of the endoscope, after the key frame is binocularly matched, it is aligned with the three-dimensional model according to the posture information provided by SLAM, and the camera motion trajectory and the position of the current lens scene at the surgical site are displayed in real time, but landmarks need to be placed in advance. As time changes, the landmarks may move and cause additional errors; CN110033465A also uses a binocular endoscope, and according to the principle of epipolar constraints, it searches for the corresponding epipolar lines under the left and right perspectives in turn, and combines the ORB operator to calculate the three-dimensional data of the skeleton through the corresponding positions.

[0006] Traditional SLAM methods usually require the installation of specific sensors, such as lidar, inertial measurement unit, odometer, etc. It is difficult to install a large number of sensors in narrow endoscopic scenes. Most endoscopes have only one camera to collect images. Therefore, this application proposes an endoscopic video positioning method based on reflection removal. Summary of the invention

[0007] The purpose of the present invention is to provide an endoscopic video positioning method based on reflection removal in order to solve the above-mentioned problems, and a camera tracking method for a colorectal monocular endoscopic image sequence based on removal of reflective areas, so as to cope with the defects of low texture in endoscopic inspection and strong lighting changes that lead to easy failure of tracking camera movement, thereby solving the problems mentioned in the background technology.

[0008] In order to solve the above problems, the present invention provides a technical solution:

[0009] The endoscope video positioning method based on reflection removal includes the following steps:

[0010] S1, obtain monocular video of moving the endoscope in the intestine;

[0011] S2, performing blur detection on a single frame image, filtering key frames and blurred images, and classifying the images into key frames and non-key frames;

[0012] S3, segment the video frame, extract the reflective area, create a mask and use the ellipse structure operator to expand, deblur the non-key frames, fill the reflective area with STTN according to the information of adjacent time frames and neighboring areas, generate the texture of the corresponding area, and repair the reflective blocks in the monocular video;

[0013] S4. Extract and match feature points of adjacent frames to obtain matching point pairs, perform triangulation, and solve the camera transformation matrix according to the PnP algorithm.

[0014] Preferably, the specific steps of screening key frames in step S2 include:

[0015] A1. Resize the processed image and extract the green channel;

[0016] A2. Use the Laplacian operator to calculate the variance response value; determine the appropriate threshold from the experiment, traverse all frames, and if the response value is less than the threshold, the current frame is considered to be a blurred image;

[0017] A3. Calculate the MSSIM value for adjacent frames and determine the camera movement distance by comparing the image similarity. If it is greater than 0.85, the current frame is determined as a non-key frame with a small displacement. Otherwise, the current frame is retained as the filtered key frame.

[0018] Preferably, the method for creating and removing the reflective mask of the video frame in step S3 is:

[0019] B1. For non-key frame sequences with blurred images, the deblurring algorithm DeblurGAN is used to remove the blur caused by camera motion.

[0020] B2. Use threshold segmentation to process the monocular endoscope inspection video to obtain a rough reflective area mask;

[0021] B3, using the ellipse operator to perform morphological dilation on the mask to cover the reflective area and make the structure complete;

[0022] B4. Obtain the spatiotemporal information corresponding to the adjacent frames of the mask area neighborhood through the spatiotemporal Transformer, repair the mask area, and generate corresponding texture information.

[0023] Preferably, the method for obtaining the camera motion trajectory by extracting feature points in step S4 is:

[0024] Extract feature points from the processed image sequence, match the feature points, obtain matching point pairs in adjacent key frames, triangulate the feature points, and obtain the camera motion trajectory;

[0025] It also includes: using the feature point extraction network SuperPoint to extract key frame image feature points, obtain the matching relationship of feature points, triangulate matching points, optimize reprojection errors, obtain camera pose, and update camera trajectory.

[0026] Preferably, the step S4 further includes network training and prediction for feature point extraction, and the specific steps are:

[0027] Use a dataset composed of various natural images to train the feature point extraction model SuperPoint, initialize model parameters, and learn general visual feature extraction capabilities;

[0028] Use the self-collected endoscopic image sequence for prediction, load the pre-trained feature point extraction model parameters into the new model, use the endoscopic images for fine-tuning, optimize the model parameters, and complete the training of the feature point extraction model.

[0029] Preferably, the pose optimization constraint condition of the motion trajectory is determined by the following formula:

[0030]

[0031] Where θ=(C 1 ,…,C m ,X 1 ,…X n ) is the quantity to be optimized, namely the camera parameters and point cloud coordinate information, S ij Indicates the state of the i-th point under camera j; C i represents the external parameters of the i-th camera, X i represents the coordinates of the i-th point in three-dimensional space, and the reprojection error is e ij =p' ij -C i (R i P j +t i ) where p' ij is the observation value of the projection point, and minimizing the reprojection error is used as the constraint condition for pose optimization.

[0032] The beneficial effects of the present invention are:

[0033] 1. Compared with non-pure visual 3D mapping solutions, it does not require complex sensing devices such as lidar, depth or binocular cameras to obtain posture or depth as input, and the cost of video acquisition equipment is lower;

[0034] 2. By repairing the reflective image, the damage to the geometric structure consistency of the endoscope image caused by different lighting conditions can be reduced, the effective information in the image can be increased, and the influence of brightness on subsequent operations can be reduced. Compared with direct feature extraction and matching, feature point pairs with higher accuracy and better quality can be obtained;

[0035] 3. Filtering key frames by Laplacian operator and MSSIM value of adjacent images can reduce the amount of calculation. If the camera moves too small, the accuracy will decrease in the triangulation stage. If the moving distance is too large, the feature matching will fail. Remove images with too small camera movement amplitude to avoid camera posture tracking failure caused by too similar images.

[0036] 4. When using Transformer for image restoration, calculating attention on the entire feature map will result in a lot of unnecessary overhead. Focusing the attention mechanism on spatiotemporal blocks with smaller camera motion can significantly reduce the amount of computation in low-information areas outside the mask area.

[0037] 5. Compared with SFM, which mostly requires matching of disordered image geometry, the matching strategy is adjusted according to the characteristics of the endoscope data source: the data is a sequence image, and the adjacent images before and after are used as image pairs for feature extraction and matching. The changes between adjacent frames of monocular video are small, and the feature point matching is more stable and reliable. Global matching of disordered images requires O(n 2 ), directly taking adjacent key frames as image pairs reduces the amount of computation. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] For ease of explanation, the present invention is described in detail with reference to the following specific implementations and the accompanying drawings.

[0039] Figure 1 It is a schematic diagram of the workflow of the present invention;

[0040] Figure 2 is a processing flow chart of the reflective area network model of the present invention;

[0041] Figure 3 This is the present invention: image comparison diagram before and after reflection removal;

[0042] Figure 4 is a schematic diagram of a mask region for threshold extraction of the present invention;

[0043] Figure 5 It is a visualized heat map of attention extracted using STTN in the present invention;

[0044] Figure 6 It is a network structure diagram of SuperPoint feature point extraction of the present invention. DETAILED DESCRIPTION

[0045] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices consistent with some aspects of the present disclosure as detailed in the appended claims.

[0046] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] like Figure 1-6 As shown, the specific implementation adopts the following technical solutions:

[0048] Embodiment 1:

[0049] The endoscope video positioning method based on reflection removal includes the following steps:

[0050] S1, obtain monocular video of moving the endoscope in the intestine;

[0051] S2, performing blur detection on a single frame image, filtering key frames and blurred images, and classifying the images into key frames and non-key frames;

[0052] S3, segment the video frame, extract the reflective area, create a mask and use the ellipse structure operator to expand, deblur the non-key frames, fill the reflective area with STTN according to the information of adjacent time frames and neighboring areas, generate the texture of the corresponding area, and repair the reflective blocks in the monocular video;

[0053] S4. Extract and match feature points of adjacent frames to obtain matching point pairs, perform triangulation, and solve the camera transformation matrix according to the PnP algorithm.

[0054] The specific steps of key frame screening in step S2 include:

[0055] A1. Resize the processed image and extract the green channel;

[0056] A2. Use the Laplacian operator to calculate the variance response value; determine the appropriate threshold from the experiment, traverse all frames, and if the response value is less than the threshold, the current frame is considered to be a blurred image;

[0057] A3. Calculate the MSSIM value for adjacent frames and determine the camera movement distance by comparing the image similarity. If it is greater than 0.85, the current frame is determined as a non-key frame with a small displacement. Otherwise, the current frame is retained as the filtered key frame.

[0058] The method for creating and removing the reflective mask of the video frame in step S3 is:

[0059] B1. For non-key frame sequences with blurred images, the deblurring algorithm DeblurGAN is used to remove the blur caused by camera motion.

[0060] B2. Use threshold segmentation to process the monocular endoscope inspection video to obtain a rough reflective area mask;

[0061] B3, using the ellipse operator to perform morphological dilation on the mask to cover the reflective area and make the structure complete;

[0062] B4. Obtain the spatiotemporal information corresponding to the adjacent frames of the mask area neighborhood through the spatiotemporal Transformer, repair the mask area, and generate corresponding texture information.

[0063] The method for obtaining the camera motion trajectory by extracting feature points in step S4 is:

[0064] Extract feature points from the processed image sequence, match the feature points, obtain matching point pairs in adjacent key frames, triangulate the feature points, and obtain the camera motion trajectory;

[0065] It also includes: using the feature point extraction network SuperPoint to extract key frame image feature points, obtain the matching relationship of feature points, triangulate matching points, optimize reprojection errors, obtain camera pose, and update camera trajectory.

[0066] Wherein, step S4 also includes network training and prediction for feature point extraction, and the specific steps are as follows:

[0067] Use a dataset composed of various natural images to train the feature point extraction model SuperPoint, initialize model parameters, and learn general visual feature extraction capabilities;

[0068] Use the self-collected endoscopic image sequence for prediction, load the pre-trained feature point extraction model parameters into the new model, use the endoscopic images for fine-tuning, optimize the model parameters, and complete the training of the feature point extraction model.

[0069] Among them, the pose optimization constraint of the motion trajectory is determined by the following formula:

[0070]

[0071] Where θ=(C 1 ,…,C m ,X 1 ,…X n ) is the quantity to be optimized, namely the camera parameters and point cloud coordinate information, S ij Indicates the state of the i-th point under camera j; C i represents the external parameters of the i-th camera, X i represents the coordinates of the i-th point in three-dimensional space, and the reprojection error is e ij =p' ij -C i (R i P j +t i ) where p' ij is the observation value of the projection point, and minimizing the reprojection error is used as the constraint condition for pose optimization.

[0072] Embodiment 2:

[0073] Step 100: Obtaining a colorectal inspection monocular video from a common medical endoscope;

[0074] Step 200: Perform blur detection on the image to filter out images with lower quality and smaller camera movement;

[0075] Step 300: Segment the video frame, extract the reflective area, create a mask and use the ellipse structure operator to expand, use STTN to fill the reflective area according to the information of adjacent frames and neighboring areas, so as to generate the texture of the corresponding area and repair the reflective block in the monocular video;

[0076] Step 400: extract and match feature points of adjacent frames to obtain matching point pairs, perform triangulation, and solve the camera transformation matrix according to the PnP algorithm.

[0077] The implementation method is as follows:

[0078] A multi-layer and multi-head STTN is used to process video data, which includes a frame-level encoder and decoder, both of which are composed of 2D convolutional layers. This design is aimed at extracting features from video frames and performing encoding and decoding.

[0079] First, frames are extracted from the endoscope video at 5 frames per second, and the images are classified to screen out key frame sequences and non-key frame sequences. For non-key frame sequences, DeblurGAN is used to remove blur caused by motion. Then, the threshold segmentation technique is used to extract the specular reflection mask from the video frame using the brightness characteristics of the reflective area. After segmentation, the mask is expanded using an elliptical structural element so that the mask can cover the surrounding of the specular reflection. These masks are randomly translated in one direction within the same frame to cover the visible texture as pseudo ground truth. The mask area is filled according to the pixel information in the adjacent similar frames and the adjacent range, so that the texture behind is predicted through the temporal adversarial generative network, and finally the endoscopic image sequence with reflection removed is generated.

[0080] For the restored image sequence, the corresponding key frames are selected for key point extraction and matching, the two-dimensional feature points are promoted to three dimensions, the PnP algorithm is used to solve the camera transformation matrix, and the camera movement position is recorded.

[0081] The picture screening in step 200 includes the following operations:

[0082] 201: Convert the image after removing the reflection into the green channel;

[0083] 202: If the variance value of the current image is less than the threshold, the image is considered to be a blurred image, and is added to the non-key frame sequence and the key frame sequence.

[0084] 203: further filter the images with a small camera movement distance, and calculate the MSSIM value of adjacent images for the filtered key frame sequence. If it is greater than 0.85, remove the key frame sequence;

[0085] Codec network structure:

[0086]

[0087] like Figure 2 As shown in the figure, the image is input into three convolution layers with a convolution kernel size of 1x1 to reduce the number of channels, and the image is extracted into spatial blocks. The product of Query and Key is scaled and normalized through the Softmax layer. The attention weight matrix A is calculated, and then the attention weight matrix is ​​multiplied by the Value matrix. The product result is used as the feature map, and the feature map of the non-key frame is output through a convolution layer with a convolution kernel size of 3x3 and spliced ​​with the original image to complete the repair of the mask area in the original image sequence.

[0088] The multiple heads of the spatiotemporal transformer process complex motion patterns at different scales, encode-match-attention all images in the video frames, and the spatial block size of each layer is different. The smallest block size is 6x6 and the largest block size is 48x48. The large-sized blocks are used to focus on repairing the background with small changes, and the small-sized blocks are used to repair the suddenly appearing reflective foreground. STTN can track this area in time and space and fill the reflective area with coherent content.

[0089] Image reflection filling is as follows:

[0090]

[0091] The camera trajectory tracking of step 400 includes the following operations:

[0092] 401: Send adjacent key frames to the pre-trained SuperPoint network to obtain feature points and descriptors, select the nearest and second nearest points for each feature point, and retain the feature point if the distance ratio between the point and the nearest and second nearest points is less than 0.7, otherwise discard the point, and further use the RANSAC algorithm to discard outliers;

[0093] 402: Triangulate the matching point pairs, obtain the 3D coordinates of the feature points, and use the PnP algorithm to solve I i To I i+1 The camera transformation matrix is ​​used to update the camera trajectory.

[0094]

[0095]

[0096] like Figure 6As shown in the figure, SuperPoint uses a shared network to encode the features of the input image. The purpose of this network is to reduce the dimensionality of the image, reduce the size of the image, and extract high-dimensional features from the image. The design of this shared network can improve the consistency and efficiency of the features. It adopts a structure similar to VGG and encodes the image through three convolution-pooling layers, reducing the spatial size of the feature map to one-eighth of the original image. A feature map with 65 channels is obtained through a feature point extraction network, where 64 channel features correspond to a local area of ​​8x8 size, and 1 channel is used to store the situation where there are no feature points. The feature map is transformed to the same size as the input image through the Softmax and Reshape functions to realize the extraction of feature points, that is, the upper half of the network in the figure. The output of the shared encoder is passed through the convolution decoder to obtain a 256-channel feature map, which is enlarged by bicubic interpolation. Finally, the descriptor is L2 normalized to obtain a feature descriptor, and the result is output in the form of two channels, one channel is used to detect feature points in the image, and the other channel is used to generate descriptors of these feature points.

[0097] For each feature point descriptor, calculate the distance to all other feature point descriptors, find the nearest and second nearest matching points, calculate the distance ratio between the nearest and second nearest matching points, if the ratio is less than 0.7, retain the point pair, after matching the feature points of the key frame by the feature matching algorithm, there will be some mismatches in the obtained feature point pairs, further use the RANSAC algorithm to remove outliers to improve the matching accuracy. The spatial position of the feature point is obtained by triangulating the feature point pairs.

[0098] A spatial point P and its projection point p satisfy p ij =C i (R i P j +t i ) where R is the rotation matrix of the camera, t is the translation matrix, C is the camera intrinsic parameter matrix, and the reprojection error is e ij =p' ij -C i (R i P j +t i ) where p' ij is the observation value of the projection point, with minimizing the reprojection error as the constraint condition, Where θ=(C 1 ,…,C m ,X 1 ,…X n ) is the quantity to be optimized, namely the camera parameters and point cloud coordinate information, S ij Indicates the state of the i-th point under camera j; C irepresents the external parameters of the i-th camera, X i Represents the coordinates of the i-th point in three-dimensional space. The graph is optimized through the nonlinear optimization algorithm library g2o in C++ to obtain the corresponding position of the camera in each key frame, and the motion trajectory of the camera is obtained through spline interpolation.

[0099] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

[0100] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the disclosure disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are intended to be exemplary only, and the true scope of the present disclosure is indicated by the following claims.

Claims

1. An endoscope video positioning method based on reflection removal, characterized in that: The following steps are involved: S1, obtain monocular video of moving the endoscope in the intestine; S2, performing blur detection on a single frame image, filtering key frames and blurred images, and classifying the images into key frames and non-key frames; S3, segment the video frame, extract the reflective area, create a mask and use the ellipse structure operator to expand, deblur the non-key frames, fill the reflective area with STTN according to the information of adjacent time frames and neighboring areas, generate the texture of the corresponding area, and repair the reflective blocks in the monocular video; S4. Extract and match feature points of adjacent frames to obtain matching point pairs, perform triangulation, and solve the camera transformation matrix according to the PnP algorithm.

2. The endoscope video positioning method based on reflection removal according to claim 1 is characterized in that: The specific steps of key frame screening in step S2 include: A1. Resize the processed image and extract the green channel; A2. Use the Laplacian operator to calculate the variance response value; determine the appropriate threshold from the experiment, traverse all frames, and if the response value is less than the threshold, the current frame is considered to be a blurred image; A3. Calculate the MSSIM value for adjacent frames and determine the camera movement distance by comparing the image similarity. If it is greater than 0.85, the current frame is determined as a non-key frame with a small displacement. Otherwise, the current frame is retained as the filtered key frame.

3. The endoscope video positioning method based on reflection removal according to claim 1 is characterized in that: The specific steps of removing the reflection in step S3 include: B1. For non-key frame sequences with blurred images, the deblurring algorithm DeblurGAN is used to remove the blur caused by camera motion. B2. Use threshold segmentation to process the monocular endoscope inspection video to obtain a rough reflective area mask; B3, using the ellipse operator to perform morphological dilation on the mask to cover the reflective area and make the structure complete; B4. Obtain the spatiotemporal information corresponding to the adjacent frames of the mask area neighborhood through the spatiotemporal Transformer, repair the mask area, and generate corresponding texture information.

4. The endoscope video positioning method based on reflection removal according to claim 1 is characterized in that: The method for obtaining the camera motion trajectory by extracting feature points in step S4 is: Extract feature points from the processed image sequence, match the feature points, obtain matching point pairs in adjacent key frames, triangulate the feature points, and obtain the camera motion trajectory; It also includes: using a data set composed of various natural images to initialize the parameters of the feature point extraction model SuperPoint, using self-collected endoscopic image sequences for fine-tuning, loading and optimizing model parameters, improving the adaptability of the model in endoscopic scenarios, and then extracting key frame image feature points, obtaining the matching relationship of feature points, triangulating matching points, optimizing reprojection errors, obtaining camera poses, and updating camera trajectories.

5. The endoscope video positioning method based on reflection removal according to claim 4 is characterized in that: The pose optimization constraint of the motion trajectory is determined by the following formula: where θ=(C1,…,C m ,X1,…X n ) is the quantity to be optimized, namely the camera parameters and point cloud coordinate information, S ij Indicates the state of the i-th point under camera j; C i represents the external parameters of the i-th camera, X i represents the coordinates of the i-th point in three-dimensional space, and the reprojection error is e ij =p' ij -C i (R i P j +t i ) where p' ij is the observation value of the projection point, and minimizing the reprojection error is used as the constraint condition for pose optimization.

Images