Capsule endoscope gastrointestinal tract positioning method based on visual SLAM

Abstract

The present invention discloses a capsule endoscope gastrointestinal tract positioning method based on visual SLAM, including: performing distortion correction on all real-time images collected by the capsule endoscope to obtain a dedistorted gastrointestinal image sequence; performing pixel association between adjacent image frames of the gastrointestinal image sequence; establishing a projection transformation equation for corresponding pixels of adjacent image frames according to the result of pixel association, and extracting camera motion pose information of adjacent image frames; obtaining sparse point cloud information according to three-dimensional position information calculation; constructing a co-viewing image with multi-view constraints, optimizing the co-viewing image, and outputting image pose sequence estimation results; combining the image pose sequence estimation results, positioning and estimating the lesion image pose. The beneficial effects of the present invention are: without adding additional sensors, equipment and mechanisms, spatial position information can be acquired only by relying on the camera carried by the capsule endoscope itself, and it has the characteristics of low cost, flexibility and convenience.

Description

Technical Field

[0001] The present invention relates to the technical field of endoscopes, and in particular to a capsule endoscope gastrointestinal tract positioning method based on visual SLAM. Background Art

[0002] Endoscopic examination is currently the only means to examine the gastrointestinal tract. However, traditional esophagogastroduodenoscopes or colonoscopes are inserted through the mouth and anus respectively, and the insertion depth is limited, making it impossible to examine the entire digestive tract; while small intestinal endoscopes need to be inserted through the mouth and anus respectively, which is technically difficult. The intubation method of traditional endoscopes will cause strong discomfort to the examinee, require anesthesia, and are easily rejected, which is not conducive to the popularization of early detection and early treatment.

[0003] Capsule endoscope is a micro-electromechanical system equipped with an optical sensing unit, a computing unit and a communication unit, which can be used for clinical whole-digestive tract biopsy and lesion identification. Compared with the foreign body sensation and discomfort caused by the traditional flexible endoscopic invasive instrument, the miniaturization of the capsule robot makes it painless, non-invasive, free of foreign body sensation, and covers the whole digestive tract in clinical examination applications. It can significantly improve patient compliance and tolerance, and has important clinical significance and application value. At present, capsule endoscope is generally used as a technical means of internal digestive tract image acquisition in clinical applications. After swallowing, the capsule endoscope relies on the peristalsis of the digestive tract to pass through the esophagus, stomach, duodenum, jejunum, ileum, colon, rectum and finally be discharged from the body. The entire digestive tract image acquisition process takes 6-8 hours. Tens of thousands of images are stored in an image recorder outside the body through wireless transmission. After the transmission is completed, medical personnel observe the image sequence one by one with the naked eye on the imaging workstation to make a judgment on the health status of the digestive tract. During the visual inspection process, in addition to detecting and distinguishing lesions based on image features, medical personnel also need to feel the movement of the capsule endoscope and determine its position in the digestive tract in order to locate the lesion and provide position guidance for subsequent treatment and biopsy sampling. The entire process takes more than a week. Therefore, the autonomous positioning of the capsule endoscope in the gastrointestinal tract is of great significance to improving the inspection efficiency of medical personnel, reducing labor intensity and reducing labor costs.

[0004] At present, the positioning method of capsule endoscopes, whether in academic research or practical application, is mainly based on magnetic field positioning. By embedding a permanent magnet in the capsule endoscope, an external magnetic sensor array device is used to sense and estimate the spatial position information of the capsule endoscope in the body. The problems with this method include: 1) The need to add magnets or other sensor devices to the capsule endoscope is not conducive to the further miniaturization of the capsule endoscope, and is not conducive to the upgrade and expansion of the sensor devices such as sensors and battery capacity of the capsule endoscope; 2) It is necessary to combine with an external magnetic sensor array device to achieve positioning. The entire examination process requires the subject to lie flat / wear the device to obtain the position of the capsule endoscope in the body, which weakens the flexibility and convenience of the capsule endoscope. Summary of the invention

[0005] In view of the above problems, the present invention proposes a capsule endoscope gastrointestinal tract positioning method based on visual SLAM, which realizes the positioning of the capsule endoscope inside the digestive tract by performing data feature analysis on the existing visual sensor of the capsule endoscope and combining computer vision processing methods, multi-view geometry and nonlinear optimization theory to perform three-dimensional motion estimation and optimization of the capsule endoscope camera.

[0006] In order to solve the above technical problems, the technical solution of the present invention is as follows:

[0007] A capsule endoscope gastrointestinal tract positioning method based on visual SLAM comprises the following steps:

[0008] Calibrate the internal parameters of the capsule endoscope camera, use the calibrated capsule endoscope to collect real-time images of the gastrointestinal tract, perform distortion correction on all real-time images collected by the capsule endoscope, and obtain a dedistorted gastrointestinal image sequence;

[0009] Performing pixel association between adjacent image frames of the gastrointestinal image sequence;

[0010] According to the pixel association result, a projection transformation equation of the corresponding pixels of the adjacent image frames is established, and camera motion pose information of the adjacent image frames is extracted;

[0011] Performing triangulation calculation on the associated pixel points of the adjacent image frames to obtain three-dimensional position information of the associated pixel points, and calculating sparse point cloud information according to the three-dimensional position information;

[0012] Performing comprehensive processing according to the camera motion pose information, the sparse point cloud information, and the pixel association result, constructing a co-view image with multi-view constraints, optimizing the co-view image, and outputting an image pose sequence estimation result;

[0013] Based on the deep learning model, lesion detection is performed on the gastrointestinal tract image, the lesion image pose is output, and the position estimation of the lesion image pose is performed based on the image pose sequence estimation result.

[0014] In some implementations, the distortion correction process includes:

[0015] According to the calibrated camera intrinsic parameters, a nonlinear coordinate transformation operation is performed on all pixels in any of the real-time images:

[0016]

[0017] In the formula, r = u′ 2 +v′ 2 , u′ is the point x uv ′ is the horizontal coordinate of point x uv ′ is the vertical coordinate, u is the point x uv ′ is the horizontal coordinate after dedistortion, v is the point x uv ′ is the ordinate after distortion removal, k1, k2, k3, p1, p2 are distortion parameters;

[0018] All pixel points of the real-time image are traversed to obtain a distortion-corrected image, and multiple frames of the distortion-corrected image are combined in time sequence to form the gastrointestinal image sequence.

[0019] In some implementations, the pixel association process includes:

[0020] Select any two adjacent images in the gastrointestinal image sequence to define them as an image pair, detect feature points of the image pair, and use the feature points to match feature descriptors of feature points from different images until all the current feature points complete the first pixel matching;

[0021] For the previous image I in the image pair i Perform random pixel sampling to obtain a pixel point set, calculate the motion vector of the pixel point set in the next image, apply the motion vector to the coordinate value of the pixel point set, and define the pixel point corresponding to the coordinate value as the pixel point set in the next image I i+1 The matching pixels on the image are matched to complete the second pixel matching;

[0022] According to the results of the first pixel matching and the second pixel matching, the images are fused according to their respective sources, the pixel matching points that do not meet the preset conditions are eliminated, and two matching pixel point sets ψ about the image pair are output. i and ψ i+1 .

[0023] In some implementations, the process of acquiring the camera motion pose information includes:

[0024] Construct geometric constraints based on the two matching pixel sets:

[0025]

[0026] In the formula, is the pixel coordinate The transposed representation of and The subscripts i and i+1 indicate that the pixel comes from the matching pixel set ψ in the i-th and i+1-th images respectively. i and ψ i+1 , the superscripts n and m indicate that the pixel point is a matching pixel point set ψ i and ψ i+1 The nth and mth pixels in , E is the essential matrix;

[0027] The essential matrix is ​​decomposed and visually verified to calculate the translation t and rotation R of the capsule endoscope between adjacent frames. The obtained translation t and rotation R are unified in the matrix to obtain the rigid body transformation rule:

[0028]

[0029] Where, T i，i+1 Represents image I i+1 Relative to image I i The rigid body transformation, denoted by image I i The pose matrix is ​​T i , then image I i+1 The pose matrix T i+1 =T i ·T i，i+1 , T i+1 is image I i+1 The camera motion pose information relative to the coordinate system of the first frame image I0 in the gastrointestinal image sequence.

[0030] In some implementations, the triangulation calculation process includes:

[0031] Construct pixel coordinates separately and Equations for the projection matrix and the associated pixels:

[0032]

[0033] Where P i and P i+1 are the projection matrices corresponding to the i-th and i+1-th images respectively, and Φ is the three-dimensional coordinate value of the associated pixel point;

[0034] Solving equations (3) and (4) simultaneously yields the three-dimensional coordinates of point Φ, and then creates a three-dimensional map point M i , change the map point M i The corresponding pixel and Establish association and transform pixel points and The feature descriptor vector given to the map point M i , and obtain the sparse point cloud information.

[0035] In some implementations, the camera intrinsic parameter is calculated as follows:

[0036]

[0037] Among them, f x , f y , c x , c y are the two focal length values ​​and two optical center offsets of the camera.

[0038] In some implementations, the projection matrix is ​​calculated as follows:

[0039] P i =K[R i t i ].

[0040] In some embodiments, the co-visual map optimization process includes:

[0041] According to a preset time interval condition and a preset pixel matching point threshold, key frames are selected from the gastrointestinal image sequence to construct a key frame sequence;

[0042] Establishing a posture transformation relationship between key frames and key frames according to the camera motion posture information, traversing the sparse point cloud information, projecting each of the sparse point cloud information into all the key frames, searching for visual feature points in the neighborhood of the projection of the sparse point cloud information of the key frame, and calculating the similarity between the descriptor of the visual feature point and the descriptor of the projected sparse point cloud information, judging whether to establish an observation connection according to the similarity, and repeating the above process until the observation connection between all the sparse point cloud information and all the key frames is completed;

[0043] The co-visual map is constructed according to the key frame and the observation connection, the node values ​​of the co-visual map are adjusted, and the values ​​of each edge are minimized while satisfying the associated constraints, and finally the image pose sequence estimation is output.

[0044] The beneficial effects of the present invention are as follows: by performing inter-frame pixel association on the gastrointestinal image sequence and combining the image pose sequence estimation to locate the lesion image pose, spatial position information can be acquired only by relying on the camera carried by the capsule endoscope itself without adding additional sensors, equipment and mechanisms, and it is characterized by low cost, flexibility and convenience. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a flow chart of a capsule endoscope gastrointestinal tract positioning method based on visual SLAM disclosed in an embodiment of the present invention;

[0046] Figure 2 is a schematic diagram of pixel association;

[0047] Figure 3 Schematic diagram for localizing the lesion image pose. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the content of the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. It is understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for the convenience of description, only the parts related to the present invention are shown in the accompanying drawings, rather than all the contents.

[0049] This embodiment proposes a capsule endoscope gastrointestinal tract positioning method based on visual SLAM. Figure 1 As shown, the following steps are included:

[0050] S1, calibrating the internal parameters of the capsule endoscope camera, using the calibrated capsule endoscope to acquire real-time images of the gastrointestinal tract, performing distortion correction on all real-time images acquired by the capsule endoscope, and obtaining a dedistorted gastrointestinal image sequence.

[0051] S101, the process of camera intrinsic calibration includes:

[0052] The camera internal calibration step is to identify the optical geometric parameters of the lens and photosensitive components involved in the capsule endoscopy imaging process, including the focal length (f x , f y ), optical center offset (c x , c y ), distortion parameters (k1, k2, k3, p1, p2), etc. Place a checkerboard calibration plate of known size and specification in the field of view of the capsule endoscope camera at different distances and postures. The camera takes at least 3 images of the calibration plate to ensure that the captured images can accurately and completely detect the corner features on the calibration plate. The collected calibration images are calculated and processed using Zhang Zhengyou's calibration method to finally obtain the two focal length values ​​f of the camera on the horizontal and vertical coordinates.x and f y , the two optical center offset c on the horizontal and vertical axes x and c y , and 5 distortion parameters k1, k2, k3, p1, p2.

[0053] S102, the distortion correction process includes:

[0054] According to the calibrated camera intrinsic parameters, all pixels (u′, v′) in any real-time image I′ are transformed into nonlinear coordinates:

[0055]

[0056] In the formula, r = u′ 2 +v′ 2 , u′ is the point x uv ′ is the horizontal coordinate of point x uv ′ is the vertical coordinate, u is the point x uv ′ is the horizontal coordinate after dedistortion, v is the point x uv ′ is the ordinate after distortion removal, k1, k2, k3, p1, p2 are distortion parameters;

[0057] All pixel points of the real-time image are traversed to obtain a distortion-corrected image I, and multiple frames of the distortion-corrected image I are combined into a gastrointestinal image sequence in time sequence.

[0058] S2, performing pixel association between adjacent image frames of the gastrointestinal image sequence.

[0059] After S1 completes the image distortion correction process, it starts the inter-frame pixel association step. This step aims to find pixel matching pairs in adjacent images so that they correspond to the same point in space, as follows Figure 2 In order to cope with the challenges of repeated gastrointestinal texture and weak feature discrimination, this embodiment uses feature point-based and optical flow-based methods to perform pixel matching on the same adjacent image frames, and then uses random sampling consistency to evaluate the matching results to detect and eliminate incorrect matching pairs.

[0060] Optionally, the pixel association process includes:

[0061] S201, pixel matching based on feature points: select any two adjacent images in the gastrointestinal image sequence, define them as image pairs, detect feature points of the image pairs, and use the feature points to match feature descriptors of feature points from different images until all current feature points have completed the first pixel matching.

[0062] First, two adjacent images collected by capsule endoscopy are subjected to distortion correction using S1 to obtain two adjacent dedistorted images I i and Ii+1 , subscripts i and i+1 are the serial labels of the images.

[0063] Next is the image I i and I i+1 Visual feature point detection is performed. Currently, common visual feature point detection algorithms include SIFT, SURF, ORB, etc. Optionally, SIFT is used as a visual feature point detection and feature descriptor generation method in the present invention. i and I i+1 SIFT feature point detection is performed separately, and SIFT descriptor vectors are generated for the detected feature points. The feature points and descriptor vector sets detected in the two images are recorded as and in and Respectively represent images from I i and I i+1 The feature point coordinates, and Represents the feature points and and descriptor vector, the superscripts n and m in the set refer to the retrieval number of the feature point.

[0064] The descriptor vectors are then used to match feature points from different images. Optionally, the Euclidean distance between two descriptor vectors is used as the similarity evaluation between the two. i Feature points Using the FLANN algorithm in image I i+1 A set of descriptors Find and The closest descriptor vector and the next closest descriptor vector After this, if you remember for and The Euclidean distance of for and If the Euclidean distance and If the ratio of is less than a certain artificially set threshold, then The corresponding feature points Considered as feature points Matching is successful. Traverse image I according to the above matching process i All feature points Complete pixel matching based on feature points.

[0065] S202, pixel matching based on optical flow method: for the previous image I in the image pairi Perform random pixel sampling to obtain a pixel set, calculate the motion vector of the pixel set in the next image, apply the motion vector to the coordinate value of the pixel set, and define the pixel point corresponding to the coordinate value as the pixel set in the next image I i+1 The matching pixels on the , complete the second pixel matching.

[0066] First, the dedistorted image I obtained in step 201 is i Perform random pixel sampling to obtain a pixel point set Remember each sampled pixel The pixel intensity value is The pixel coordinate value is Then The pixel gradient of the neighborhood is used as the initial condition and the Lucas-Kanade algorithm is used to calculate In the dedistorted image I i+1 The moving vector in Apply the movement vector to On the coordinates of Then the coordinate value The corresponding pixel is the sampling pixel In the dedistorted image I i+1 According to the above matching process, traverse image I i All sampled pixels Complete pixel matching based on optical flow method.

[0067] S203, elimination of matching outliers: Based on the results of the first pixel matching and the second pixel matching, the pixels that do not meet the preset conditions are eliminated according to their respective image sources, and two matching pixel sets ψ about the image pair are output. i and ψ i+1 .

[0068] The two pixel matching results obtained in S201 and S202 are fused according to the images from which they come, and the image I can be obtained. i The matching pixel set χ i , and from image I i+1 The matching pixel set χ i+1 , and there is a pair of sets χ i For any pixel in i+1 The pixels in are paired with it.

[0069] Specifically, from the set χ i Randomly sample at least 4 points from the i ′, and from the set χ i+1Find the pixel points corresponding to these sampling points and form a subset x i+1 ′. Using the subset χ i ′ and χ i+1 ′Estimated image I i and I i+1 The homography matrix between Apply H to χ i For the remaining points that have not been sampled, obtain these points in I i+1 The estimated coordinates of the point on χ i+1 The Euclidean distance is calculated based on the coordinate values ​​of the corresponding points in the set. The sum of the Euclidean distances of all the action points, e, is regarded as the model error of the homography matrix H, which is also the consistency evaluation of this random sampling. Repeat the above process until e is less than the set threshold. The homography matrix H estimated in this sampling is regarded as meeting the set χ i and χ i+1 The projective transformation model.

[0070] Apply the obtained homography matrix H to χ i , and calculate the coordinate value of each point after the action and the value of the point in χ i+1 The Euclidean distance e of the coordinate values ​​of the matching points in i , if e i Less than the artificially set threshold e th , then the pixel matching pair is called an internal point and is stored; otherwise, it is called an external point and is not retained.

[0071] After the above operations, we finally get the image I i The set of matching pixels ψ i , and from image I i+1 The set of matching pixels ψ i+1 , where ψ i is x i A subset of i+1 is x i+1 Subset of, same as above, for the set ψ i For any pixel in, there is only one set ψ i+1 The pixels in are paired with it.

[0072] S3, based on the result of pixel association, establish the projection transformation equation of the corresponding pixels of adjacent image frames, and extract the camera motion posture information of the adjacent image frames.

[0073] The process of obtaining camera motion pose information includes:

[0074] Construct geometric constraints based on two sets of matching pixels:

[0075]

[0076] In the formula, is the pixel coordinate The transposed representation of and The subscripts i and i+1 indicate that the pixel comes from the matching pixel set ψ in the i-th and i+1-th images respectively. i and ψ i+1 , the superscripts n and m indicate that the pixel point is a matching pixel point set ψ i and ψ i+1 The nth and mth pixels in the image are E, which is the essential matrix and the camera translation motion between adjacent frames. The cross product of the rotational motion R∈SO(3), that is, E=t×R;

[0077] The essential matrix is ​​decomposed and visualized to calculate the translation t and rotation R of the capsule endoscope between adjacent frames. The obtained translation t and rotation R are unified in the matrix to obtain the rigid body transformation rule:

[0078]

[0079] Where, T i，i+1 Represents image I i+1 Relative to image I i The rigid body transformation, denoted by image I i The pose matrix is ​​T i , then image I i+1 The pose matrix T i+1 =T i ·T i，i+1 , T i+1 is image I i+1 The camera motion pose information relative to the coordinate system of the first frame image I0 in the gastrointestinal image sequence.

[0080] S4, performing triangulation calculation on the associated pixel points of adjacent image frames to obtain the three-dimensional position information of the associated pixel points, and calculating the sparse point cloud information according to the three-dimensional position information.

[0081] The triangulation process includes:

[0082] Construct pixel coordinates separately and Equations for the projection matrix and associated pixels:

[0083]

[0084] Where P i and P i+1are the projection matrices corresponding to the i-th and i+1-th images, respectively, including the camera intrinsic parameters and shooting posture information of the image, and Φ is the three-dimensional coordinate value of the associated pixel point;

[0085] Solving equations (3) and (4) simultaneously yields the three-dimensional coordinates of point Φ, and then creates a three-dimensional map point M i , change the map point M i The corresponding pixel and Establish association and transform pixel points and The feature descriptor vector given to the map point M i , and obtain the sparse point cloud information.

[0086] The calculation method of the camera intrinsic parameters is:

[0087]

[0088] Among them, f x , f y , c x , c y are the two focal length values ​​and two optical center offsets of the camera.

[0089] The projection matrix is ​​calculated as:

[0090] P i =K[R i t i ].

[0091] S5, based on the camera motion pose information, sparse point cloud information, and pixel association results, a multi-view constrained co-view image is constructed, the co-view image is optimized, and the image pose sequence estimation result is output.

[0092] After S1-S4, pixel matching, motion estimation and pixel triangulation of adjacent frames have been completed, and the motion information of adjacent frames and sparse local point cloud maps have been obtained. However, relying solely on the cumulative motion estimation of multiple adjacent frames will cause the rapid accumulation of estimation errors and fail to accurately position the capsule endoscope in the entire gastrointestinal tract. Therefore, it is necessary to optimize the estimated pose sequence. In order to improve the optimization speed and robustness, it is necessary to first formulate a key frame extraction strategy to reduce the optimization scale; secondly, the pixels in the key frames are associated with cross-frame data to construct a multi-constrained co-view map; finally, the constructed co-view map is optimized to achieve the optimization of the entire pose sequence and improve the positioning accuracy of the capsule endoscope.

[0093] The optimization process of the co-visual graph includes:

[0094] S501, selecting key frames from a gastrointestinal image sequence according to a preset time interval condition and a preset pixel matching point threshold, and constructing a key frame sequence.

[0095] Most capsule endoscopes on the market collect images at a frequency of 3-5 frames per second. During the examination of a digestive tract about 9 meters in length, tens of thousands of images will be collected, most of which are highly overlapping and redundant. In order to improve the efficiency of image pose optimization, a strategy needs to be formulated to optimize the pose of some key and important image frames. The extraction of key frames must meet two conditions: 1) Time interval: t image frames must pass from the previous key frame; 2) Number of pixel matching points: The current image frame must retain at least N successfully matched pixels after S203; t and N are manually set parameters. Using these two conditions, it is determined whether to convert the current image I i As key frame K i Save it. When saving, save the image I i The image information, posture information, feature point extraction and matching are assigned to the key frame K i . Key frame extraction and judgment are performed on each newly collected image to construct a key frame sequence.

[0096] S502, establish the posture transformation relationship between key frames according to the camera motion posture information, traverse the sparse point cloud information, project each sparse point cloud information into all the key frames, search for visual feature points in the neighborhood of the projection of the sparse point cloud information of the key frame, and calculate the similarity between the descriptor of the visual feature point and the descriptor of the projected sparse point cloud information, determine whether to establish an observation connection based on the similarity, and repeat the above process until the observation connection between all sparse point cloud information and all key frames is completed.

[0097] The co-view construction realizes the cross-frame data association between key frames and establishes the observation constraint relationship between key frames and key frames, and key frames and map points. First, the pose transformation relationship between each adjacent image frame can be obtained by using step 3. The special image frame of the key frame is retained, so it also carries the pose relationship of the image frame. Therefore, the pose transformation relationship between key frames can be established by using the pose transformation chain. Secondly, for each map point M i , find the corresponding relationship with the key frame, that is, determine which key frames the map point exists in. To this end, first traverse all map points and project each map point into all key frames; find visual feature points in the neighborhood of the key frame map point projection, and calculate the similarity between the descriptor of the visual feature point and the descriptor of the projected map point to determine whether to establish an observation connection; repeat the above process to complete the observation connection between all map points and all key frames.

[0098] S503, constructing a co-visual map based on key frames and observation connections, adjusting the node values ​​of the co-visual map, minimizing the values ​​of each edge while satisfying the association constraints, and finally outputting the image pose sequence estimation result.

[0099] After step 502, the connections between keyframes and keyframes, and keyframes and map points are obtained and saved in a graph structure. In the graph, there are two types of nodes, one for storing the pose information of keyframes, denoted as NC i ∈SE(3); one type is used to store map point coordinate information, denoted as Therefore, there are two types of edges. One is the associated edge between key frames, which reflects the posture transformation relationship between key frames and is recorded as The other type is the associated edge between the keyframe and the map point, which reflects the observation relationship between the map points in the image and is recorded as The pose sequence optimization process is to adjust the node values ​​to minimize the values ​​of each edge while satisfying the associated constraints. This process can be solved using graph optimization tools such as g2o and ceres.

[0100] S6, based on the deep learning model, performs lesion detection on the gastrointestinal tract image, outputs the lesion image pose, and combines the image pose sequence estimation results to estimate the position of the lesion image pose, so as to realize the positioning of the lesion tissue at the image sequence level, such as Figure 3 shown.

[0101] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable ordinary technicians in the field to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made based on the essence of the content of the present invention should be included in the protection scope of the present invention.

Claims

1. A capsule endoscope gastrointestinal tract positioning method based on visual SLAM, characterized in that: The following steps are involved: Calibrate the internal parameters of the capsule endoscope camera, use the calibrated capsule endoscope to collect real-time images of the gastrointestinal tract, perform distortion correction on all real-time images collected by the capsule endoscope, and obtain a dedistorted gastrointestinal image sequence; Performing pixel association between adjacent image frames of the gastrointestinal image sequence; According to the pixel association result, a projection transformation equation of the corresponding pixels of the adjacent image frames is established, and camera motion pose information of the adjacent image frames is extracted; Performing triangulation calculation on the associated pixel points of the adjacent image frames to obtain three-dimensional position information of the associated pixel points, and calculating sparse point cloud information according to the three-dimensional position information; Performing comprehensive processing according to the camera motion pose information, the sparse point cloud information, and the pixel association result, constructing a co-view image with multi-view constraints, optimizing the co-view image, and outputting an image pose sequence estimation result; Based on the deep learning model, lesion detection is performed on the gastrointestinal tract image, and the lesion image pose is output. Combined with the image pose sequence estimation result, the lesion image pose is positioned and estimated.

2. The capsule endoscope gastrointestinal tract positioning method based on visual SLAM as claimed in claim 1, characterized in that: The distortion correction process includes: According to the calibrated camera intrinsic parameters, a nonlinear coordinate transformation operation is performed on all pixels in any of the real-time images: In the formula, , For point The horizontal axis of For point The vertical coordinate of For point The horizontal axis after dedistortion, For point The ordinate after dedistortion, , , , , is the distortion parameter; All pixel points of the real-time image are traversed to obtain a distortion-corrected image, and multiple frames of the distortion-corrected images are combined in time sequence to form the gastrointestinal image sequence.

3. The capsule endoscope gastrointestinal tract positioning method based on visual SLAM as claimed in claim 1, characterized in that: The pixel association process includes: Select any two adjacent images in the gastrointestinal image sequence to define them as an image pair, detect feature points of the image pair, and use the feature points to match feature descriptors of feature points from different images until all the current feature points complete the first pixel matching; For the previous image in the image pair Perform random pixel sampling to obtain a pixel point set, calculate the motion vector of the pixel point set in the next image, apply the motion vector to the coordinate value of the pixel point set, and define the pixel point corresponding to the coordinate value as the pixel point set in the next image. The matching pixels on the image are matched to complete the second pixel matching; According to the results of the first pixel matching and the second pixel matching, the images are fused according to their respective sources, pixel matching points that do not meet the preset conditions are eliminated, and two matching pixel point sets about the image pair are output. .

4. The capsule endoscope gastrointestinal tract positioning method based on visual SLAM as claimed in claim 3, characterized in that: The process of obtaining the camera motion posture information includes: Construct geometric constraints based on the two matching pixel sets: ； In the formula, The transposed representation of and Subscript and Respectively indicate that the pixel comes from Zhang Hedi The set of matching pixels in the image , superscript and Indicates that the pixel point is the matching pixel point set The and Pixels is the essential matrix; The essential matrix is ​​decomposed and visualized to calculate the translation of the capsule endoscope between adjacent frames. and the amount of rotation , the translation amount obtained is and the amount of rotation Unified in the matrix, we get the rigid body transformation rules: ； In the formula, Representing images Relative to the image The rigid body transformation, record image The pose matrix is , then the image The pose matrix , is an image Relative to the first frame image in the gastrointestinal image sequence The camera motion pose information in terms of the coordinate system.

5. The capsule endoscope gastrointestinal tract positioning method based on visual SLAM as claimed in claim 4, characterized in that: The triangulation calculation process includes: Construct pixel coordinates separately and Equations for the projection matrix and the associated pixels: ； ； In the formula, and They are and The projection matrix corresponding to the image, is the three-dimensional coordinate value of the associated pixel point; Solving the simultaneous equations (3) and (4) yields The three-dimensional coordinate values ​​are then used to create three-dimensional map points , point the map The corresponding pixel and Establish association and transform pixel points and The feature descriptor vector given to the map point , and obtain the sparse point cloud information.

6. The capsule endoscope gastrointestinal tract positioning method based on visual SLAM as claimed in claim 5, characterized in that: The calculation method of the camera intrinsic parameter is: ； in, and are the two focal length values ​​of the camera, and is the offset between the two optical centers of the camera.

7. The capsule endoscope gastrointestinal tract positioning method based on visual SLAM as claimed in claim 6, characterized in that: The calculation method of the projection matrix is: ； in, It is The rotation of the camera pose for the image, It is The translation of the camera pose for each image.

8. The capsule endoscope gastrointestinal tract positioning method based on visual SLAM as claimed in claim 1, characterized in that: The optimization process of the co-visual graph includes: According to a preset time interval condition and a preset pixel matching point threshold, key frames are selected from the gastrointestinal image sequence to construct a key frame sequence; Establishing a posture transformation relationship between key frames and key frames according to the camera motion posture information, traversing the sparse point cloud information, projecting each of the sparse point cloud information into all the key frames, searching for visual feature points in the neighborhood of the projection of the sparse point cloud information of the key frame, and calculating the similarity between the descriptor of the visual feature point and the descriptor of the projected sparse point cloud information, judging whether to establish an observation connection according to the similarity, and repeating the above process until the observation connection between all the sparse point cloud information and all the key frames is completed; The co-visual map is constructed according to the key frame and the observation connection, the node values ​​of the co-visual map are adjusted, and the values ​​of each edge are minimized while satisfying the associated constraints, and finally the image pose sequence estimation is output.

Images