Capsule endoscope intestinal panoramic image splicing method and device, computer readable storage medium and computer equipment
By matching and expanding the intestinal images collected by the capsule endoscope, calculating the displacement difference of the new feature points and cropping the step area image, the problem of insufficient fluency and continuity at the image stitching in the prior art is solved, and more accurate step distance calculation and image stitching effect are achieved.
Patent Information
- Application Number
- CN202510724741.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-03
- Publication Date
- 2025-07-01
- Estimated Expiration
- 2045-06-03
AI Technical Summary
The existing capsule endoscopic intestinal panoramic image stitching method is too ideal for judging the stepping area, resulting in abrupt and redundant information at the joints, affecting the fluency and continuity of the image.
By matching the original intestinal images collected by the capsule endoscope, the feature set of feature points of each image is obtained, and polar coordinates are expanded, the displacement difference of the new feature points is calculated, and the stepping area image is cropped, and polar coordinates are expanded and stitched to obtain a panoramic intestinal image.
This method can calculate a more accurate real step distance, improve the fluency and continuity of image stitching, reduce information redundancy, and enhance diagnostic accuracy.
Smart Images

Figure CN120235753A_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the technical field of capsule endoscopes. Specifically, it relates to a method and device for stitching panoramic images of the intestinal tract of a capsule endoscope, a computer-readable storage medium, and a computer device. Background Art
[0002] Digestive tract diseases are common and highly prevalent health problems globally, covering a variety of pathological conditions including gastritis, gastric ulcer, colitis, Crohn's disease, digestive tract bleeding, intestinal tumors, etc. The incidence of these diseases remains high, and their symptoms are complex and diverse, often involving chronic inflammation, bleeding, dysfunction, and even malignant lesions, seriously threatening the health and quality of life of patients. With the accelerating process of global population aging and the changes in eating habits and lifestyles, the incidence of digestive tract diseases continues to rise, becoming one of the major problems affecting public health. In addition, the diagnosis and treatment of digestive tract diseases still face multiple challenges. Early lesions usually have atypical symptoms and are easily overlooked, while late-stage lesions may have progressed to severe complication stages, increasing the complexity and risk of treatment. Therefore, how to achieve early screening, accurate diagnosis, and personalized treatment of digestive tract diseases has become a key research direction in the medical field.
[0003] Medical endoscopes play an important role in the diagnosis of digestive tract diseases. Traditional endoscopy usually requires a doctor to insert a long cable into the patient's body to image key areas and provide diagnosis and treatment plans. However, due to its invasive nature, traditional intestinal endoscopes require a certain amount of mechanical force to advance the endoscope, which can cause strong pain and discomfort to patients, especially those with sensitive intestines; in addition, the use of such endoscopes also poses a risk of potential complications, such as infection, perforation, and tearing; moreover, due to the long length of the intestine (usually 4-6 meters) and its coiled shape, it is more difficult to advance traditional endoscopes in the intestine. Compared with the stomach and colon, the examination of the intestine is more difficult; in addition, the imaging resolution of traditional intestinal endoscopes is limited, and they may not be able to identify very small or early lesions, and lack intelligent auxiliary functions such as image enhancement and lesion recognition, increasing the difficulty of doctor diagnosis.
[0004] Capsule endoscope is a non-invasive inspection device for examining the digestive tract. It is shaped like a capsule and is equipped with a high-definition camera, light source, image sensor and wireless transmission device. The patient only needs to swallow the capsule endoscope, which will move downward with the natural peristalsis of the digestive tract and capture and record images of the digestive tract in the process. These images are transmitted wirelessly to an external receiving device for doctors to examine and analyze. The application range of capsule endoscopes usually includes the stomach, intestines and large intestine. It is a comfortable, convenient and painless endoscopic examination method. The capsule enteroscope is a capsule endoscope specially used for intestinal examination. It is suitable for detailed examination and lesion detection of the intestine. Compared with traditional endoscopes, it is simple to operate, has a wider inspection range, and can reduce damage to the intestine. At the same time, the operation of the enteroscope is relatively simple, does not require complex endoscopic operations, and patients do not need to stay in the hospital for observation, which makes the examination more flexible and convenient; most importantly, modern capsule enteroscopes can be combined with various advanced algorithms such as artificial intelligence technology to perform intelligent analysis of images, helping doctors to more accurately detect lesions and improve diagnostic efficiency and accuracy.
[0005] For tens of thousands of intestinal images taken by capsule endoscopes, an intestinal panoramic image can be constructed by image stitching. The algorithm stitches the continuous images taken in the intestine into a complete image or video, which helps doctors better observe the internal structure and lesions of the intestine. At present, the intestinal stitching algorithm mainly uses feature point recognition and matching algorithms, selects uniform circular rings as the stepping distance of two frames of continuous images, and stitches the cut rectangular stepping distances by polar coordinate expansion method. Through the stitching construction of intestinal panoramic images, the whole picture of the intestine can be intuitively displayed, helping doctors to observe the structure of the intestine and possible lesions more comprehensively; at the same time, the continuous area inside the intestine can be fully displayed, reducing the risk of misdiagnosis and missed diagnosis caused by ignoring local frames; in addition, intestinal stitching can also help doctors more easily locate specific lesion areas and record the exact location of the lesion (such as the distance from the pylorus or ileocecal valve), thereby providing a reference for further diagnosis or treatment (such as endoscopic intervention).
[0006] However, the current panoramic image stitching method is too idealistic in judging the stepping area. Since the position of the capsule is not fixed during its movement in the intestine, the stepping distance between two consecutive frames of images is not a uniform annular area. If it is selected according to the existing method, the seam will be too abrupt and contain too much redundant information. Summary of the invention
[0007] The technical problem solved by the present application is: how to obtain a more accurate step distance between two consecutive frames of images to improve the smoothness and continuity of the image splicing.
[0008] The present application provides a method for stitching panoramic images of the intestinal tract of a capsule endoscope, and the method includes: Performing feature matching on the original intestinal image set collected by the capsule endoscope to obtain the feature point set of each image in the original intestinal image set; Performing polar coordinate expansion on each original intestinal image in the original intestinal image set to obtain a rectangular intestinal image and a new feature point set corresponding to the feature point set in the rectangular intestinal image; Calculating the displacement difference of each new feature point according to the coordinates of each new feature point in the new feature point sets of two adjacent frames of rectangular intestinal images; Cropping the stepping area image between two adjacent original intestinal images according to the displacement difference of each new feature point; Performing polar coordinate expansion on the stepping area image as an additional image segment between two adjacent frames of rectangular intestinal images; Sequentially stitching the obtained additional image segments to obtain a panoramic intestinal image.
[0009] Optionally, the method for performing feature matching on the original intestinal image set collected by the capsule endoscope includes: Removing noise and redundant images from the original intestinal image set to obtain a preprocessed original intestinal image; Extracting the initial feature point set of each image of the preprocessed original intestinal image based on the SuperPoint feature extraction model; Performing feature matching on the initial feature point sets of two adjacent frames of original intestinal images in sequence based on the SuperGlue feature matching model to obtain the matching feature point set of each original intestinal image; Dividing each original intestinal image into several regions, screening matching feature points from each region according to different confidence thresholds, and the screened matching feature points constitute the feature point set of each original intestinal image.
[0010] Optionally, the method for dividing each original intestinal image into several regions and screening matching feature points from each region according to different confidence thresholds includes: Dividing each original intestinal image into a central circle region and several concentric ring regions with gradually increasing radii, and the confidence thresholds of each region decrease in order from the inside to the outside; Screening out the matching feature points greater than the corresponding confidence threshold from each region.
[0011] Optionally, the performing polar coordinate expansion on each original intestinal image in the original intestinal image set to obtain a rectangular intestinal image and a new feature point set corresponding to the feature point set in the rectangular intestinal image includes: Calculating the relative rotation angle between two adjacent frames of original intestinal images; The original intestinal image is expanded in polar coordinates according to the relative rotation angle to obtain a rectangular intestinal image; Polar coordinate transformation is performed according to the coordinates of each feature point in the feature point set and the relative rotation angle to obtain the polar coordinate values corresponding to each feature point; Normalization processing is performed on each polar coordinate value according to the width and height of the rectangular intestinal image to obtain a new feature point set corresponding to the rectangular intestinal image, where the new feature points that match between two adjacent frames of rectangular intestinal images are on the same horizontal line.
[0012] Optionally, calculating the displacement difference of each new feature point according to the coordinates of each new feature point in the new feature point set of two adjacent frames of rectangular intestinal images includes: Taking the difference between the radius coordinates of the new feature points that match between two adjacent frames of rectangular intestinal images as the displacement difference of the new feature points between two adjacent frames of rectangular intestinal images.
[0013] Optionally, cropping the stepped area image between two adjacent frames of original intestinal images according to the displacement difference of each new feature point includes: According to the width of the rectangular intestinal image and the displacement difference of each new feature point, the stepped distance points of the latter frame of rectangular intestinal image in the adjacent former frame of rectangular intestinal image are obtained; Performing polar coordinate transformation on the polar coordinates of each stepped distance point to obtain the marked points corresponding to each stepped distance point in the former frame of original intestinal image; According to the maximum inscribed circle fitted by each marked point, the area image of the former frame of original intestinal image outside the maximum inscribed circle is the stepped area image between two adjacent frames of original intestinal images.
[0014] Optionally, the method of expanding the stepped area image in polar coordinates as the new image segment between two adjacent frames of rectangular intestinal images includes: During the process of expanding the stepped area image in polar coordinates, the remap function is used to perform stretching processing on the pixels at different radii in different proportions to obtain a rectangular image segment as the new image segment between two adjacent frames of rectangular intestinal images.
[0015] The present application also discloses a capsule endoscope intestinal panoramic image stitching device, and the device includes: A feature matching module, configured to perform feature matching on the original intestinal image set collected by the capsule endoscope to obtain the feature point set of each image in the original intestinal image set; An image expansion module, configured to expand each original intestinal image in the original intestinal image set in polar coordinates to obtain a rectangular intestinal image and a new feature point set corresponding to the feature point set in the rectangular intestinal image; A step distance calculation module, configured to calculate the displacement difference of each new feature point based on the coordinates of each new feature point in the new feature point sets of two adjacent frames of rectangular intestinal images; A step area cropping module, configured to crop the step area image between two adjacent frames of original intestinal images according to the displacement difference of each new feature point; An image stretching and unfolding module, configured to perform polar coordinate unfolding on the step area image as the new image segment between two adjacent frames of rectangular intestinal images; An image stitching module, configured to sequentially stitch the obtained new image segments to obtain a panoramic intestinal image.
[0016] This application also discloses a computer-readable storage medium storing a capsule endoscope intestinal panoramic image stitching program, which, when executed by a processor, implements the above-mentioned capsule endoscope intestinal panoramic image stitching method.
[0017] This application also discloses a computer device, which includes a computer-readable storage medium, a processor, and a capsule endoscope intestinal panoramic image stitching program stored in the computer-readable storage medium, and the capsule endoscope intestinal panoramic image stitching program, when executed by the processor, implements the above-mentioned capsule endoscope intestinal panoramic image stitching method.
[0018] A capsule endoscope intestinal panoramic image stitching method, device, computer-readable storage medium, and computer device provided by this application have the following technical effects: By calculating the displacement difference based on the coordinates of each feature point and then cropping the step area image from the original image according to the displacement difference, this method can calculate a more accurate real step distance, which is beneficial to improving the smoothness and continuity of the image stitching. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] Figure 1 It is a flowchart of a capsule endoscope intestinal panoramic image stitching method according to one or more embodiments; Figure 2 It is a schematic block diagram of a capsule endoscope intestinal panoramic image stitching device according to one or more embodiments; Figure 3 It is a schematic block diagram of a computer device according to one or more embodiments. DETAILED DESCRIPTION OF THE EMBODIMENTS
[0020] In order to make the objectives, technical solutions, and advantages of this application clearer, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not used to limit this application.
[0021] Before describing the various embodiments of the present application in detail, first briefly describe the technical concept of the present application: Currently, when stitching intestinal images of a capsule endoscope, due to the factors of the pose change during the movement of the capsule, the step distance between two consecutive images is not a uniform circular area, and existing stitching methods will cause problems such as abruptness and redundancy at the stitching location. For this reason, the present application provides a method for stitching panoramic intestinal images of a capsule endoscope. The key improvement point is to expand the original intestinal image in the polar coordinate system, calculate the displacement difference through the coordinates of each feature point, and then return to the original image to crop the step area image according to the displacement difference. This method can calculate a more accurate real step distance, which is beneficial to improving the smoothness and continuity of the image stitching location. The following describes the specific principle of the method for stitching panoramic intestinal images of the capsule endoscope of the present application in combination with more embodiments.
[0022] Specifically, as Figure 1 shown, the method for stitching panoramic intestinal images of a capsule endoscope in the first embodiment includes the following steps: Step S10: Perform feature matching on the original intestinal image set collected by the capsule endoscope to obtain the feature point set of each image in the original intestinal image set.
[0023] Step S20: Expand each original intestinal image in the original intestinal image set in polar coordinates to obtain a rectangular intestinal image and the corresponding new feature point set of the feature point set in the rectangular intestinal image.
[0024] Step S30: Calculate the displacement difference of each new feature point according to the coordinates of each new feature point in the new feature point sets of two adjacent rectangular intestinal images.
[0025] Step S40: Crop the step area image between two adjacent original intestinal images according to the displacement difference of each new feature point.
[0026] Step S50: Expand the step area image in polar coordinates as a new image segment between two adjacent rectangular intestinal images.
[0027] Step S60: Sequentially stitch the obtained new image segments to obtain a panoramic intestinal image.
[0028] In one or more embodiments, before step S10, a set of original intestinal images collected by a capsule endoscope. Specifically, the patient swallows a capsule endoscope, which enters the intestinal environment along the digestive tract and advances along the human intestine under the action of intestinal peristalsis. At this time, with the assistance of a supplementary light LED, the built-in lens takes a set of human intestinal pictures in a certain time sequence and stores them in a data recorder worn by the patient. After the capsule completes the entire shooting process and is discharged from the human body, the data recorder can be removed, and the taken pictures can be imported into a computer to obtain a set of original intestinal images.
[0029] Further, when performing feature matching, it is necessary to remove noise and redundant images from the set of original intestinal images to obtain the preprocessed original intestinal images. Specifically, first, the original intestinal images are converted to the HLS space, and histogram equalization and 5×5 average filtering are performed on the L channel respectively to remove the noise in the images. Since the movement of the intestine is non-uniform, a large number of redundant pictures will appear in the areas where the intestinal peristalsis is slower in the images collected by the data recorder. Therefore, next, the histogram of the pictures needs to be statistically analyzed, and the histogram similarity of two adjacent pictures is compared, and the pictures with a similarity exceeding 0.9 with the previous one are removed. After that, all the images are arranged in reverse order to obtain the preprocessed original intestinal images.
[0030] After the preprocessing of the images, in order to reflect the stepping distance of the capsule endoscope between two adjacent frames of images, it is necessary to perform feature matching on the two images, and the displacement change of the feature points is used to reflect the stepping distance of the nearby area between the two frames of images. Traditional feature point recognition methods such as SIFT (Scale-Invariant Feature Transform) and SURF (Speeded-Up Robust Features) have limited matching accuracy in the intestinal environment, and the generated feature points are often too dense or unevenly distributed. Traditional matching methods such as FLANN (Fast Library for Approximate Nearest Neighbors) and Brute-Force Matching have certain limitations respectively. FLANN uses approximate search and will sacrifice the accuracy of matching to a certain extent; the time complexity of brute-force matching is too high, and when the number of feature points is large, the calculation time is extremely slow, making it difficult to meet the real-time requirements.
[0031] Based on the limitations of the traditional methods, in this embodiment, first, an initial set of feature points of each image of the preprocessed original intestinal images is extracted based on the SuperPoint feature extraction model. Then, based on the SuperGlue feature matching model, the initial sets of feature points of two adjacent frames of original intestinal images are sequentially subjected to feature matching to obtain a set of matching feature points for each original intestinal image. Finally, each original intestinal image is divided into several regions, and matching feature points are screened from each region according to different confidence thresholds. The selected matching feature points constitute the set of feature points of each original intestinal image.
[0032] Specifically, SuperPoint is an end-to-end deep learning model for keypoint detection and descriptor generation of images. It extracts features from images through a convolutional neural network (CNN). With the advantages of end-to-end learning, high robustness, and real-time performance, SuperPoint is particularly suitable for feature point extraction and matching of capsule endoscope images. In addition, SuperPoint can be seamlessly integrated with SuperGlue to further optimize the feature point matching relationship between image pairs. The SuperGlue method realizes accurate and robust feature point matching by combining graph neural networks and attention mechanisms. The SuperGlue method can better understand the global and local information in images, greatly improving the matching accuracy and robustness, and can obtain a set of feature point matches with a larger number and higher accuracy compared to traditional methods. Exemplarily, the SuperPoint used here can directly adopt the pre-trained model SuperPoint_v1.pth on the github website, which can be directly downloaded and used without modification and adaptation, and relatively robust feature points can be extracted. The website address is: https: / / github.com / magicleap / SuperGluePretrainedNetwork / tree / master / models / weights .
[0033] Among them, the basic process of feature matching of the initial feature point sets of two adjacent frames of original intestinal images based on the SuperGlue feature matching model in sequence includes: (1) Extraction of input feature points: Obtain feature points based on the SuperPoint network. (2) Construction of a feature point graph: Represent feature points as graph nodes and input them into the feature point graph. (3) Processing by graph neural networks to obtain the global relationship between feature points through iterative updates. (4) Optimization of matching by the attention mechanism to find possible matching points and generate a matching matrix. (5) Preliminary screening process: Set an initial threshold to filter out some points with very low confidence. (6) Output the final matching result.
[0034] Exemplarily, the SuperGlue used here can directly adopt the pre-trained model SuperGlue_indoor.pth on the github website, which can be directly downloaded and used without modification and adaptation. Using this model, relatively robust matching feature point pairs can be extracted. The specific download website address of this model is: https: / / github.com / magicleap / SuperGluePretrainedNetwork / tree / master / models / weights .
[0035] In other embodiments, other feature matching models can also be used to replace the SuperGlue-based feature matching model. The feature point matching method LoFTR based on Transformer can directly perform dense feature matching between image pairs, can handle image pairs with low texture regions and scale changes, and has higher matching accuracy and robustness in complex scenarios; in addition, the local descriptor method D2-Net based on deep features, this end-to-end learning method for feature detection and description uses a deep neural network to directly extract robust feature points and descriptors from images.
[0036] Furthermore, after obtaining the matching feature point sets of each original intestinal image, the feature points will be screened by confidence. However, due to the particularity of the intestinal environment, some good feature points may be filtered out due to low confidence, which may lead to the lack of step distances in some areas, and ultimately result in more information loss in the stitched image. Therefore, the method for screening matching feature points in this embodiment includes: dividing each original intestinal image into a central circular region and several concentric circular ring regions with gradually increasing radii, and in the order from the inside to the outside, the confidence thresholds of each region decrease; screening out the matching feature points greater than the corresponding confidence thresholds from each region.
[0037] Exemplarily, for any two consecutive original intestinal images, the matching feature point sets of the two original intestinal images are respectively denoted as and , and the confidence of the matching feature point pairs is denoted as . Then, the circular rings are used later to partition the feature points. Among them, the confidence is output by the above-mentioned SuperGlue model, which represents the probability that two points are matched. First, calculate the maximum radius of the effective region of the original intestinal image from the center of the image. After that, set three circular ring regions from the inside to the outside, with radii from the inside to the outside being to respectively. The image is divided into three parts by the circular rings, and the central region is a circular region. For the outer ring region ( to interval) with a relatively close shooting distance and obvious feature influence, set a lower confidence threshold of 0.3 to retain more feature points. For the inner ring region ( to interval) with a relatively far shooting distance and large deformation, which will cause a large error in the step distance, set a higher confidence threshold of 0.7 and retain only a few excellent feature points. Through this step, the screened matching feature points are obtained, re-ordered, and the feature point sets and of two adjacent frames of original intestinal images are obtained.It should be noted that the number and radius of the circular areas can be set according to actual needs. The confidence threshold is obtained based on the test results of the test set. It is a completely empirical value rather than a calculated value. After testing, this screening value can only retain the key folds of the inner ring image and can greatly retain the features of the outer ring.
[0038] In one or more embodiments, step S20 of performing polar coordinate expansion on each original intestinal image of the original intestinal image set to obtain a rectangular intestinal image and a new feature point set corresponding to the feature point set in the rectangular intestinal image includes the following steps: Step S201, calculating and obtaining the relative rotation angle between two adjacent frames of original intestinal images; Step S202, performing polar coordinate expansion on the original intestinal image according to the relative rotation angle to obtain a rectangular intestinal image; Step S203, performing polar coordinate transformation according to the coordinates and relative rotation angle of each feature point in the feature point set to obtain the polar coordinate value corresponding to each feature point; Step S204, normalizing each polar coordinate value according to the width and height of the rectangular intestinal image to obtain a corresponding new feature point set in the rectangular intestinal image, wherein the new feature points matched between two adjacent rectangular intestinal images are on the same horizontal line.
[0039] Since the capsule endoscope will rotate along the forward direction during the shooting process, the matching point pairs are not distributed on the same horizontal line in the expanded image, which makes the displacement difference between the feature points unable to reflect the actual axial forward distance. Therefore, before polar coordinate expansion, the image needs to be rotated and corrected so that after the image is rotated, all feature points are at the same angle as the matching feature points in the previous frame of the image. By correcting the rotation angle, the self-rotation of the capsule along the axial direction during the forward process can be eliminated to the greatest extent.
[0040] For example, the relative rotation angle of two adjacent frames of original intestinal images is solved by using the least squares angle difference method. The specific optimization problem to be solved is:
[0041]
[0042] in is the center point coordinate of two adjacent frames of original intestinal images, , is the coordinates of the feature points of two adjacent frames of original intestinal images, and the angle obtained by solving the optimization problem is This is the rotation angle of the capsule when taking these two frames of images. By traversing all the images, we can get the relative rotation angle of each original intestinal image relative to the original intestinal image of the previous frame in its time sequence, and record them as 。
[0043] Further, in the above steps, a sequence of relative rotation angles of each original intestinal image with respect to the original intestinal image of the previous frame in time series is obtained , and then the center point of the image is selected as the origin for polar coordinate expansion. Among them, the maximum radius of the expansion is , and the positive half-axis of the initial axis is selected as the ray corresponding to the radius where the darkest point of the first frame of the original intestinal image is located. Then, the positive half-axis of the axis of each subsequent frame of the image is the positive half-axis of the axis of the previous frame in time series rotated clockwise by the corresponding degrees. According to this method, each frame of the image is successively subjected to polar coordinate expansion. That is, for the th frame of the image, the relative rotation angle of its positive half-axis with respect to the initial image is . By this method, the spin of the capsule endoscope during forward movement can be eliminated, so that the matching point pairs are on the same horizontal line in the expanded image.
[0044] Further, for each original intestinal image, if the clockwise angle between the positive half-axis of the axis and the rightward ray of the center point is , then its feature point
[0045]
[0046]
[0047] Here, needs to be normalized to the range of . Then, through the following formula for normalization processing, the coordinates of the new feature point in the new polar coordinate system for and are obtained:
[0048]
[0049]
[0050] Among them, width represents the width of each frame of the rectangular intestinal image, and height represents the height of each frame of the rectangular intestinal image.
[0051] In one or more embodiments, the new feature point sets of two adjacent frames of rectangular intestinal images obtained by solving according to the above steps are respectively represented as and To represent the step distance of feature points on two images, the difference in the radius coordinates of the newly matched feature points between two adjacent frames of rectangular intestinal images is used as the displacement difference of the newly matched feature points between two adjacent frames of rectangular intestinal images, denoted as Here, it is assumed that there are groups of matching point pairs.
[0052] Furthermore, according to the width of the rectangular intestinal image and the displacement differences of each newly matched feature point, the step distance points of the subsequent frame of rectangular intestinal image in the adjacent previous frame of rectangular intestinal image can be obtained, which can be expressed as follows:
[0053]
[0054] Here, it is assumed that there are step distance points. Since what needs to be recorded is the step distance, the negative displacement differences are filtered out and re-numbered. Therefore .
[0055] Next, the polar coordinates of each step distance point are transformed to obtain the marked points corresponding to each step distance point in the previous frame of the original intestinal image, that is, from the polar coordinate system to the Cartesian coordinate system. Exemplarily, for the rd step distance point, assuming its coordinates in the new polar coordinate system are , first, it is converted to the coordinates on the corresponding matrix intestinal image, that is, the radius and the angle:
[0056]
[0057]
[0058] Furthermore, calculate its abscissa and ordinate on the original intestinal image, which are respectively:
[0059]
[0060]
[0061] Therefore, the marked points corresponding to the step distance points on the original intestinal image can be obtained .
[0062] Furthermore, according to the largest inscribed circle fitted by each marked point, the regional image of the previous frame of the original intestinal image outside the largest inscribed circle is the step regional image between two adjacent frames of the original intestinal images. Exemplarily, the fitting method is the traversal method. For each point on the graph, record the largest eccentric circle with a radius from 0 to the maximum width that does not contain all displacement points, and judge whether a certain marked point is at the center of the circle and the radius is The method inside the eccentric circle is to compare and the magnitude relationship between them. By traversing all points, the largest inscribed circle is obtained.
[0063] In other embodiments, during the fitting process of the eccentric circle, due to the existence of some singular points, the inscribed eccentric circle may be too small, resulting in a relatively large step distance, which in turn causes the seam to be abrupt and the splicing quality to decrease. Therefore, as an alternative solution, the minimum least - square distance can be used to fit an eccentric circle, and then a relatively small step distance can be solved. This can reduce the influence brought by noise to a certain extent. However, in the case of good matching, it may cause information loss.
[0064] When performing polar coordinate expansion after obtaining the stepped - area image, during the process of polar coordinate expansion, regions with different radii will be expanded into the same height, resulting in different stretching ratios on the left and right sides of each small part of the step distance. To solve this problem, by calculating the aspect ratio that each ring should originally have, it is mapped to columns with different widths, the same height, and the same proportional scale to restore the true scale - ratio relationship between different pixel columns in the cropped step distance.
[0065] Exemplarily, assuming that the maximum width of the obtained stepped - area image is , then pixels between the radius of and (that is, pixels with true circumferences between and ) are all compressed (stretched) into pixels with a width of and a height of height after expansion, resulting in the phenomenon of non - proportional compression (stretching) and causing image distortion. Exemplarily, the remap function in the OpenCV library takes the compression ratio at the maximum radius of (that is, pixels of are compressed (stretched) into pixels of ) as a benchmark, and stretches the pixels at other radii by the method of stretching the width to expand them into pixels with the same ratio (that is, wider) as at the maximum radius of . By this method, it can be ensured that the stepped - area image will not be distorted after polar coordinate expansion, which is more conducive to the clinical doctor's film reading and diagnosis process.
[0066] Finally, after sequentially splicing each newly added image segment, a panoramic intestinal image with some black regions is obtained. For the sake of image aesthetics and facilitating doctors' observation, in this embodiment, the Navier-Stokes filling method is used to repair the image. This repair method fills the black regions by simulating the diffusion process of the image content, is suitable for repairing regions with smooth edges and relatively regular shapes, and has a high edge retention rate.
[0067] In one or more embodiments, as Figure 2 shown, the panoramic intestinal image stitching device for a capsule endoscope includes a feature matching module 100, an image unfolding module 200, a step distance calculation module 300, a step region cutting module 400, an image stretching and unfolding module 500, and an image stitching module 600. The feature matching module 100 is used to perform feature matching on the original intestinal image set collected by the capsule endoscope to obtain the feature point set of each image in the original intestinal image set; the image unfolding module 200 is used to perform polar coordinate unfolding on each original intestinal image in the original intestinal image set to obtain a rectangular intestinal image and the new feature point set corresponding to the feature point set in the rectangular intestinal image; the step distance calculation module 300 is used to calculate the displacement difference of each new feature point according to the coordinates of the new feature points in the new feature point sets of two adjacent frames of rectangular intestinal images; the step region cutting module 400 is used to cut the step region image between two adjacent original intestinal images according to the displacement difference of each new feature point; the image stretching and unfolding module 500 is used to perform polar coordinate unfolding on the step region image as the newly added image segment between two adjacent frames of rectangular intestinal images; the image stitching module 600 is used to sequentially stitch the obtained newly added image segments to obtain a panoramic intestinal image.
[0068] Exemplarily, the feature matching module 100 includes an image preprocessing unit, a feature extraction unit, a feature matching unit, and a feature screening unit. The image preprocessing unit is used to remove noise and redundant images from the original intestinal image set to obtain the preprocessed original intestinal image; the feature extraction unit is used to extract the initial feature point set of each image in the preprocessed original intestinal image based on the SuperPoint feature extraction model; the feature matching unit is used to perform feature matching on the initial feature point sets of two adjacent frames of original intestinal images in sequence based on the SuperGlue feature matching model to obtain the matching feature point set of each original intestinal image; the feature screening unit is used to divide each original intestinal image into several regions, screen the matching feature points from each region according to different confidence thresholds, and the screened matching feature points constitute the feature point set of each original intestinal image.
[0069] Further, the feature screening unit is also used to divide each original intestinal image into a central circle area and several concentric ring areas with gradually increasing radii. In the order from the inside to the outside, the confidence threshold of each area decreases, and matching feature points greater than the corresponding confidence threshold are screened out from each area.
[0070] Further, the image unfolding module 200 is also used to: calculate the relative rotation angle between two adjacent frames of original intestinal images; perform polar coordinate unfolding on the original intestinal images according to the relative rotation angle to obtain rectangular intestinal images; perform polar coordinate transformation on the coordinates of each feature point in the feature point set and the relative rotation angle to obtain the polar coordinate values corresponding to each feature point; perform normalization processing on each polar coordinate value according to the width and height of the rectangular intestinal image to obtain a new feature point set corresponding to the rectangular intestinal image, where the new feature points that match between two adjacent frames of rectangular intestinal images are on the same horizontal line.
[0071] Further, the step distance calculation module 300 is used to take the difference between the radius coordinates of the new feature points that match between two adjacent frames of rectangular intestinal images as the displacement difference of the new feature points between two adjacent frames of rectangular intestinal images.
[0072] Further, the step area cropping module 400 is also used to: obtain the step distance points of the latter frame of rectangular intestinal image in the adjacent former frame of rectangular intestinal image according to the width of the rectangular intestinal image and the displacement difference of each new feature point; perform transformation on the polar coordinates of each step distance point to obtain the marked points corresponding to each step distance point in the former frame of original intestinal image; according to the largest inscribed circle fitted by each marked point, the area image of the former frame of original intestinal image outside the largest inscribed circle is the step area image between two adjacent frames of original intestinal images.
[0073] Further, the image scaling and unfolding module 500 is also used to, during the process of performing polar coordinate unfolding on the step area image, perform scaling processing on the pixels at different radii with different ratios using the remap function to obtain a rectangular image segment as the new image segment between two adjacent frames of rectangular intestinal images.
[0074] Among them, the more detailed working processes of each module of the capsule endoscope intestinal panoramic image stitching device can refer to the relevant descriptions of the capsule endoscope intestinal panoramic image stitching method in the previous embodiment, which will not be elaborated here.
[0075] In one or more embodiments, a computer-readable storage medium stores a capsule endoscope intestinal panoramic image stitching program, and when the capsule endoscope intestinal panoramic image stitching program is executed by a processor, it implements the capsule endoscope intestinal panoramic image stitching method of Embodiment 1.
[0076] This embodiment also discloses a computer device. At the hardware level, such as Figure 3As shown, the computer device includes a processor 12, an internal bus 13, a network interface 14, and a computer-readable storage medium 11. The processor 12 reads the corresponding computer program from the computer-readable storage medium and then runs it, forming a request processing device at the logical level. Of course, in addition to the software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logical unit, and can also be hardware or a logic device. A capsule endoscope intestinal panoramic image stitching program is stored on the computer-readable storage medium 11. When the capsule endoscope intestinal panoramic image stitching program is executed by the processor, the above-mentioned capsule endoscope intestinal panoramic image stitching method is implemented.
[0077] Computer-readable storage media include permanent and non-permanent, removable and non-removable media and can be implemented by any method or technology for information storage. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer-readable storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette tapes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device.
[0078] The specific implementation manners of the present application have been described in detail above. Although some embodiments have been shown and described, those skilled in the art should understand that these embodiments can be modified and improved without departing from the principles and spirit of the present application as defined by the claims and their equivalents, and these modifications and improvements should also be within the protection scope of the present application.
Claims
1. A method for stitching panoramic images of the intestinal tract of a capsule endoscope, characterized in that, The method includes: Performing feature matching on the original intestinal image set collected by the capsule endoscope to obtain the feature point set of each image in the original intestinal image set; Performing polar coordinate expansion on each original intestinal image in the original intestinal image set to obtain a rectangular intestinal image and a new feature point set corresponding to the feature point set in the rectangular intestinal image; Calculating the displacement difference of each new feature point based on the coordinates of the new feature points in the new feature point sets of two adjacent frames of rectangular intestinal images; Cropping the stepped area image between two adjacent frames of original intestinal images according to the displacement difference of each new feature point; Performing polar coordinate expansion on the stepped area image as the new image segment between two adjacent frames of rectangular intestinal images; Sequentially splicing the obtained new image segments to obtain an intestinal panoramic image.
2. The method for stitching panoramic images of the intestinal tract of a capsule endoscope according to claim 1, wherein The method for performing feature matching on the original intestinal image set collected by the capsule endoscope includes: Removing noise and redundant images from the original intestinal image set to obtain the preprocessed original intestinal images; Extracting the initial feature point set of each image of the preprocessed original intestinal images based on the SuperPoint feature extraction model; Performing feature matching on the initial feature point sets of two adjacent frames of original intestinal images in sequence based on the SuperGlue feature matching model to obtain the matching feature point set of each original intestinal image; Dividing each original intestinal image into several regions, screening matching feature points from each region according to different confidence thresholds, and the screened matching feature points constitute the feature point set of each original intestinal image.
3. The method for stitching panoramic images of the intestinal tract of a capsule endoscope according to claim 2, wherein, The method for dividing each original intestinal image into several regions and screening matching feature points from each region according to different confidence thresholds includes: Dividing each original intestinal image into a central circle region and several concentric ring regions with gradually increasing radii, and the confidence thresholds of each region decrease in order from the inside to the outside; Screening out the matching feature points greater than the corresponding confidence thresholds from each region.
4. The method for stitching panoramic images of the intestinal tract of a capsule endoscope according to claim 1, wherein The step of performing polar coordinate expansion on each original intestinal image in the original intestinal image set to obtain a rectangular intestinal image and a new feature point set corresponding to the feature point set in the rectangular intestinal image includes: Calculating the relative rotation angle between two adjacent frames of original intestinal images; Performing polar coordinate expansion on the original intestinal image according to the relative rotation angle to obtain a rectangular intestinal image; Performing polar coordinate transformation on the coordinates of each feature point in the feature point set according to the relative rotation angle to obtain the polar coordinate values corresponding to each feature point; Normalizing each polar coordinate value according to the width and height of the rectangular intestinal image to obtain the new feature point set corresponding to the rectangular intestinal image, where the matching new feature points between two adjacent frames of rectangular intestinal images are on the same horizontal line.
5. The method for stitching panoramic images of the intestinal tract of a capsule endoscope according to claim 4, wherein The step of calculating the displacement difference of each new feature point based on the coordinates of the new feature points in the new feature point sets of two adjacent frames of rectangular intestinal images includes: Taking the difference between the radius coordinates of the matching new feature points between two adjacent frames of rectangular intestinal images as the displacement difference of the new feature point between two adjacent frames of rectangular intestinal images.
6. The method for stitching panoramic images of the intestinal tract of a capsule endoscope according to claim 4, wherein Cropping the stepped area image between two adjacent frames of original intestinal images according to the displacement differences of each new feature point includes: Obtaining the stepped distance points of the subsequent frame of rectangular intestinal image in each adjacent previous frame of rectangular intestinal image according to the width of the rectangular intestinal image and the displacement differences of each new feature point; Performing polar coordinate transformation on the polar coordinates of each of the stepped distance points to obtain the corresponding marked points of each of the stepped distance points in the previous frame of original intestinal image; According to the maximum inscribed circle fitted by each marked point, the area image of the previous frame of original intestinal image outside the maximum inscribed circle is the stepped area image between two adjacent frames of original intestinal images.
7. The method for stitching panoramic images of the intestinal tract of a capsule endoscope according to claim 4, wherein, The method of performing polar coordinate expansion on the stepped area image as the new image segment between two adjacent frames of rectangular intestinal images includes: During the process of performing polar coordinate expansion on the stepped area image, the remap function is used to perform stretching processing on pixels at different radii in different proportions to obtain a rectangular image segment as the new image segment between two adjacent frames of rectangular intestinal images.
8. An intestinal panoramic image stitching device for a capsule endoscope, characterized in that The device includes: A feature matching module for performing feature matching on the set of original intestinal images collected by the capsule endoscope to obtain the set of feature points of each image in the set of original intestinal images; An image expansion module for performing polar coordinate expansion on each original intestinal image in the set of original intestinal images to obtain a rectangular intestinal image and the corresponding set of new feature points of the set of feature points in the rectangular intestinal image; A stepped distance calculation module for calculating the displacement differences of each new feature point according to the coordinates of each new feature point in the set of new feature points of two adjacent frames of rectangular intestinal images; A stepped area cropping module for cropping the stepped area image between two adjacent frames of original intestinal images according to the displacement differences of each new feature point; An image stretching and expanding module for performing polar coordinate expansion on the stepped area image as the new image segment between two adjacent frames of rectangular intestinal images; An image splicing module for sequentially splicing the obtained new image segments to obtain a panoramic intestinal image.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a capsule endoscope intestinal panoramic image splicing program, and when the capsule endoscope intestinal panoramic image splicing program is executed by a processor, it implements the capsule endoscope intestinal panoramic image splicing method according to any one of claims 1 to 7.
10. A computer device, characterized in that, The computer device includes a computer-readable storage medium, a processor, and a capsule endoscope intestinal panoramic image splicing program stored in the computer-readable storage medium. When the capsule endoscope intestinal panoramic image splicing program is executed by the processor, it implements the capsule endoscope intestinal panoramic image splicing method according to any one of claims 1 to 7.
Citation Information
Patent Citations
Capsule endoscope system
CN106618454A
Capsule endoscopy intestinal tract image splicing method and device, storage medium and equipment
CN114972144A
Splicing method and device of endoscopic images and computer equipment
CN116563118A
System and method of scanning a body cavity using a multiple viewing elements endoscope
WO2015168066A1
Three-dimensional panoramic recognition and positioning method for digestive tract, storage medium, and computer device
WO2023246441A1
Cited By
Method and device for screening redundant images of digestive tract capsule endoscope, storage medium and computer equipment
CN120931660A
Small intestine capsule endoscopy auxiliary method and system based on image processing
CN121169909A
Image processing-based small intestine capsule endoscopy auxiliary method and system
CN121169909B
