Capsule endoscope intestinal panoramic image stitching method, device, computer-readable storage medium and computer equipment

Abstract

The present application provides a capsule endoscope intestinal panoramic image stitching method, device, computer-readable storage medium and computer equipment. The method includes: performing feature matching on the original intestinal image set collected by the capsule endoscope to obtain a feature point set of each image in the original intestinal image set; performing polar coordinate expansion on each original intestinal image to obtain a rectangular intestinal image and a new feature point set corresponding to the feature point set in the rectangular intestinal image; calculating the displacement difference of each new feature point based on the coordinates of each new feature point in the new feature point set of two adjacent rectangular intestinal images; cropping the step area image between the two adjacent original intestinal images based on the displacement difference; performing polar coordinate expansion on the step area image to obtain newly added image segments; and stitching the newly added image segments in sequence to obtain a panoramic intestinal image. This method calculates a more accurate real step distance, which is conducive to improving the smoothness and continuity of the image stitching.

Description

Technical Field

[0001] The present application belongs to the technical field of capsule endoscopy, and specifically relates to a capsule endoscope intestinal panoramic image stitching method, stitching device, computer-readable storage medium and computer equipment. Background Art

[0002] Gastrointestinal diseases are common and prevalent health issues worldwide, encompassing a wide range of pathological conditions, including gastritis, gastric ulcers, colitis, Crohn's disease, gastrointestinal bleeding, and intestinal tumors. These diseases have a high incidence rate, and their symptoms are complex and diverse, often involving chronic inflammation, bleeding, functional disorders, and even malignant lesions, posing a serious threat to patients' health and quality of life. With the accelerated aging of the global population and changes in dietary habits and lifestyles, the incidence of gastrointestinal diseases continues to rise, becoming a major public health issue. Furthermore, the diagnosis and treatment of gastrointestinal diseases still face multiple challenges. Early-stage lesions often present atypical symptoms and are easily overlooked, while late-stage lesions may have already progressed to the stage of severe complications, increasing the complexity and risk of treatment. Therefore, achieving early screening, accurate diagnosis, and personalized treatment for gastrointestinal diseases has become a key research direction in the medical field.

[0003] Medical endoscopes play an important role in diagnosing gastrointestinal diseases. Traditional endoscopic examinations usually require doctors to insert a long cable into the patient's body to image key areas and provide diagnostic and treatment plans. However, due to its invasive nature, traditional intestinal endoscopes require a certain amount of mechanical force to advance the endoscope, which can cause significant pain and discomfort to patients, especially those with intestinal sensitivities. In addition, the use of such endoscopes also carries the risk of potential complications such as infection, perforation, and tearing. Moreover, due to the long length of the intestine (usually 4-6 meters) and its curly shape, it is more difficult to advance traditional endoscopes in the intestine. Compared to the stomach and colon, the intestine is more difficult to examine. In addition, the imaging resolution of traditional intestinal endoscopes is limited and may not be able to identify very small or early lesions. In addition, the lack of intelligent auxiliary functions such as image enhancement and lesion recognition increases the difficulty of diagnosis for doctors.

[0004] A capsule endoscope is a noninvasive device used to examine the digestive tract. It's shaped like a capsule and contains a high-definition camera, light source, image sensor, and wireless transmission device. Patients simply swallow the capsule endoscope, which moves downward with the natural peristalsis of the digestive tract, capturing and recording images of the tract. These images are wirelessly transmitted to an external receiver for examination and analysis by the physician. Capsule endoscopes are typically used in the stomach, intestines, and large intestine, offering a comfortable, convenient, and painless endoscopic examination method. A capsule enteroscope, specifically designed for intestinal examination, is suitable for detailed intestinal examination and lesion detection. Compared to traditional endoscopes, it is simpler to use, covers a wider range, and causes less damage to the intestine. Furthermore, enteroscopes are simple to use, eliminating the need for complex endoscopic procedures and requiring no hospital stay for observation, making the examination more flexible and convenient. Most importantly, modern capsule enteroscopes can incorporate various advanced algorithms, such as artificial intelligence, to intelligently analyze images, helping physicians more accurately identify lesions and improving diagnostic efficiency and accuracy.

[0005] For the tens of thousands of intestinal images captured by capsule endoscopy, image stitching can be used to construct a panoramic intestinal image. This algorithm stitches sequential images of the intestine into a complete image or video, helping doctors better visualize the internal structure and lesions. Current intestinal stitching algorithms primarily utilize feature point recognition and matching algorithms, selecting a uniform circular ring as the stepping distance between two consecutive frames, and stitching the cropped rectangular stepping distances using polar coordinate expansion. Stitching panoramic intestinal images visually displays the entire intestinal landscape, helping doctors more comprehensively observe intestinal structure and potential lesions. It also fully displays continuous regions within the intestine, reducing the risk of misdiagnosis and missed diagnoses caused by ignoring partial frames. Furthermore, intestinal stitching can help doctors more easily locate specific lesions and record their exact location (e.g., distance from the pylorus or ileocecal valve), providing a reference for further diagnosis or treatment (such as endoscopic intervention).

[0006] However, the current panoramic image stitching method is too idealistic in determining the stepping area. Since the position of the capsule is not fixed during its movement in the intestine, the stepping distance between two consecutive frames of images is not a uniform annular area. If selected according to the existing method, the seam will be too abrupt and contain too much redundant information. Summary of the Invention

[0007] The technical problem solved by this application is: how to obtain a more accurate step distance between two consecutive frames of images to improve the smoothness and continuity of the image splicing.

[0008] The present application provides a method for stitching panoramic images of the intestine through a capsule endoscope, the method comprising:

[0009] Performing feature matching on an original intestinal image set collected by a capsule endoscope to obtain a feature point set of each image in the original intestinal image set;

[0010] Performing polar coordinate expansion on each original intestinal image in the original intestinal image set to obtain a rectangular intestinal image and a new feature point set corresponding to the feature point set in the rectangular intestinal image;

[0011] The displacement difference of each new feature point is calculated based on the coordinates of each new feature point in the new feature point set of two adjacent rectangular intestinal images;

[0012] The step area image between two adjacent frames of original intestinal images is obtained by cropping according to the displacement difference of each new feature point;

[0013] Expanding the stepping region image in polar coordinates to serve as a new image segment between two adjacent rectangular intestinal images;

[0014] The newly added image segments are stitched together in sequence to obtain a panoramic image of the intestine.

[0015] Optionally, the method for performing feature matching on the original intestinal image set collected by the capsule endoscope includes:

[0016] removing noise and redundant images from the original intestinal image set to obtain preprocessed original intestinal images;

[0017] The initial feature point set of each image of the preprocessed original intestinal image is extracted based on the SuperPoint feature extraction model;

[0018] Based on the SuperGlue feature matching model, the initial feature point sets of two adjacent frames of original intestinal images are matched in sequence to obtain the matching feature point set of each original intestinal image;

[0019] Each original intestinal image is divided into several regions, and matching feature points are screened from each region according to different confidence thresholds. The screened matching feature points constitute the feature point set of each original intestinal image.

[0020] Optionally, each original intestinal image is divided into several regions, and the method of screening matching feature points from each region according to different confidence thresholds includes:

[0021] Each original intestinal image is divided into a central circle area and several concentric ring areas with increasing radius. From the inside to the outside, the confidence threshold of each area decreases;

[0022] Filter out matching feature points with a value greater than the corresponding confidence threshold from each region.

[0023] Optionally, performing polar coordinate expansion on each original intestinal image in the original intestinal image set to obtain a rectangular intestinal image and a new feature point set corresponding to the feature point set in the rectangular intestinal image includes:

[0024] Calculate the relative rotation angle between two adjacent frames of original intestinal images;

[0025] The original intestinal image is expanded in polar coordinates according to the relative rotation angle to obtain a rectangular intestinal image;

[0026] Performing polar coordinate transformation according to the coordinates and relative rotation angle of each feature point in the feature point set to obtain the polar coordinate value corresponding to each feature point;

[0027] Each polar coordinate value is normalized according to the width and height of the rectangular intestinal image to obtain a corresponding new feature point set in the rectangular intestinal image, wherein the new feature points matched between two adjacent rectangular intestinal images are on the same horizontal line.

[0028] Optionally, the step of calculating the displacement difference of each new feature point based on the coordinates of each new feature point in the new feature point set of two adjacent rectangular intestinal images includes:

[0029] The difference in radius coordinates of the new feature points matched between two adjacent rectangular intestinal images is used as the displacement difference of the new feature points between the two adjacent rectangular intestinal images.

[0030] Optionally, the stepping region image between two adjacent frames of original intestinal images is obtained by cropping according to the displacement difference of each new feature point, including:

[0031] According to the width of the rectangular intestinal image and the displacement difference of each new feature point, each step distance point of the next rectangular intestinal image in the adjacent previous rectangular intestinal image is obtained;

[0032] Transforming the polar coordinates of each step distance point to obtain a marking point corresponding to each step distance point in the previous frame of the original intestinal image;

[0033] According to the maximum inscribed circle fitted by each marking point, the area image of the previous frame of original intestinal image outside the maximum inscribed circle is the step area image between two adjacent frames of original intestinal image.

[0034] Optionally, the method of performing polar coordinate expansion on the stepping region image to serve as a new image segment between two adjacent rectangular intestinal images includes:

[0035] In the process of polar coordinate expansion of the stepping region image, the remap function is used to perform scaling processing on pixels at different radii at different ratios to obtain rectangular image segments, which are used as new image segments between two adjacent rectangular intestinal images.

[0036] The present application also discloses a capsule endoscope intestinal panoramic image stitching device, the device comprising:

[0037] A feature matching module is used to perform feature matching on the original intestinal image set collected by the capsule endoscope to obtain a feature point set of each image in the original intestinal image set;

[0038] an image expansion module, configured to perform polar coordinate expansion on each original intestinal image in the original intestinal image set to obtain a rectangular intestinal image and a new feature point set corresponding to the feature point set in the rectangular intestinal image;

[0039] A step distance calculation module is used to calculate the displacement difference of each new feature point based on the coordinates of each new feature point in the new feature point set of two adjacent rectangular intestinal images;

[0040] The step region cropping module is used to crop the step region image between two adjacent frames of original intestinal images according to the displacement difference of each new feature point;

[0041] An image expansion module is used to expand the stepping region image in polar coordinates as a new image segment between two adjacent rectangular intestinal images;

[0042] The image stitching module is used to stitch the newly added image segments in sequence to obtain a panoramic image of the intestine.

[0043] The present application also discloses a computer-readable storage medium, which stores a capsule endoscope intestinal panoramic image stitching program. When the capsule endoscope intestinal panoramic image stitching program is executed by a processor, the above-mentioned capsule endoscope intestinal panoramic image stitching method is implemented.

[0044] The present application also discloses a computer device, which includes a computer-readable storage medium, a processor, and a capsule endoscope intestinal panoramic image stitching program stored in the computer-readable storage medium. When the capsule endoscope intestinal panoramic image stitching program is executed by the processor, the above-mentioned capsule endoscope intestinal panoramic image stitching method is implemented.

[0045] The present application provides a capsule endoscope intestinal panoramic image stitching method, device, computer-readable storage medium, and computer equipment, which have the following technical effects:

[0046] The displacement difference is calculated by the coordinates of each feature point, and then the original image is returned to the cropped step area image based on the displacement difference. This method can calculate a more accurate real step distance, which is beneficial to improving the smoothness and continuity of the image splicing. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 is a flowchart of a capsule endoscope intestinal panoramic image stitching method according to one or more embodiments;

[0048] Figure 2 is a principle block diagram of a capsule endoscope intestinal panoramic image stitching device according to one or more embodiments;

[0049] Figure 3 is a functional block diagram of a computer device according to one or more embodiments. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of this application more clearly understood, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0051] Before describing the various embodiments of the present application in detail, the technical concept of the present application is first briefly described: Currently, when performing capsule endoscope intestinal image stitching, due to the factors of the change in posture of the capsule during its movement, the step distance between two frames of continuous images is not a uniform annular area. The existing stitching method will cause problems such as abruptness and redundancy in the splicing. To this end, the present application provides a capsule endoscope intestinal panoramic image stitching method, the key improvement of which is to unfold the original intestinal image in a polar coordinate system, calculate the displacement difference through the coordinates of each feature point, and then return to the original image to crop the step area image based on the displacement difference. This method can calculate a more accurate real step distance, which is conducive to improving the smoothness and continuity of the image splicing. The specific principles of the capsule endoscope intestinal panoramic image stitching method of the present application are described below in combination with more embodiments.

[0052] Specifically, if Figure 1 As shown, the capsule endoscope intestinal panoramic image stitching method of this embodiment includes the following steps:

[0053] Step S10: performing feature matching on the original intestinal image set collected by the capsule endoscope to obtain a feature point set of each image in the original intestinal image set.

[0054] Step S20: performing polar coordinate expansion on each original intestinal image in the original intestinal image set to obtain a rectangular intestinal image and a new feature point set corresponding to the feature point set in the rectangular intestinal image.

[0055] Step S30: Calculating the displacement difference of each new feature point based on the coordinates of each new feature point in the new feature point set of two adjacent rectangular intestinal images.

[0056] Step S40: cropping the stepping region image between two adjacent frames of original intestinal images according to the displacement difference of each new feature point.

[0057] Step S50: performing polar coordinate expansion on the stepping region image to serve as a new image segment between two adjacent rectangular intestinal images.

[0058] Step S60: stitching the newly added image segments in sequence to obtain a panoramic image of the intestine.

[0059] In one or more embodiments, prior to step S10, a set of original intestinal images is collected by a capsule endoscope. Specifically, after the patient swallows the capsule enters the intestinal environment along the digestive tract, it moves along the human intestine under the action of intestinal peristalsis. At this time, the built-in lens, assisted by the fill light LED, captures a set of human intestinal images in a certain time sequence. The images are stored in a data recorder worn by the patient. After the capsule completes the entire capturing process and is expelled from the body, the data recorder can be removed and the captured images can be imported into a computer to obtain the original intestinal image set.

[0060] Furthermore, when performing feature matching, the original intestinal image set needs to be denoised and redundant images removed to obtain the preprocessed original intestinal images. Specifically, the original intestinal images are first converted to HLS space, and histogram equalization and 5×5 average filtering are performed on the L channel to remove noise from the image. Because intestinal movement is non-uniform, the images captured by the data recorder will contain a large number of redundant images in areas with slower intestinal peristalsis. Therefore, it is necessary to calculate the histogram of the image and compare the histogram similarity of two adjacent images. Images with a similarity of more than 0.9 to the previous image are removed. After that, all images are sorted in reverse order to obtain the preprocessed original intestinal images.

[0061] After image preprocessing, in order to reflect the step distance between two adjacent frames of the capsule endoscope, feature matching is required between the two images. The displacement changes of feature points are used to reflect the step distance of the nearby area between the two frames. Traditional feature point recognition methods such as SIFT (Scale Invariant Feature Transform) and SURF (Speeded Robust Features) have limited matching accuracy in the intestinal environment, and the generated feature points are often too dense or unevenly distributed. Traditional matching methods such as FLANN (Fast Approximate Nearest Neighbor) and Brute-Force Matching each have certain limitations. FLANN uses an approximate search, which can sacrifice matching accuracy to a certain extent. Brute-Force Matching has a high time complexity and, when the number of feature points is large, the calculation time is extremely slow, making it difficult to meet real-time requirements.

[0062] Based on the limitations of traditional methods, this embodiment first extracts the initial feature point set of each preprocessed original intestinal image based on the SuperPoint feature extraction model, then performs feature matching on the initial feature point sets of two adjacent frames of original intestinal images based on the SuperGlue feature matching model to obtain the matching feature point set of each original intestinal image. Finally, each original intestinal image is divided into several regions, and matching feature points are screened from each region according to different confidence thresholds. The screened matching feature points constitute the feature point set of each original intestinal image.

[0063] Specifically, SuperPoint is an end-to-end deep learning model for image keypoint detection and descriptor generation. It uses a convolutional neural network (CNN) to extract image features. With its end-to-end learning, high robustness, and real-time performance, SuperPoint is particularly suitable for feature point extraction and matching in capsule endoscopy images. Furthermore, SuperPoint can be seamlessly integrated with SuperGlue to further optimize feature point matching between image pairs. The SuperGlue method combines a graph neural network and an attention mechanism to achieve accurate and robust feature point matching. The SuperGlue method can better understand global and local information in images, significantly improving matching accuracy and robustness, and can produce a larger and more accurate set of feature point matches compared to traditional methods. For example, the SuperPoint model used here can be directly downloaded and used from the pre-trained model SuperPoint_v1.pth on GitHub without modification or adaptation, resulting in highly robust feature point extraction.

[0064] The basic process of feature matching the initial feature point sets of two adjacent frames of original intestinal images based on the SuperGlue feature matching model includes: (1) Extraction of input feature points: Obtain feature points based on the SuperPoint network. (2) Construction of feature point graph: Represent feature points as graph nodes and input feature point graph. (3) Graph neural network processing: Obtain the global relationship between feature points through iterative updates. (4) Attention mechanism optimizes matching, finds possible matching points and generates a matching matrix. (5) Preliminary screening process: Set the initial threshold to filter out some points with very low confidence. (6) Output the final matching result.

[0065] For example, the SuperGlue used here can directly adopt the pre-trained model SuperGlue_indoor.pth on the github website, which can be directly downloaded and used without modification or adaptation. The model can be used to extract matching feature point pairs with strong robustness.

[0066] In other embodiments, other feature matching models can be used instead of the SuperGlue feature matching model. The Transformer-based feature point matching method LoFTR can directly perform dense feature matching between image pairs, can handle image pairs with low texture areas and scale changes, and has higher matching accuracy and robustness in complex scenes. In addition, the deep feature-based local descriptor method D2-Net, this end-to-end learning feature detection and description method uses a deep neural network to directly extract robust feature points and descriptors from images.

[0067] Furthermore, after obtaining the set of matching feature points for each original intestinal image, the feature points are screened by confidence. However, due to the particularity of the intestinal environment, some good feature points may be filtered out due to low confidence, which in turn leads to incorrect step distances in some areas, and ultimately results in a large amount of information missing in the spliced image. To this end, the method of screening matching feature points in this embodiment includes: dividing each original intestinal image into a central circle area and a number of concentric ring areas with increasing radii, and decreasing the confidence threshold of each area from the inside to the outside; and screening matching feature points with a confidence threshold greater than the corresponding confidence threshold from each area.

[0068] For example, for any two consecutive frames of original intestinal images, the matching feature point sets of the two original intestinal images are respectively recorded as and , and the confidence of the matched feature point pair is recorded as The circular ring is then used to partition the feature points. The confidence level is output by the SuperGlue model mentioned above, which represents the probability of two points matching. First, the maximum radius of the effective area of the original intestinal image from the center of the image is calculated. Then, set up three layers of circular areas from the inside to the outside, with the radius from the inside to the outside respectively , the image is divided into three parts by the ring, and the central area is the circular area. For the outer ring area ( arrive interval), set a lower confidence threshold of 0.3 to retain more feature points. For the inner ring area (where the shooting distance is far and the deformation is large, the step distance will have a large error) arrive interval), set a higher confidence threshold of 0.7, and retain a small number of excellent feature points. Through this step, the filtered matching feature points are obtained and re-sorted to obtain the feature point set of two adjacent frames of original intestinal images. and It should be noted that the number and radius of the circular areas can be set according to actual needs. The confidence threshold is derived from the test results of the test set and is a purely empirical value rather than a calculated value. After testing, this screening value can only retain the key folds of the inner ring image while largely preserving the features of the outer ring.

[0069] In one or more embodiments, step S20 of performing polar coordinate expansion on each original intestinal image in the original intestinal image set to obtain a rectangular intestinal image and a new feature point set corresponding to the feature point set in the rectangular intestinal image includes the following steps:

[0070] Step S201: Calculate the relative rotation angle between two adjacent frames of original intestinal images;

[0071] Step S202: performing polar coordinate expansion on the original intestinal image according to the relative rotation angle to obtain a rectangular intestinal image;

[0072] Step S203: performing polar coordinate transformation according to the coordinates and relative rotation angle of each feature point in the feature point set to obtain the polar coordinate value corresponding to each feature point;

[0073] Step S204 : normalize each polar coordinate value according to the width and height of the rectangular intestinal image to obtain a corresponding new feature point set in the rectangular intestinal image, wherein the new feature points matched between two adjacent rectangular intestinal images are on the same horizontal line.

[0074] Since the capsule endoscope will rotate along the forward direction during the shooting process, the matching point pairs are not distributed on the same horizontal line in the expanded image. This will make the displacement difference between the feature points unable to reflect the actual axial forward distance. Therefore, before polar coordinate expansion, the image needs to be corrected for the rotation angle so that after the image is rotated, all feature points are as close as possible to the same angle as the matching feature points in the previous frame. By correcting the rotation angle, the self-rotation of the capsule along the axial direction during the forward process can be eliminated to the greatest extent.

[0075] For example, the relative rotation angle of two adjacent frames of original intestinal images is solved by the least squares angle difference method. The specific optimization problem to be solved is:

[0076]

[0077] in is the center point coordinate of two adjacent frames of original intestinal images, 、 is the coordinate of the feature points of the two adjacent frames of original intestinal images, and the angle obtained by solving the optimization problem is That is, the rotation angle of the capsule when taking these two frames of images. By traversing all the images, we can get the relative rotation angle of each original intestinal image relative to the original intestinal image of the previous frame in its time sequence, and record them as .

[0078] Furthermore, in the above steps, a sequence of relative rotation angles of each original intestinal image relative to the original intestinal image of the previous frame in its time sequence is obtained. , then select the center point of the image Polar coordinate expansion is performed as the origin of polar coordinate expansion, where the maximum radius of the expansion is ,initial The positive half axis of the axis is the ray corresponding to the radius of the darkest point of the original intestinal image in the first frame. The positive half axis of the axis is the previous frame of the time series The positive half axis rotates clockwise Degrees. According to this method, polar coordinate expansion is performed on each frame of image in turn, that is, for the first The positive half axis of the frame image is rotated relative to the initial image by an angle of ,Through this method, the rotation of the capsule endoscope during ,the forward process can be eliminated, and the matching point pairs are on the same horizontal line on the ,unfolded image.

[0079] Furthermore, for each original intestinal image, if The clockwise angle between the positive semi-axis and the ray from the center point to the right is , then its feature points After polar coordinate expansion, there will be a corresponding point on the rectangular intestinal image. Calculate its polar coordinate value, that is, the radius and angle are:

[0080]

[0081]

[0082] Here Need to be normalized to Then, the new feature point is normalized by the following formula to obtain the new feature point in the new polar coordinate system. and Coordinates:

[0083]

[0084]

[0085] Among them, width represents the width of each frame of rectangular intestinal image, and height represents the height of each frame of rectangular intestinal image.

[0086] In one or more embodiments, the new feature point sets of two adjacent rectangular intestinal images obtained by solving the above steps are respectively expressed as and In order to express the step distance of the feature point on the two images, the difference in the radius coordinates of the new feature point matched between the two adjacent rectangular intestinal images is taken as the displacement difference of the new feature point between the two adjacent rectangular intestinal images, which is expressed as , here we assume that Group matching point pairs.

[0087] Furthermore, the step distance points of the next rectangular intestinal image in the adjacent previous rectangular intestinal image are obtained based on the width of the rectangular intestinal image and the displacement difference of each new feature point, which can be expressed as follows:

[0088]

[0089] Here we assume that Step distance points, since what needs to be recorded is the step distance, the negative displacement differences are filtered out and re-labeled, so .

[0090] Then, the polar coordinates of each step distance point are transformed to obtain the corresponding marking points of each step distance point in the previous frame of the original intestinal image, that is, the polar coordinate system is converted to the Cartesian coordinate system. Step distance points, assuming that their coordinates in the new polar coordinate system are , first convert it into coordinates on the corresponding matrix intestinal image, namely radius and angle:

[0091]

[0092]

[0093] Furthermore, the horizontal and vertical coordinates of the original intestinal image are calculated as follows:

[0094]

[0095]

[0096] Therefore, the marking point corresponding to the step distance point in the original intestinal image can be obtained .

[0097] Furthermore, according to the maximum inscribed circle fitted by each mark point, the image of the area outside the maximum inscribed circle of the previous frame of the original intestinal image is the step area image between the two adjacent frames of the original intestinal image. For example, the fitting method is the traversal method. For each point on the graph, the maximum eccentric circle with a radius from 0 to the maximum width that does not contain all displacement points is recorded to determine whether a certain mark point Is the center of the circle , the radius is The method of comparing the interior of the eccentric circle is and The size relationship between them is determined by traversing all the points to find the largest inscribed circle.

[0098] In other embodiments, during the fitting of the eccentric circle, due to the existence of some singular points, the inscribed eccentric circle may be too small, resulting in a large step distance, which in turn causes the seam to become abrupt and the stitching quality to deteriorate. Therefore, as an alternative, the least squares distance can be minimized to fit an eccentric circle, and then a smaller step distance can be solved. This can reduce the impact of noise to a certain extent, but may cause information loss when the match is good.

[0099] When performing polar coordinate expansion after obtaining the step region image, regions of different radii are expanded to the same height during the polar expansion process, resulting in different stretching ratios on the left and right sides of each small step distance. To address this issue, the original aspect ratio of each ring is calculated and mapped to columns of different widths, the same height, and the same scale, thus restoring the true scale relationship between different pixel columns in the cropped step distance.

[0100] For example, it is assumed that the maximum width of the stepping area image is , then we can get the radius arrive The pixels between and The pixels between the width and height are compressed (stretched) into , high height pixels, resulting in unequal compression (stretching) phenomenon, making the image distorted. For example, the remap function in the OpenCV library uses the maximum radius The compression ratio at (and The pixels are compressed (stretched) into The pixels at other radii are expanded to the same size as the maximum radius by stretching the width. This method ensures that the stepping area image is not distorted after polar coordinate expansion, which is more conducive to the doctor's reading and diagnosis process in clinical practice.

[0101] Finally, the newly added image segments are stitched together sequentially to create a panoramic image of the intestine with some black areas. To enhance image aesthetics and facilitate observation by the physician, this example uses the Navier-Stokes infill method to repair the image. This method simulates the diffusion process of image content to fill black areas and is suitable for repairing areas with smooth, regular edges, with high edge preservation.

[0102] In one or more embodiments, Figure 2 As shown, the capsule endoscope intestinal panoramic image stitching device includes a feature matching module 100, an image expansion module 200, a step distance calculation module 300, a step area cropping module 400, an image telescopic expansion module 500 and an image stitching module 600. The feature matching module 100 is used to perform feature matching on the original intestinal image set collected by the capsule endoscope to obtain a feature point set for each image in the original intestinal image set; the image expansion module 200 is used to perform polar coordinate expansion on each original intestinal image in the original intestinal image set to obtain a rectangular intestinal image and a new feature point set corresponding to the feature point set in the rectangular intestinal image; the step distance calculation module 300 is used to calculate the displacement difference of each new feature point based on the coordinates of each new feature point in the new feature point set of two adjacent frames of rectangular intestinal images; the step area cropping module 400 is used to crop the step area image between two adjacent frames of original intestinal images based on the displacement difference of each new feature point; the image telescoping and expansion module 500 is used to perform polar coordinate expansion on the step area image as a new image segment between two adjacent frames of rectangular intestinal images; the image splicing module 600 is used to splice the obtained new image segments in sequence to obtain a panoramic image of the intestine.

[0103] Exemplarily, the feature matching module 100 includes an image preprocessing unit, a feature extraction unit, a feature matching unit, and a feature screening unit. The image preprocessing unit is used to remove noise and redundant images from the original intestinal image set to obtain the preprocessed original intestinal image; the feature extraction unit is used to extract the initial feature point set of each preprocessed original intestinal image based on the SuperPoint feature extraction model; the feature matching unit is used to perform feature matching on the initial feature point sets of two adjacent frames of original intestinal images in sequence based on the SuperGlue feature matching model to obtain the matching feature point set of each original intestinal image; the feature screening unit is used to divide each original intestinal image into several regions, and screen matching feature points from each region according to different confidence thresholds. The screened matching feature points constitute the feature point set of each original intestinal image.

[0104] Furthermore, the feature screening unit is also used to divide each original intestinal image into a central circle area and several concentric ring areas with increasing radii. The confidence threshold of each area decreases from the inside to the outside, and matching feature points with a value greater than the corresponding confidence threshold are screened out from each area.

[0105] Furthermore, the image expansion module 200 is also used to: calculate the relative rotation angle between two adjacent frames of original intestinal images; perform polar coordinate expansion on the original intestinal image according to the relative rotation angle to obtain a rectangular intestinal image; perform polar coordinate transformation according to the coordinates and relative rotation angle of each feature point in the feature point set to obtain the polar coordinate value corresponding to each feature point; normalize each polar coordinate value according to the width and height of the rectangular intestinal image to obtain the corresponding new feature point set in the rectangular intestinal image, wherein the new feature points matched between two adjacent frames of rectangular intestinal images are on the same horizontal line.

[0106] Furthermore, the step distance calculation module 300 is configured to use the difference in radius coordinates of the new feature point matched between two adjacent rectangular intestinal image frames as the displacement difference of the new feature point between the two adjacent rectangular intestinal image frames.

[0107] Furthermore, the step area cropping module 400 is also used to: obtain each step distance point of the subsequent rectangular intestinal image in the adjacent previous rectangular intestinal image based on the width of the rectangular intestinal image and the displacement difference of each new feature point; transform the polar coordinates of each step distance point to obtain the marking point corresponding to each step distance point in the previous original intestinal image; according to the maximum inscribed circle fitted by each marking point, the area image of the previous original intestinal image outside the maximum inscribed circle is the step area image between the two adjacent original intestinal images.

[0108] Furthermore, the image scaling module 500 is also used to use the remap function to perform scaling processing on pixels at different radii at different proportions during the polar coordinate expansion of the step area image, so as to obtain rectangular image segments as new image segments between two adjacent rectangular intestinal images.

[0109] Among them, the more detailed working process of each module of the capsule endoscope intestinal panoramic image stitching device can refer to the relevant description of the capsule endoscope intestinal panoramic image stitching method in the previous embodiment, which will not be repeated here.

[0110] In one or more embodiments, a computer-readable storage medium stores a capsule endoscope intestinal panoramic image stitching program, and when the capsule endoscope intestinal panoramic image stitching program is executed by a processor, the capsule endoscope intestinal panoramic image stitching method of embodiment one is implemented.

[0111] This embodiment also discloses a computer device, at the hardware level, such as Figure 3 As shown, the computer device includes a processor 12, an internal bus 13, a network interface 14, and a computer-readable storage medium 11. The processor 12 reads the corresponding computer program from the computer-readable storage medium and then runs it, forming a request processing device at the logical level. Of course, in addition to software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices. The computer-readable storage medium 11 stores a capsule endoscope intestinal panoramic image stitching program, and when the capsule endoscope intestinal panoramic image stitching program is executed by the processor, the above-mentioned capsule endoscope intestinal panoramic image stitching method is implemented.

[0112] Computer-readable storage media include both permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer-readable storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0113] The above describes in detail the specific implementation methods of the present application. Although some embodiments have been shown and described, those skilled in the art should understand that these embodiments can be modified and improved without departing from the principles and spirit of the present application, the scope of which is defined by the claims and their equivalents. These modifications and improvements should also be within the scope of protection of the present application.

Claims

1. A capsule endoscope intestinal panoramic image stitching method, characterized in that: The method comprises: Performing feature matching on an original intestinal image set collected by a capsule endoscope to obtain a feature point set of each image in the original intestinal image set; Performing polar coordinate expansion on each original intestinal image in the original intestinal image set to obtain a rectangular intestinal image and a new feature point set corresponding to the feature point set in the rectangular intestinal image; The displacement difference of each new feature point is calculated based on the coordinates of each new feature point in the new feature point set of two adjacent rectangular intestinal images; The step area image between two adjacent frames of original intestinal images is obtained by cropping according to the displacement difference of each new feature point; Expanding the stepping region image in polar coordinates to serve as a new image segment between two adjacent rectangular intestinal images; The newly added image segments are stitched together in sequence to obtain a panoramic image of the intestine.

2. The capsule endoscope intestinal panoramic image stitching method according to claim 1, characterized in that: Methods for performing feature matching on the original intestinal image set collected by capsule endoscopy include: removing noise and redundant images from the original intestinal image set to obtain preprocessed original intestinal images; The initial feature point set of each image of the preprocessed original intestinal image is extracted based on the SuperPoint feature extraction model; Based on the SuperGlue feature matching model, the initial feature point sets of two adjacent frames of original intestinal images are matched in sequence to obtain the matching feature point set of each original intestinal image; Each original intestinal image is divided into several regions, and matching feature points are screened from each region according to different confidence thresholds. The screened matching feature points constitute the feature point set of each original intestinal image.

3. The capsule endoscope intestinal panoramic image stitching method according to claim 2, characterized in that: Each original intestinal image is divided into several regions, and the method of screening matching feature points from each region according to different confidence thresholds includes: Each original intestinal image is divided into a central circle area and several concentric ring areas with increasing radius. From the inside to the outside, the confidence threshold of each area decreases; Filter out matching feature points with a value greater than the corresponding confidence threshold from each region.

4. The capsule endoscope intestinal panoramic image stitching method according to claim 1, characterized in that: The polar coordinate expansion is performed on each original intestinal image in the original intestinal image set to obtain a rectangular intestinal image and a new feature point set corresponding to the feature point set in the rectangular intestinal image, including: Calculate the relative rotation angle between two adjacent frames of original intestinal images; The original intestinal image is expanded in polar coordinates according to the relative rotation angle to obtain a rectangular intestinal image; Performing polar coordinate transformation according to the coordinates and relative rotation angle of each feature point in the feature point set to obtain the polar coordinate value corresponding to each feature point; Each polar coordinate value is normalized according to the width and height of the rectangular intestinal image to obtain a corresponding new feature point set in the rectangular intestinal image, wherein the new feature points matched between two adjacent rectangular intestinal images are on the same horizontal line.

5. The capsule endoscope intestinal panoramic image stitching method according to claim 4, characterized in that: The calculation of the displacement difference of each new feature point based on the coordinates of each new feature point in the new feature point set of two adjacent rectangular intestinal images includes: The difference in radius coordinates of the new feature points matched between two adjacent rectangular intestinal images is used as the displacement difference of the new feature points between the two adjacent rectangular intestinal images.

6. The capsule endoscope intestinal panoramic image stitching method according to claim 4, characterized in that: The stepping region image between two adjacent frames of original intestinal images is obtained by cropping according to the displacement difference of each new feature point, including: According to the width of the rectangular intestinal image and the displacement difference of each new feature point, each step distance point of the next rectangular intestinal image in the adjacent previous rectangular intestinal image is obtained; Transforming the polar coordinates of each step distance point to obtain a marking point corresponding to each step distance point in the previous frame of the original intestinal image; According to the maximum inscribed circle fitted by each marking point, the area image of the previous frame of original intestinal image outside the maximum inscribed circle is the step area image between two adjacent frames of original intestinal image.

7. The capsule endoscope intestinal panoramic image stitching method according to claim 4, characterized in that: The method of performing polar coordinate expansion on the stepping region image to use the image segment as a new image segment between two adjacent rectangular intestinal images includes: In the process of polar coordinate expansion of the stepping region image, the remap function is used to perform scaling processing on pixels at different radii at different ratios to obtain rectangular image segments, which are used as new image segments between two adjacent rectangular intestinal images.

8. A capsule endoscope intestinal panoramic image stitching device, characterized in that: The device comprises: A feature matching module is used to perform feature matching on the original intestinal image set collected by the capsule endoscope to obtain a feature point set of each image in the original intestinal image set; an image expansion module, configured to perform polar coordinate expansion on each original intestinal image in the original intestinal image set to obtain a rectangular intestinal image and a new feature point set corresponding to the feature point set in the rectangular intestinal image; A step distance calculation module is used to calculate the displacement difference of each new feature point based on the coordinates of each new feature point in the new feature point set of two adjacent rectangular intestinal images; The step region cropping module is used to crop the step region image between two adjacent frames of original intestinal images according to the displacement difference of each new feature point; An image expansion module is used to expand the stepping region image in polar coordinates as a new image segment between two adjacent rectangular intestinal images; The image stitching module is used to stitch the newly added image segments in sequence to obtain a panoramic image of the intestine.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a capsule endoscope intestinal panoramic image stitching program, and when the capsule endoscope intestinal panoramic image stitching program is executed by a processor, the capsule endoscope intestinal panoramic image stitching method according to any one of claims 1 to 7 is implemented.

10. A computer device, characterized in that: The computer device includes a computer-readable storage medium, a processor, and a capsule endoscope intestinal panoramic image stitching program stored in the computer-readable storage medium. When the capsule endoscope intestinal panoramic image stitching program is executed by the processor, the capsule endoscope intestinal panoramic image stitching method according to any one of claims 1 to 7 is implemented.

Images