Method and device for stitching images of digestive tract magnetically controlled capsule endoscope

Abstract

The present application provides a method and device for stitching images of the digestive tract through magnetically controlled capsule endoscopy. The method selects a candidate lesion region image sequence from an original digestive tract image sequence acquired by a magnetically controlled capsule endoscope; preprocesses the candidate lesion region image sequence to obtain a preprocessed candidate lesion region image sequence; extracts features from the preprocessed candidate lesion region image sequence based on an improved EfficientLoFTR model; further performs feature matching on the preprocessed candidate lesion region image sequence to obtain matching feature point pairs; and performs image registration processing on each image in the preprocessed candidate lesion region image sequence based on the matching feature point pairs. The improved model can improve the accuracy of feature detection and matching, laying a good foundation for subsequent image registration and image fusion. It can also combine global homography transformation and local thin plate spline transformation for image registration to overcome the problem of non-rigid deformation.

Description

Technical Field

[0001] The present application belongs to the technical field of capsule endoscopy, and in particular, relates to a method and a device for splicing images of a magnetically controlled capsule endoscope of the digestive tract. Background Art

[0002] With the advancement of medical imaging technology, magnetically controlled capsule endoscopy (MCCE), a novel gastrointestinal examination tool, has been widely used for the early detection of gastrointestinal ulcers, inflammation, and cancer. Because it is painless, noninvasive, and requires no anesthesia, it has become a powerful supplement to traditional endoscopic examinations. MCCE relies on an external magnetic control system to remotely control the capsule, enabling precise movement, rotation, and retention within the digestive tract. This improves the flexibility and coverage of gastrointestinal examinations and facilitates a more comprehensive observation of potential lesions within the digestive tract.

[0003] In the practical application of magnetically controlled capsule endoscopy, the complex internal environment and variable imaging angles often cause the imaging system to capture a large number of repetitive, fragmented images. This large number of fragmented images limits doctors' ability to accurately identify and judge candidate lesion areas, increasing the difficulty of diagnosis. To address the limitations of fragmented images and achieve a continuous view of the digestive tract, image stitching technology has become key. Image stitching technology combines multiple image segments into a complete image, thereby expanding the field of view and improving image continuity and integrity. By stitching images acquired from different angles and positions, a local panoramic image of the candidate lesion area is generated, allowing doctors to observe within a wider field of view, promoting more accurate gastric lesion detection, and promoting the wider application of non-invasive endoscopic technology in clinical practice. However, due to the irregular shape, sparse texture, and high repetitiveness of the digestive tract, image stitching faces many challenges in feature matching, image registration, and seamless fusion. Therefore, developing an image stitching method for magnetically controlled capsule endoscopy of the stomach has become a key need to improve examination results.

[0004] Image stitching technology is widely used in the medical field. It can break through the limitations of traditional imaging examinations and provide key technical support for the development of precise medical diagnosis and treatment. Based on the perspective of technical characteristics, image stitching technology can be mainly divided into spatial domain and transform domain stitching technology and feature point stitching technology.

[0005] Spatial domain and transform domain stitching techniques primarily rely on global image information for stitching. Spatial domain stitching techniques operate directly in the pixel space of the image, finding overlapping areas between adjacent images and comparing pixel grayscale values ​​or color information in the overlapping areas to match and fuse edge regions. Transform domain stitching techniques convert images from the spatial domain to a transform domain (such as the Fourier transform domain) for processing, performing registration, fusion, and reconstruction through phase information or spectral similarity. Due to the high computational complexity of spatial domain and transform domain stitching techniques, they are easily affected by image background, noise, and illumination changes, making it difficult to maintain high stitching accuracy. Their applicability to gastric image stitching is greatly limited.

[0006] Feature point stitching technology is currently a mainstream research direction. It achieves stitching by detecting and matching key feature points in an image. Feature point stitching technology offers higher computational efficiency, greater robustness, and stability, and performs well in stitching images with complex textures and from varying camera angles. Common feature point extraction algorithms include Scale-Invariant Feature Transform (SIFT), Speed ​​Up Robust Feature (SURF), and Oriented FAST and Rotated BRIEF (ORB). Although feature point image stitching technology has potential in medical image processing, it still faces certain challenges.

[0007] First, magnetically controlled capsule endoscopy images are often affected by uneven lighting and blurred focus, which results in poor image quality and difficulty in extracting feature points. Especially given the complex textures and irregular shapes within the stomach, feature matching algorithms used in traditional stitching, such as SIFT and SURF, struggle to effectively extract sufficiently stable feature points. This leads to problems such as misalignment and noticeable seams in the stitching results, preventing the provision of continuous, seamless images. Second, the morphology and structure of the stomach change with the movement and contraction of the gastrointestinal tract. Traditional homography-based stitching methods cannot effectively handle non-rigid deformations. This results in noticeable misalignment and distortion in the stitching results, making it difficult to align images at different angles. This ultimately leads to misalignment and noticeable seams between images, and can even cause misalignment or omission of diseased areas, impacting detection accuracy. Summary of the Invention

[0008] The technical problem solved by this application is: how to overcome the problems of inaccurate feature detection and low matching accuracy caused by uneven lighting and sparse texture in the process of gastrointestinal magnetically controlled capsule endoscope image stitching.

[0009] The present application provides a method for stitching images of a digestive tract magnetically controlled capsule endoscopy, the method comprising:

[0010] Screening out candidate lesion area image sequences from the original digestive tract image sequences acquired by the magnetically controlled capsule endoscope;

[0011] Preprocessing the candidate lesion region image sequence to obtain a preprocessed candidate lesion region image sequence;

[0012] Feature extraction is performed on the pre-processed candidate lesion area image sequence based on the pre-trained EfficientLoFTR improved model. The EfficientLoFTR improved model is obtained by combining the RepVGG re-parameterized backbone network of the EfficientLoFTR original model with a fusion feature pyramid network structure and a depth-separable convolution based on a linear attention structure;

[0013] Performing feature matching on a pre-processed candidate lesion region image sequence based on the extracted features to obtain matching feature point pairs, and performing image registration processing on each image in the pre-processed candidate lesion region image sequence based on the matching feature point pairs;

[0014] The images obtained by image registration are fused to obtain a panoramic image of the candidate lesion area.

[0015] Optionally, the method of preprocessing the candidate lesion region image sequence includes:

[0016] Image cropping: cropping and removing irrelevant black background areas and edge areas in each image of the candidate lesion area image sequence;

[0017] Distortion correction: Use checkerboard images and Zhang Zhengyou's distortion correction to calibrate the camera, obtain the camera intrinsic parameters and distortion coefficients, and correct the radial and tangential distortions of each cropped image;

[0018] Image denoising: denoising each image after distortion correction based on a preset denoising algorithm;

[0019] Data enhancement: Perform image enhancement and standardization operations on each image after denoising.

[0020] Optionally, the method of performing image registration processing on each image in the pre-processed candidate lesion region image sequence according to the matching feature points includes:

[0021] According to the matching feature point pairs, image registration processing is performed on each image in the candidate lesion area image sequence based on global homography transformation and local thin plate spline transformation.

[0022] Optionally, before performing image registration processing, the stitching method includes:

[0023] Eliminate incorrect matching points in the matching feature point pairs.

[0024] Optionally, the method of performing image registration processing on each image in the candidate lesion region image sequence based on the matching feature point pairs and global homography transformation and local thin plate spline transformation includes:

[0025] The weighted homography model in the original APAP stitching algorithm is replaced by thin plate spline transformation;

[0026] Use the replaced APAP stitching algorithm to perform global homography transformation on each image to obtain preliminary registered images;

[0027] The preliminary registered image is divided into several grid areas, and each grid area is locally optimized using the replaced APAP stitching algorithm.

[0028] Optionally, the method of performing image fusion on the images obtained through the image registration process to obtain a panoramic image of the candidate lesion area includes:

[0029] Performing projection transformation on each image obtained by image registration processing;

[0030] The projected images are stitched together using an image fusion strategy to obtain a stitched image.

[0031] A stitching quality assessment is performed on the stitched image. If the stitching quality assessment is qualified, the stitched image is used as the final panoramic image of the candidate lesion area.

[0032] Optionally, the splicing method further includes:

[0033] The improved EfficientLoFTR model is trained using a public endoscopy dataset with depth information.

[0034] The present application also discloses a device for splicing images of a digestive tract magnetically controlled capsule endoscope, the device comprising:

[0035] An image screening module is used to screen candidate lesion area image sequences from the original digestive tract image sequences acquired by the magnetically controlled capsule endoscope;

[0036] An image preprocessing module, configured to preprocess the candidate lesion region image sequence to obtain a preprocessed candidate lesion region image sequence;

[0037] A feature extraction module is used to extract features from the preprocessed candidate lesion area image sequence based on the pre-trained EfficientLoFTR improved model. The EfficientLoFTR improved model is obtained by combining the RepVGG re-parameterized backbone network of the EfficientLoFTR original model with a fusion feature pyramid network structure and a depth-separable convolution based on a linear attention structure;

[0038] An image registration module is used to perform feature matching on a pre-processed candidate lesion region image sequence based on the extracted features to obtain matching feature point pairs, and perform image registration processing on each image in the pre-processed candidate lesion region image sequence based on the matching feature point pairs;

[0039] The image fusion module is used to fuse the images obtained by the image registration process to obtain a panoramic image of the candidate lesion area.

[0040] The present application also discloses a computer-readable storage medium, which stores a splicing program for magnetically controlled capsule endoscopy images of the digestive tract. When the splicing program for magnetically controlled capsule endoscopy images of the digestive tract is executed by a processor, the above-mentioned splicing method for magnetically controlled capsule endoscopy images of the digestive tract is implemented.

[0041] The present application also discloses a computer device, which includes a computer-readable storage medium, a processor, and a splicing program for magnetically controlled capsule endoscopy images of the digestive tract stored in the computer-readable storage medium. When the splicing program for magnetically controlled capsule endoscopy images of the digestive tract is executed by the processor, the above-mentioned splicing method for magnetically controlled capsule endoscopy images of the digestive tract is implemented.

[0042] The present application provides a method and device for stitching images of a digestive tract magnetically controlled capsule endoscopy, which have the following technical effects:

[0043] When performing feature extraction, in order to overcome the problem of mismatching or feature point loss in sparse and repeated areas, the original EfficientLoFTR model is re-parameterized by RepVGG backbone network, fused with feature pyramid network structure and combined with depthwise separable convolution on the basis of linear attention structure. The improved model can improve the accuracy of feature detection and matching, laying a good foundation for subsequent image registration and image fusion.

[0044] At the same time, in order to cope with the non-rigid deformation of the digestive tract, improvements were made on the basic framework of the original APAP stitching algorithm, and an image registration method combining global homography transformation and local thin plate spline transformation was obtained. Replacing the traditional weighted homography model with thin plate spline transformation can effectively handle non-rigid deformation and ensure more precise alignment of image details in local areas. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is a flowchart of a method for stitching digestive tract magnetically controlled capsule endoscopy images according to one or more embodiments;

[0046] Figure 2 is a principle block diagram of a device for stitching images of a digestive tract magnetically controlled capsule endoscopy according to one or more embodiments;

[0047] Figure 3 is a functional block diagram of a computer device according to one or more embodiments. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solutions and advantages of this application more clearly understood, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0049] Before describing the various embodiments of the present application in detail, the technical concept of the present application is first briefly described: Currently, in the process of stitching images of the digestive tract magnetically controlled capsule endoscopy, problems such as inaccurate feature detection and low matching accuracy are caused by uneven illumination and sparse texture, which affects the accuracy of subsequent stitching and makes it difficult to obtain high-quality panoramic images. To this end, the present application provides a method for stitching images of the digestive tract magnetically controlled capsule endoscopy. The key improvement is to extract features from the pre-processed candidate lesion area image sequence based on the pre-trained EfficientLoFTR improved model. The EfficientLoFTR improved model is obtained by combining the RepVGG re-parameterized backbone network, the fusion feature pyramid network structure and the linear attention structure with the depth-separable convolution of the EfficientLoFTR original model. The model can improve the accuracy of feature detection and matching, and lay a good foundation for subsequent image registration and image fusion. The specific principles of the stitching method for digestive tract magnetically controlled capsule endoscopy images of the present application are described below in combination with more embodiments.

[0050] Specifically, if Figure 1 As shown, the method for stitching digestive tract magnetically controlled capsule endoscopy images in this embodiment includes the following steps:

[0051] Step S10: selecting a candidate lesion region image sequence from the original digestive tract image sequence acquired by the magnetically controlled capsule endoscope;

[0052] Step S20: preprocessing the candidate lesion region image sequence to obtain a preprocessed candidate lesion region image sequence;

[0053] Step S30: extracting features from the pre-processed candidate lesion area image sequence based on the pre-trained EfficientLoFTR improved model, where the EfficientLoFTR improved model is obtained by performing a RepVGG re-parameterized backbone network on the EfficientLoFTR original model, integrating a feature pyramid network structure and a linear attention structure with depthwise separable convolution;

[0054] Step S40: performing feature matching on the pre-processed candidate lesion region image sequence according to the extracted features to obtain matching feature point pairs, and performing image registration processing on each image in the pre-processed candidate lesion region image sequence according to the matching feature point pairs;

[0055] Step S50: performing image fusion on the images obtained through the image registration process to obtain a panoramic image of the candidate lesion area.

[0056] In one or more embodiments, before step S10, the stomach image is first collected by a magnetically controlled capsule endoscope device. The device has a built-in miniature camera, which obtains the stomach image through wireless transmission technology and transmits the image data to an external receiving device. The capsule endoscope scans the stomach under the guidance of external magnetic control to capture fragmented images of all areas. After the acquisition is completed, the stomach image is preliminarily screened to check whether the image contains lesions. When a suspected lesion area is found, the continuous image sequence before and after the frame is selected as the candidate lesion area image sequence for subsequent image stitching. The images of the candidate lesion area image sequence contain multiple perspectives of the target area, which helps to improve the accuracy of stitching and ensure that the stitched image can fully and accurately present the lesion area, providing support for subsequent diagnosis.

[0057] In one or more embodiments, the pre-processing of step S20 includes the following steps:

[0058] Step S201: Image cropping: Cropping removes irrelevant black background and edge regions from each image in the candidate lesion region image sequence. For example, because capsule endoscopy images contain irrelevant black background or unnecessary portions, cropping these areas helps reduce redundant computations during stitching. Cropping techniques include, but are not limited to, center cropping, automatic cropping (detecting the valid image area and removing irrelevant background and edge portions), and region of interest (ROI) selection (selecting a specific stomach region as the focal area for stitching, removing unnecessary areas, and enhancing the stitching effect).

[0059] Step S202, distortion correction: Use the checkerboard image and Zhang Zhengyou's distortion correction to calibrate the camera, obtain the camera's intrinsic parameters and distortion coefficients, and correct the radial distortion and tangential distortion of each cropped image. For example, the magnetically controlled capsule endoscope image has barrel distortion caused by the wide-angle optical lens. Use the checkerboard image and Zhang Zhengyou's distortion correction to calibrate the camera, obtain the camera's intrinsic parameters and distortion coefficients, and correct the radial distortion and tangential distortion of the image. Ensure the accuracy of the image geometry, avoid registration errors caused by distortion, and provide a more reliable basic image sequence for the overall stitching process. The distortion correction formula is as follows:

[0060]

[0061]

[0062] Where, and represents the tangential distortion coefficient, represents the radial distortion coefficient, represents the distance from the optical center, represents the undistorted image coordinates, represents the distorted image coordinates, Represents the thin prism distortion parameter.

[0063] Step S203: Image denoising: Denoise each distortion-corrected image using a preset denoising algorithm. Exemplarily, denoising algorithms include, but are not limited to, mean filtering, median filtering, Gaussian filtering, and deep learning-based denoising algorithms. The primary purpose is to remove image noise, improve image quality, enhance image clarity and contrast, and enhance the accuracy of subsequent feature extraction and image matching, thereby providing cleaner and more reliable image data for the registration and stitching steps.

[0064] Step S204, data enhancement: Perform image enhancement and standardization operations on each image after denoising. Since a feature matching algorithm based on deep learning is used in the subsequent image registration process, the model needs to be fine-tuned to meet the input requirements of the deep learning model and improve the efficiency of feature extraction and matching, and image enhancement and standardization operations are performed. Through data enhancement, the conditions under different shooting angles, lighting conditions and image quality are simulated to expand the diversity of the data set and enhance the adaptability of the matching model to images in different environments. Image enhancement methods include rotation, folding and color space transformation. Standardization aims to eliminate differences between different images and make the input of the model consistent. The mean normalization formula is as follows:

[0065]

[0066] Where, is the value before normalization, is the normalized value, represents the mean of the data set, Represents the standard deviation of the data set.

[0067] In one or more embodiments, the improved EfficientLoFTR model of step S30 is obtained by making three changes to the original EfficientLoFTR model. First, in the stitching of gastrointestinal capsule endoscopy images, time consumption is an important performance indicator. Although the original EfficientLoFTR model performs well in feature matching accuracy, its computational complexity is relatively high. When processing capsule endoscopy images, it may cause excessive time overhead, affecting the efficiency of practical applications. In order to take into account both lightweight and high-precision feature extraction, the backbone network is processed by RepVGG reparameterization technology, and the model structure is simplified in the inference stage, which effectively reduces the computational complexity while maintaining the accuracy of feature extraction. The core of reparameterization is to merge the convolutional layer with the batch normalization layer to effectively reduce the computational consumption during inference. The key formula is as follows:

[0068]

[0069] Where, is the convolution kernel weight, and represent input and bias respectively. and are the mean and variance, and is a learnable parameter, Represents infinitesimal values ​​to prevent the denominator from being zero, represents the output of the convolutional layer, Represents the output of the normalization layer. Represent the weights and biases of the re-parameterized convolutional layer, Represents the output of the re-parameterized convolutional layer. Parameterization can effectively reduce the complexity of the inference model layer structure.

[0070] Second, to enhance the diversity and richness of feature expression, a feature pyramid network structure is added to capture detailed information in images of different resolutions through multi-scale feature fusion, improving feature matching. Third, the aggregated attention structure of the original EfficientLoFTR model is based on linear attention learning to locate and match stable and representative pixel features from the image. The formula is as follows:

[0071]

[0072] Linear attention reduces time complexity and memory requirements at the expense of performance, first computing and The loss of the ability to express complex feature relationships and the loss of dependency information. To address the above problems, this embodiment combines linear attention with depthwise separable convolution to capture rich local features and enhance the model's sensitivity to fine-grained information. The improved attention formula is as follows:

[0073]

[0074] Furthermore, after obtaining the improved EfficientLoFTR model, it was trained using a public endoscopy dataset with depth information to adapt the improved model to capsule endoscopy image data. Based on the EfficientLoFTR model structure, supervised training was performed using the SCARED dataset, a public endoscopy image dataset with depth information, to improve the model's feature matching performance in endoscopy image scenarios. The training process included constructing a dataset based on the MegaDepth dataset format used by the base model. The model was optimized accordingly, following the basic training strategy and using the default matching loss function of EfficientLoFTR. Real matching points generated from the dataset's registration information served as a supervisory signal to guide the model in optimizing matching accuracy. AdamW was selected as the optimizer, along with a warm-up learning rate strategy to ensure stable training convergence. Model parameters were initialized using weights pre-trained on a large-scale general image matching dataset and fine-tuned through transfer learning to adapt to the feature distribution of capsule endoscopy images.

[0075] Through pre-training and fine-tuning using transfer learning, the goal is to make the improved EfficientLoFTR model more adaptable to the illumination complexity and distortion characteristics of endoscopic images, and to improve matching accuracy in weak-texture environments. The trained improved EfficientLoFTR model extracts features from pre-processed image sequences of candidate lesion regions. Through an adaptive feature matching mechanism, the improved model ensures the accurate identification and matching of stable feature point pairs in images of suspected lesion regions, while obtaining the confidence level of the matched feature point pairs. This addresses the problem of featureless or mismatched features in traditional methods, and provides high-quality, precise feature point pairs to support subsequent transformation estimation, thereby improving the accuracy and robustness of registration.

[0076] In one or more embodiments, in step 40, the method of performing image registration processing on each image of the preprocessed candidate lesion area image sequence according to matching feature point pairs includes: performing image registration processing on each image of the candidate lesion area image sequence based on global homography transformation and local thin plate spline transformation according to matching feature point pairs.

[0077] For example, to address the unique non-rigid deformation of the digestive tract, improvements were made to the basic framework of the original APAP stitching algorithm, resulting in an image registration method that combines global homography transformation and local thin plate spline transformation, using thin plate spline transformation to replace the traditional weighted homography model. Global homography transformation is used for preliminary image registration. The homography matrix describes the global projection relationship between the two images and is used to model the overall perspective transformation. Global homography ensures the consistency of the main structure of the image. The homography is calculated as follows:

[0078]

[0079] Where, represents the homography matrix, Represent the coordinates between the matching feature point pairs respectively. In capsule endoscopy images with non-rigid deformation or complex surface characteristics, the fitting effect of the local area may be insufficient, resulting in misalignment or deformation of local details. To address the above limitations, the APAP algorithm is used to locally optimize the image. After performing a global homography transformation and obtaining a preliminary registration image, the preliminary registration image is divided into a grid, and the weighted homography is optimized using a local thin plate spline transformation. The local area mapping is adjusted and each grid area is locally optimized. The thin plate spline transformation is a transformation method based on control points. It can effectively handle non-rigid deformations and ensure that the details of the image in the local area are more accurately aligned. It is suitable for non-rigid deformations of the digestive tract. The thin plate spline transformation is as follows:

[0080]

[0081] Where, represents the thin plate spline kernel, and x and y represent the coordinates of the transformation point.

[0082] In one or more embodiments, before performing image registration processing, the stitching method includes: eliminating erroneous matching points in the matching feature point pairs. For example, in order to effectively eliminate erroneous matching points, the global homography transformation matrix is ​​calculated more accurately, and the MAGSAC (M-estimator SAmple Consensus) algorithm is used to estimate the homography matrix between the candidate lesion area image sequences. The algorithm automatically evaluates the quality of matching points by M estimation, and dynamically adjusts the weights to effectively eliminate erroneous matching points. The specific steps are as follows: First, randomly sample 4 pairs of matching points and calculate the homography transformation matrix between image sequences. Then all matching feature point pairs are brought into the current , calculate the reprojection error. If the error is less than a threshold, the point is considered an inlier; otherwise, it is an outlier. Next, count the number of inliers and use the M-estimator to weight inliers and outliers. Finally, repeat the iterations and select the homography matrix with the most inliers and the smallest error as the optimal model.

[0083] In one or more embodiments, in step S40, the method of performing image fusion on the images obtained by the image registration process to obtain a panoramic image of the candidate lesion area includes:

[0084] Step S401, perform projection transformation on each image obtained by the image registration process. For example, in order to prevent the destruction of the image field consistency in the subsequent fusion process, the projection transformation adjusts the spatial relationship of the image sequence to ensure a smooth transition between adjacent images and avoid unnatural effects after splicing. The projection transformations that can be used include spherical projection, cylindrical projection and fisheye projection. The commonly used cylindrical projection projects the image onto a cylindrical surface and then unfolds it into a two-dimensional plane, which can better handle horizontal image stitching. At the same time, in order to avoid obvious seams and uneven brightness caused by exposure differences, exposure compensation is used to adjust the images to be spliced ​​to keep the brightness balanced. The method of use is to multiply different images by different compensation coefficients. The cylindrical projection formula is as follows:

[0085]

[0086] Where, and represents the projected coordinates, and represents the coordinates before projection, and Indicates the image height and width, Represents the focal length of the camera.

[0087] Step S402: Use image fusion strategy to stitch the projected transformed images to obtain a stitched image. Specifically, the purpose of the image fusion operation is to smoothly combine multiple images into a seamless stitching result, avoid obvious seams or unnatural transitions, ensure image consistency and visual naturalness. For medical endoscopic images, the most important thing is to ensure the integrity of the information and allow certain seams. Image fusion strategies mainly include weighted average fusion, multi-band fusion and optimal stitching. Weighted average fusion is obtained by weighted averaging the pixel values ​​of the overlapping area of ​​the stitched image; multi-band fusion decomposes the image into multiple bands through the Laplace pyramid, and different bands are weightedly fused with different strategies; the optimal stitching strategy uses a dynamic programming algorithm or a greedy algorithm to find the optimal seam path.

[0088] Step S403: Perform stitching quality assessment on the stitched image. If the stitching quality assessment is qualified, the stitched image is used as the final panoramic image of the candidate lesion area. For example, after image fusion is completed, stitching quality assessment is a key step to ensure that the stitched image can be used to assist in lesion detection. The indicators of stitching quality assessment mainly include Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), and subjective perception. Peak Signal-to-Noise Ratio represents the ratio of the maximum intensity of the signal to the noise in the image. The structural similarity index is an indicator that measures the similarity in terms of structure, brightness, and contrast. Subjective perception evaluates the overall effect of the stitched image through manual visual inspection. Among them, the calculation process of each indicator of stitching quality assessment is a well-known technology in the field and will not be described in detail here.

[0089] Through the above step S40, each image is sequentially fused to obtain a panoramic image of the candidate lesion area.

[0090] In one or more embodiments, Figure 2As shown, the device for stitching digestive tract magnetically controlled capsule endoscopy images includes an image screening module 100, an image preprocessing module 200, a feature extraction module 300, an image registration module 400, and an image fusion module 500. The image screening module 100 is used to screen out a candidate lesion area image sequence from the original digestive tract image sequence collected by the magnetically controlled capsule endoscope; the image preprocessing module 200 is used to preprocess the candidate lesion area image sequence to obtain a preprocessed candidate lesion area image sequence; the feature extraction module 300 is used to extract features from the preprocessed candidate lesion area image sequence based on the pretrained EfficientLoFTR improved model. The improved EfficientLoFTR model is obtained by combining the RepVGG re-parameterized backbone network, the fusion feature pyramid network structure and the depth-separable convolution on the basis of the linear attention structure of the original EfficientLoFTR model; the image registration module 400 is used to perform feature matching on the pre-processed candidate lesion area image sequence according to the extracted features to obtain matching feature point pairs, and perform image registration processing on each image of the pre-processed candidate lesion area image sequence according to the matching feature point pairs; the image fusion module 500 is used to perform image fusion on each image obtained by the image registration processing to obtain a panoramic image of the candidate lesion area.

[0091] Exemplarily, the image preprocessing module 200 includes an image cropping unit, a distortion correction unit, an image denoising unit, and a data enhancement unit. The image cropping unit is used to crop and remove irrelevant black background areas and edge areas from each image in the candidate lesion area image sequence; the distortion correction unit uses checkerboard images and Zhang Zhengyou's distortion correction to perform camera calibration, obtain camera intrinsic parameters and distortion coefficients, and correct radial and tangential distortions of each cropped image; the image denoising unit performs denoising on each distortion-corrected image based on a preset denoising algorithm; and the data enhancement unit performs image enhancement and standardization on each denoised image.

[0092] Exemplarily, the image registration module 400 is used to perform image registration processing on each image in the candidate lesion area image sequence based on matching feature point pairs, based on global homography transformation and local thin plate spline transformation, specifically including: replacing the weighted homography model in the original APAP stitching algorithm with thin plate spline transformation; using the replaced APAP stitching algorithm to perform global homography transformation on each image to obtain a preliminary registered image; dividing the preliminary registered image into several grid areas, and using the replaced APAP stitching algorithm to perform local optimization on each grid area.

[0093] Exemplarily, the image fusion module 500 is used to: perform projection transformation on each image obtained by image registration processing, stitch each projection transformed image using an image fusion strategy to obtain a stitched image, and perform stitching quality evaluation on the stitched image. If the stitching quality evaluation is qualified, the stitched image is used as the final panoramic image of the candidate lesion area.

[0094] The detailed working process of each module of the device for splicing images of the digestive tract magnetically controlled capsule endoscopy can be referred to the relevant description of the splicing method in the previous embodiment, which will not be repeated here.

[0095] In one or more embodiments, a computer-readable storage medium stores a stitching program for magnetically controlled capsule endoscopy images of the digestive tract. When the stitching program for magnetically controlled capsule endoscopy images of the digestive tract is executed by a processor, the stitching method for magnetically controlled capsule endoscopy images of the digestive tract of embodiment one is implemented.

[0096] This embodiment also discloses a computer device, at the hardware level, such as Figure 3 As shown, the computer device includes a processor 12, an internal bus 13, a network interface 14, and a computer-readable storage medium 11. The processor 12 reads the corresponding computer program from the computer-readable storage medium and then runs it, forming a request processing device at the logical level. Of course, in addition to software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices. The computer-readable storage medium 11 stores a splicing program for the magnetically controlled capsule endoscopy images of the digestive tract. When the splicing program for the magnetically controlled capsule endoscopy images of the digestive tract is executed by the processor, the above-mentioned splicing method for the magnetically controlled capsule endoscopy images of the digestive tract is implemented.

[0097] Computer-readable storage media include both permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer-readable storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0098] The above describes in detail the specific implementation methods of the present application. Although some embodiments have been shown and described, those skilled in the art should understand that these embodiments can be modified and improved without departing from the principles and spirit of the present application, the scope of which is defined by the claims and their equivalents. These modifications and improvements should also be within the scope of protection of the present application.

Claims

1. A method for stitching images of a digestive tract magnetically controlled capsule endoscopy, characterized in that: The splicing method comprises: Screening out candidate lesion area image sequences from the original digestive tract image sequences acquired by the magnetically controlled capsule endoscope; Preprocessing the candidate lesion region image sequence to obtain a preprocessed candidate lesion region image sequence; Feature extraction is performed on the pre-processed candidate lesion area image sequence based on the pre-trained EfficientLoFTR improved model. The EfficientLoFTR improved model is obtained by combining the RepVGG re-parameterized backbone network of the EfficientLoFTR original model with a fusion feature pyramid network structure and a depth-separable convolution based on a linear attention structure; Performing feature matching on a pre-processed candidate lesion region image sequence based on the extracted features to obtain matching feature point pairs, and performing image registration processing on each image in the pre-processed candidate lesion region image sequence based on the matching feature point pairs; The images obtained by image registration are fused to obtain a panoramic image of the candidate lesion area.

2. The method for stitching digestive tract magnetically controlled capsule endoscopy images according to claim 1, characterized in that: The method for preprocessing the candidate lesion region image sequence includes: Image cropping: cropping and removing irrelevant black background areas and edge areas in each image of the candidate lesion area image sequence; Distortion correction: Use checkerboard images and Zhang Zhengyou's distortion correction to calibrate the camera, obtain the camera intrinsic parameters and distortion coefficients, and correct the radial and tangential distortions of each cropped image; Image denoising: denoising each image after distortion correction based on a preset denoising algorithm; Data enhancement: Perform image enhancement and standardization operations on each image after denoising.

3. The method for stitching digestive tract magnetically controlled capsule endoscopy images according to claim 1, characterized in that: The method for performing image registration processing on each image of the pre-processed candidate lesion area image sequence according to the matching feature points includes: According to the matching feature point pairs, image registration processing is performed on each image in the candidate lesion area image sequence based on global homography transformation and local thin plate spline transformation.

4. The method for stitching digestive tract magnetically controlled capsule endoscopy images according to claim 3, characterized in that: Before performing image registration processing, the stitching method includes: Eliminate incorrect matching points in the matching feature point pairs.

5. The method for stitching digestive tract magnetically controlled capsule endoscopy images according to claim 3, characterized in that: The method of performing image registration processing on each image in the candidate lesion area image sequence based on the matching feature point pairs and global homography transformation and local thin plate spline transformation includes: The weighted homography model in the original APAP stitching algorithm is replaced by thin plate spline transformation; Use the replaced APAP stitching algorithm to perform global homography transformation on each image to obtain preliminary registered images; The preliminary registered image is divided into several grid areas, and each grid area is locally optimized using the replaced APAP stitching algorithm.

6. The method for stitching digestive tract magnetically controlled capsule endoscopy images according to claim 1, characterized in that: The method of performing image fusion on the images obtained by the image registration process to obtain a panoramic image of the candidate lesion area includes: Performing projection transformation on each image obtained by image registration processing; The projected images are stitched together using an image fusion strategy to obtain a stitched image. A stitching quality assessment is performed on the stitched image. If the stitching quality assessment is qualified, the stitched image is used as the final panoramic image of the candidate lesion area.

7. The method for stitching digestive tract magnetically controlled capsule endoscopy images according to claim 1, characterized in that: The splicing method further comprises: The improved EfficientLoFTR model is trained using a public endoscopy dataset with depth information.

8. A device for splicing images of a digestive tract magnetically controlled capsule endoscopy, characterized in that: The splicing device comprises: An image screening module is used to screen candidate lesion area image sequences from the original digestive tract image sequences acquired by the magnetically controlled capsule endoscope; An image preprocessing module, configured to preprocess the candidate lesion region image sequence to obtain a preprocessed candidate lesion region image sequence; A feature extraction module is used to extract features from the preprocessed candidate lesion area image sequence based on the pre-trained EfficientLoFTR improved model. The EfficientLoFTR improved model is obtained by combining the RepVGG re-parameterized backbone network of the EfficientLoFTR original model with a fusion feature pyramid network structure and a depth-separable convolution based on a linear attention structure; An image registration module is used to perform feature matching on a pre-processed candidate lesion region image sequence based on the extracted features to obtain matching feature point pairs, and perform image registration processing on each image in the pre-processed candidate lesion region image sequence based on the matching feature point pairs; The image fusion module is used to fuse the images obtained by the image registration process to obtain a panoramic image of the candidate lesion area.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a splicing program for magnetically controlled capsule endoscopy images of the digestive tract. When the splicing program for magnetically controlled capsule endoscopy images of the digestive tract is executed by a processor, the splicing method for magnetically controlled capsule endoscopy images of the digestive tract according to any one of claims 1 to 7 is implemented.

10. A computer device, characterized in that: The computer device includes a computer-readable storage medium, a processor, and a stitching program for magnetically controlled capsule endoscopy images of the digestive tract stored in the computer-readable storage medium. When the stitching program for magnetically controlled capsule endoscopy images of the digestive tract is executed by the processor, the stitching method for magnetically controlled capsule endoscopy images of the digestive tract described in any one of claims 1 to 7 is implemented.

Images