Splicing method and splicing device for digestive tract magnetic control capsule endoscope images

Abstract

The invention provides a splicing method and a splicing device for a magnetic control capsule endoscope image of a digestive tract. The method comprises the following steps: screening out a candidate lesion area image sequence from an original alimentary canal image sequence collected by a magnetic control capsule endoscope; preprocessing the candidate lesion area image sequence to obtain a preprocessed candidate lesion area image sequence; performing feature extraction on the preprocessed candidate lesion area image sequence based on an OfficientLoFTR improved model, further performing feature matching on the preprocessed candidate lesion area image sequence to obtain matched feature point pairs, and performing image registration processing on each image of the preprocessed candidate lesion area image sequence according to the matched feature point pairs to obtain a pre-processed candidate lesion area image sequence; the improved model can improve the accuracy of feature detection and matching, a good foundation is laid for subsequent image registration and image fusion, image registration can be carried out by combining global homography transformation and local thin plate spline transformation, and the problem of non-rigid deformation is solved.

Description

Technical Field

[0001] The present application belongs to the technical field of capsule endoscopes, and in particular, relates to a method and a device for stitching images of a digestive tract magnetically controlled capsule endoscope. Background Art

[0002] With the development of medical imaging technology, Magnetically Controlled Capsule Endoscopy (MCCE) is a new type of digestive tract examination tool, which is widely used in the early detection of digestive tract ulcers, inflammation and cancer. Because of its painless, non-invasive and anesthesia-free characteristics, it has gradually become a powerful supplement to traditional endoscopic examinations. Magnetically controlled capsule endoscopy relies on an external magnetic control system to remotely control the capsule, which can accurately move, rotate and stay in the digestive tract, improve the flexibility and coverage of digestive tract examinations, and help to more comprehensively observe potential lesion areas in the digestive tract.

[0003] In the actual application of magnetically controlled capsule endoscopes, due to the complex internal environment and variable shooting angles, the shooting system often collects a large number of repeated and fragmented images. The huge fragmented images limit the doctor's accurate identification and judgment of the candidate lesion area, increasing the difficulty of diagnosis. In order to solve the limitations brought by fragmented images and realize the construction of a continuous view of the digestive tract, image stitching technology becomes the key. Image stitching technology expands the field of view and improves the continuity and integrity of the image by merging multiple image fragments into a complete image. By stitching images collected at different angles and positions, a local panoramic image of the candidate lesion area is generated, allowing doctors to observe in a wider field of view, promote more accurate gastric lesion detection, and promote the wider application of non-invasive endoscopic technology in clinical practice. However, due to the irregular shape, sparse texture and strong repeatability of the digestive tract, image stitching faces many challenges in feature matching, image registration and seamless fusion. Therefore, developing an image stitching method for magnetically controlled capsule endoscopes for the stomach has become a key requirement to improve the inspection effect.

[0004] Image stitching technology is widely used in the medical field. It can break through the limitations of traditional imaging examinations and provide key technical support for the development of precision medical diagnosis and treatment. From the perspective of technical characteristics, image stitching technology can be mainly divided into spatial domain and transform domain stitching technology and feature point stitching technology.

[0005] The spatial domain and transform domain stitching techniques mainly rely on the global information of the image for stitching. The spatial domain stitching technique directly operates in the pixel space of the image. By finding the overlapping regions between adjacent images, it compares the pixel gray values or color information in the overlapping regions to perform matching and fuse the edge regions. The transform domain stitching technique processes the image by transforming it from the spatial domain to the transform domain (such as the Fourier transform domain). It performs registration fusion and reconstruction through phase information or spectral similarity. Since the spatial domain and transform domain stitching techniques have a high computational complexity and are easily affected by image background, noise, and illumination changes, it is difficult to maintain a high stitching accuracy, and their applicability in gastric image stitching is greatly limited.

[0006] The feature point stitching technique is the current mainstream research direction, which realizes stitching by detecting and matching key feature points in the image. The feature point stitching technique has higher computational efficiency, stronger robustness and stability, and has good performance in image stitching under complex textures and different shooting angles. Common feature point extraction algorithms include Scale-Invariant Feature Transform (SIFT), Speed Up Robust Feature (SURF), and Oriented FAST and Rotated BRIEF (ORB). Although the feature point image stitching technique has potential in medical image processing, it still faces certain challenges.

[0007] First, the images of the magnetically controlled capsule endoscope are often affected by problems such as uneven illumination and defocus blur. These factors result in poor image quality and difficulty in extracting feature points. Especially under the complex textures and irregular shapes in the stomach, traditional feature matching algorithms used in stitching, such as SIFT and SURF, are difficult to effectively extract sufficient stable feature points, which in turn leads to problems such as misalignment and obvious seams in the stitching result, and cannot provide continuous and seamless images. Second, the shape and structure of the stomach change with the movement and contraction of the gastrointestinal tract. Traditional homography-based stitching methods cannot effectively handle non-rigid deformations. This results in obvious misalignment and distortion in the stitching result, making it difficult to handle image alignment at different angles. Eventually, there are misalignments and obvious seams between images, and even mis-stitching or omission of the lesion area, affecting the detection accuracy. Summary of the Invention

[0008] The technical problem solved by this application is: how to overcome the problems such as inaccurate feature detection and low matching accuracy caused by uneven illumination and sparse texture during the stitching process of the magnetically controlled capsule endoscope images of the digestive tract.

[0009] This application provides a stitching method for magnetically controlled capsule endoscope images of the digestive tract, and the stitching method includes: Select the image sequence of candidate lesion regions from the original digestive tract image sequence collected by the magnetically controlled capsule endoscope; Preprocess the image sequence of the candidate lesion regions to obtain a preprocessed image sequence of the candidate lesion regions; Extract features from the preprocessed image sequence of the candidate lesion regions based on a pre-trained improved EfficientLoFTR model, where the improved EfficientLoFTR model is obtained by performing RepVGG reparameterized backbone network, fusing feature pyramid network structure and online linear attention structure on the original EfficientLoFTR model and combining depthwise separable convolution; Perform feature matching on the preprocessed image sequence of the candidate lesion regions according to the extracted features to obtain matching feature point pairs, and perform image registration processing on each image of the preprocessed image sequence of the candidate lesion regions according to the matching feature point pairs; Perform image fusion on each image obtained by the image registration processing to obtain a panoramic image of the candidate lesion regions.

[0010] Optionally, the method for preprocessing the image sequence of the candidate lesion regions includes: Image cropping: Crop and remove the irrelevant black background regions and edge regions in each image of the image sequence of the candidate lesion regions; Distortion correction: Use a checkerboard image and Zhang Zhengyou's distortion correction to calibrate the camera, obtain the camera internal parameters and distortion coefficients, and correct the radial distortion and tangential distortion of each cropped image; Image denoising: Perform denoising processing on each image after distortion correction based on a preset denoising algorithm; Data augmentation: Perform image enhancement and normalization operations on each image after denoising processing.

[0011] Optionally, the method for performing image registration processing on each image of the preprocessed image sequence of the candidate lesion regions according to the matching feature point pairs includes: Perform image registration processing on each image of the image sequence of the candidate lesion regions based on global homography transformation and local thin plate spline transformation according to the matching feature point pairs.

[0012] Optionally, before performing the image registration processing, the stitching method includes: Remove the incorrect matching points in the matching feature point pairs.

[0013] Optionally, the method for performing image registration processing on each image of the image sequence of the candidate lesion regions based on global homography transformation and local thin plate spline transformation according to the matching feature point pairs includes: Replace the weighted homography model in the original APAP stitching algorithm with a thin plate spline transformation; Use the replaced APAP stitching algorithm to perform global homography transformation on each image to obtain a preliminary registered image; Divide the preliminary registered image into several grid regions, and use the replaced APAP stitching algorithm to perform local optimization on each grid region.

[0014] Optionally, the method for image fusion of each image obtained by image registration processing to obtain a panoramic image of the candidate lesion region includes: Perform projective transformation on each image obtained by image registration processing; Stitch the images obtained by projective transformation using an image fusion strategy to obtain a stitched image; Perform stitching quality assessment on the stitched image. If the stitching quality assessment is qualified, use the stitched image as the final panoramic image of the candidate lesion region.

[0015] Optionally, the stitching method further includes: Train an improved EfficientLoFTR model using a publicly available endoscopic dataset with depth information.

[0016] This application also discloses a stitching device for gastrointestinal magnetic control capsule endoscope images. The stitching device includes: An image screening module for screening out a candidate lesion region image sequence from the original gastrointestinal image sequence collected by the magnetic control capsule endoscope; An image preprocessing module for preprocessing the candidate lesion region image sequence to obtain a preprocessed candidate lesion region image sequence; A feature extraction module for extracting features from the preprocessed candidate lesion region image sequence based on a pre-trained improved EfficientLoFTR model. The improved EfficientLoFTR model is obtained by performing RepVGG reparameterized backbone network, fusing feature pyramid network structure and online linear attention structure on the original EfficientLoFTR model and combining depthwise separable convolution; An image registration module for feature matching of the preprocessed candidate lesion region image sequence according to the extracted features to obtain matching feature point pairs, and performing image registration processing on each image of the preprocessed candidate lesion region image sequence according to the matching feature point pairs; An image fusion module for performing image fusion on each image obtained by image registration processing to obtain a panoramic image of the candidate lesion region.

[0017] The present application also discloses a computer-readable storage medium storing a splicing program for gastrointestinal magnetically controlled capsule endoscope images. When the splicing program for gastrointestinal magnetically controlled capsule endoscope images is executed by a processor, the above-mentioned splicing method for gastrointestinal magnetically controlled capsule endoscope images is implemented.

[0018] The present application also discloses a computer device, which includes a computer-readable storage medium, a processor, and a splicing program for gastrointestinal magnetically controlled capsule endoscope images stored in the computer-readable storage medium. When the splicing program for gastrointestinal magnetically controlled capsule endoscope images is executed by the processor, the above-mentioned splicing method for gastrointestinal magnetically controlled capsule endoscope images is implemented.

[0019] A splicing method and a splicing device for gastrointestinal magnetically controlled capsule endoscope images provided by the present application have the following technical effects: When performing feature extraction, in order to overcome the problems of easy false matching or feature point loss in feature-sparse and repetitive regions, the original EfficientLoFTR model is reparameterized with a RepVGG backbone network, combined with a feature pyramid network structure and an online linear attention structure, and depthwise separable convolution is incorporated. The improved model can improve the accuracy of feature detection and matching, laying a good foundation for subsequent image registration and image fusion.

[0020] At the same time, to cope with the non-rigid deformation of the digestive tract, the original APAP splicing algorithm is improved on the basis of its basic framework to obtain an image registration method that combines global homography transformation and local thin plate spline transformation. Using the thin plate spline transformation to replace the traditional weighted homography model can effectively handle non-rigid deformation and ensure more accurate alignment of details in the local area of the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a flowchart of a splicing method for gastrointestinal magnetically controlled capsule endoscope images according to one or more embodiments; Figure 2 is a schematic block diagram of a splicing device for gastrointestinal magnetically controlled capsule endoscope images according to one or more embodiments; Figure 3 is a schematic block diagram of a computer device according to one or more embodiments. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0022] In order to make the objectives, technical solutions, and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0023] Before describing the various embodiments of the present application in detail, first briefly describe the technical concept of the present application: Currently, in the process of stitching images of a magnetically controlled capsule endoscope in the digestive tract, problems such as inaccurate feature detection and low matching accuracy are caused by uneven illumination and sparse texture, thus affecting the accuracy of subsequent stitching and making it difficult to obtain high-quality panoramic images. To this end, the present application provides a stitching method for images of a magnetically controlled capsule endoscope in the digestive tract. The key improvement point is to extract features from the preprocessed candidate lesion region image sequence based on a pre-trained EfficientLoFTR improved model. The EfficientLoFTR improved model is obtained by performing RepVGG reparameterized backbone network, fusing feature pyramid network structure and online linear attention structure on the original EfficientLoFTR model and combining depthwise separable convolution. This model can improve the accuracy of feature detection and matching, laying a good foundation for subsequent image registration and image fusion. The specific principle of the stitching method for images of a magnetically controlled capsule endoscope in the digestive tract of the present application will be described below in combination with more embodiments.

[0024] Specifically, as Figure 1 shown, the stitching method for images of a magnetically controlled capsule endoscope in the digestive tract in this first embodiment includes the following steps: Step S10: Screen out the candidate lesion region image sequence from the original digestive tract image sequence collected by the magnetically controlled capsule endoscope; Step S20: Preprocess the candidate lesion region image sequence to obtain the preprocessed candidate lesion region image sequence; Step S30: Extract features from the preprocessed candidate lesion region image sequence based on a pre-trained EfficientLoFTR improved model. The EfficientLoFTR improved model is obtained by performing RepVGG reparameterized backbone network, fusing feature pyramid network structure and online linear attention structure on the original EfficientLoFTR model and combining depthwise separable convolution; Step S40: Perform feature matching on the preprocessed candidate lesion region image sequence according to the extracted features to obtain matching feature point pairs, and perform image registration processing on each image of the preprocessed candidate lesion region image sequence according to the matching feature point pairs; Step S50: Perform image fusion on each image obtained by the image registration processing to obtain the panoramic image of the candidate lesion region.

[0025] In one or more embodiments, before step S10, first, a gastric image is acquired by a magnetically controlled capsule endoscope device. The device is equipped with a miniature camera, and the gastric image is obtained through wireless transmission technology and the image data is transmitted to an external receiving device. The capsule endoscope scans the stomach under external magnetic control to capture fragmented images of the entire area. After the acquisition is completed, the gastric images are preliminarily screened to check whether there are lesions in the images. When a suspected lesion area is found, a continuous image sequence before and after that frame is selected as the candidate lesion area image sequence for subsequent image stitching. The images in the candidate lesion area image sequence contain multiple perspectives of the target area, which helps to improve the stitching accuracy and ensure that the stitched image can completely and accurately present the lesion area, providing support for subsequent diagnosis.

[0026] In one or more embodiments, the preprocessing in step S20 includes the following steps: Step S201, image cropping: Crop and remove the irrelevant black background area and edge area in each image of the candidate lesion area image sequence. Exemplarily, since the capsule endoscope image contains an irrelevant black background area or unnecessary parts, cropping these areas helps to reduce redundant calculations during stitching. Cropping techniques include but are not limited to: central cropping, automatic cropping (detecting the effective area in the image and removing the irrelevant background and edge parts), and region of interest (ROI) selection (selecting a specific gastric area as the key area for stitching, removing unnecessary areas, and enhancing the stitching effect).

[0027] Step S202, distortion correction: Use a checkerboard image and Zhang Zhengyou's distortion correction for camera calibration to obtain the camera internal parameters and distortion coefficients, and correct the radial distortion and tangential distortion of each cropped image. Exemplarily, the magnetically controlled capsule endoscope image has barrel distortion caused by a wide-angle optical lens. Use a checkerboard image and Zhang Zhengyou's distortion correction for camera calibration to obtain the camera internal parameters and distortion coefficients, and correct the radial distortion and tangential distortion of the image. Ensure the accuracy of the image geometry, avoid registration errors caused by distortion, and provide a more reliable basic image sequence for the overall stitching process. The distortion correction formula is as follows:

[0028]

[0029]

[0030] In the formula, and represent the tangential distortion coefficients, represents the radial distortion coefficient, represents the distance from the optical center, represents the undistorted image coordinates, Represent the image coordinates of distortion, Represent the thin prism distortion parameters.

[0031] Step S203, Image denoising: Denoise each of the images after distortion correction based on a preset denoising algorithm. Exemplarily, the denoising algorithm includes but is not limited to mean filtering, median filtering, Gaussian filtering (GaussianFiltering), and deep learning-based denoising algorithms. The main purpose is to remove image noise, improve the quality of the images, enhance the clarity and contrast of the images, and enhance the accuracy of subsequent feature extraction and image matching, so as to provide cleaner and more reliable image data for the registration and stitching steps.

[0032] Step S204, Data augmentation: Perform image enhancement and normalization operations on each of the denoised images. Since a deep learning-based feature matching algorithm is used in the subsequent image registration process and the model needs to be fine-tuned, in order to meet the input requirements of the deep learning model and improve the efficiency of feature extraction and matching, image enhancement and normalization operations are carried out. Through data augmentation, situations under different shooting angles, lighting conditions, and image qualities are simulated, the diversity of the dataset is expanded, and the adaptability of the matching model to images under different environments is enhanced. Image enhancement methods include rotation, flipping, and color space transformation, etc. Normalization aims to eliminate the differences between different images and make the input of the model consistent. The mean normalization formula is as follows:

[0033]

[0034] In the formula, is the value before normalization, is the value after normalization, represents the mean of the dataset, represents the standard deviation of the dataset.

[0035] In one or more embodiments, the EfficientLoFTR improved model in step S30 is obtained by making three modifications to the EfficientLoFTR original model. First, in the image stitching of digestive tract capsule endoscopes, time consumption is an important performance indicator. Although the EfficientLoFTR original model performs well in terms of feature matching accuracy, its computational complexity is relatively high. When processing capsule endoscope images, it may lead to excessive time overhead and affect the efficiency of practical applications. To balance lightweight and high-precision feature extraction, the backbone network is processed through the RepVGG reparameterization technique to simplify the model structure during the inference stage, effectively reducing the computational complexity while maintaining the accuracy of feature extraction unchanged. The core of reparameterization is to merge the convolutional layer and the batch normalization layer, effectively reducing the computational consumption during inference. The key formula is as follows:

[0036]

[0037] In the formula, is the convolution kernel weight, and represent the input and bias respectively. and are the mean and variance, and are learnable parameters, represents an infinitesimal value to prevent the denominator from being zero, represents the output of the convolutional layer, represents the output of the normalization layer. represent the weight and bias of the reparameterized convolutional layer respectively, represents the output of the reparameterized convolutional layer. After parameterization, it can effectively reduce the complexity of the inference model layer structure.

[0038] Second, to enhance the diversity and richness of feature expression, a feature pyramid network structure is added. By fusing multi-scale features, it captures the detailed information in images with different resolutions and improves the feature matching effect. Third, the aggregation attention structure of the original EfficientLoFTR model learns how to locate and match stable and representative pixel features from images based on linear attention. The formula is as follows:

[0039]

[0040] Linear attention reduces the time complexity and memory requirements at the cost of performance, and first calculates and The loss's ability to express complex feature relationships and the loss of dependent information. To address the above problems, in this embodiment, depthwise separable convolution is combined with linear attention to capture rich local features and enhance the model's sensitivity to fine-grained information. The improved attention formula is as follows:

[0041]

[0042] Furthermore, after obtaining the EfficientLoFTR improved model, it is trained using a publicly available endoscopic dataset with depth information so that the improved model can adapt to the capsule endoscopy image data. Based on the EfficientLoFTR model structure, supervised training is carried out using the publicly available endoscopic image dataset SCARED with depth information to improve the feature matching performance of the model in the endoscopic image scenario. The training process includes constructing a dataset by referring to the MegaDepth dataset format used by the basic model. The model is correspondingly optimized, and the basic training strategy is followed. The default matching loss function of EfficientLoFTR is adopted, and the real matching points generated from the dataset registration information are used as the supervision signal to guide the model to optimize the matching accuracy. The optimizer is selected as AdamW, and the learning rate warm-up strategy is used to ensure stable convergence of the training. The model parameters are initialized using the weights pre-trained on a large-scale general image matching dataset and fine-tuned through transfer learning to adapt to the feature distribution of capsule endoscopy images.

[0043] Through the pre-training fine-tuning of transfer learning, it aims to make the improved EfficientLoFTR model more adaptable to the illumination complexity and distortion characteristics of endoscopic images and improve the matching accuracy in weak texture environments. The trained EfficientLoFTR improved model extracts features from the preprocessed candidate lesion area image sequence. Through an adaptive feature matching mechanism, the improved model ensures accurate identification and matching of stable feature point pairs in the images of suspected lesion areas, while obtaining the confidence of the matching feature point pairs, solving the problem of no features or mis-matching in traditional methods, and providing high-quality and accurate support for feature point pairs for subsequent transformation estimation, thereby improving the accuracy and robustness of registration.

[0044] In one or more embodiments, in step 40, the method for image registration of each image in the preprocessed candidate lesion area image sequence according to the matching feature point pairs includes: performing image registration on each image in the candidate lesion area image sequence based on the global homography transformation and the local thin plate spline transformation according to the matching feature point pairs.

[0045] Exemplarily, for the unique non-rigid deformation of the digestive tract, an improvement is made on the basic framework of the original APAP stitching algorithm to obtain an image registration method that combines global homography transformation and local thin plate spline transformation, and the thin plate spline transformation is used to replace the traditional weighted homography model. The global homography transformation performs preliminary image registration. The homography matrix describes the global projection relationship between two images and is used for overall perspective transformation modeling. The global homography ensures the consistency of the main structure of the image. The calculation method of the homography is as follows:

[0046]

[0047] In the formula, represents the homography matrix, respectively represent the coordinates between the pairs of matching feature points. In the capsule endoscope images with non-rigid deformation or complex surface characteristics, the fitting effect of the local area may be insufficient, resulting in misalignment or deformation of local details. To address the above limitations, referring to the APAP algorithm, the thin plate spline transformation is used to locally optimize the image. After performing the global homography transformation to obtain the preliminary registered image, the preliminary registered image is divided into grids, and the local thin plate spline transformation is used to optimize the weighted homography, adjust the local area mapping, and perform local optimization on each grid area. The thin plate spline transformation is a transformation method based on control points, which can effectively handle non-rigid deformations and ensure more accurate alignment of details in the local area of the image. It is applicable to the non-rigid deformation of the digestive tract. The form of the thin plate spline transformation is as follows:

[0048]

[0049] In the formula, represents the thin plate spline kernel, and x, y represent the coordinates of the transformation points.

[0050] In one or more embodiments, before performing the image registration process, the stitching method includes: removing the incorrect matching points in the pairs of matching feature points. Exemplarily, to effectively remove the incorrect matching points and calculate the global homography transformation matrix more precisely, the MAGSAC (M-estimator SAmple Consensus) algorithm is used to estimate the homography matrix between the image sequences of the candidate lesion regions. This algorithm automatically evaluates the quality of the matching points through M-estimation, dynamically adjusts the weights, and effectively removes the incorrect matching points. The specific steps are as follows: First, randomly sample 4 pairs of matching points and calculate the homography transformation matrix between the image sequences. Then, bring all the pairs of matching feature points into the current , and calculate the reprojection error. If the error is less than the threshold, the point is considered an inlier; otherwise, it is an outlier. Next, count the number of inliers, and use the M-estimator to weight the inliers and outliers. Finally, repeat the iteration and select the homography matrix with the most inliers and the smallest error as the optimal model.

[0051] In one or more embodiments, in step S40, the method for performing image fusion on each image obtained by the image registration process to obtain the panoramic image of the candidate lesion region includes: Step S401: Perform projective transformation on each image obtained from image registration processing. Exemplarily, to prevent the destruction of image field consistency during subsequent fusion, projective transformation adjusts the spatial relationship of the image sequence to ensure smooth transitions between adjacent images and avoid unnatural effects after stitching. Available projective transformations include spherical projection, cylindrical projection, and fisheye projection, etc. The commonly used cylindrical projection projects the image onto a cylindrical surface and then unfolds it into a two-dimensional plane, which can better handle horizontal image stitching. At the same time, to avoid problems such as obvious seams and uneven brightness caused by exposure differences, exposure compensation is used to adjust the images to be stitched to keep the brightness balanced. The method is to multiply different images by different compensation coefficients. The cylindrical projection formula is as follows:

[0052]

[0053] In the formula, and represent the coordinates after projection, and represent the coordinates before projection, and represent the height and width of the image, represents the focal length of the camera.

[0054] Step S402: Stitch the images after projective transformation using an image fusion strategy to obtain a stitched image. Specifically, the purpose of image fusion operation is to smoothly combine multiple images into a seamless stitching result, avoid obvious seams or unnatural transitions, and ensure image consistency and visual naturalness. For medical endoscope images, it is most important to ensure the integrity of information, allowing certain seams to appear. Image fusion strategies mainly include weighted average fusion, multi-band fusion, and optimal seam, etc. Weighted average fusion is obtained by weighted averaging the pixel values in the overlapping area according to the weights of the stitched images; multi-band fusion decomposes the image into multiple bands through the Laplacian pyramid, and different bands are weighted and fused with different strategies; the optimal seam strategy finds the optimal seam path through dynamic programming algorithm or greedy algorithm.

[0055] Step S403: Evaluate the stitching quality of the stitched image. If the stitching quality evaluation is qualified, use the stitched image as the panoramic image of the final candidate lesion area. Exemplarily, after image fusion is completed, stitching quality evaluation is a key step to ensure that the stitched image can be used for assisting lesion detection. The indicators for stitching quality evaluation mainly include Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), and subjective perception, etc. The Peak Signal-to-Noise Ratio represents the ratio of the maximum intensity of the signal in the image to the noise. The Structural Similarity Index is an indicator for measuring the similarity in aspects such as structure, brightness, and contrast. Subjective perception is evaluated by manual visual inspection to assess the overall effect of the stitched image. Among them, the calculation process of each indicator for stitching quality evaluation is well-known technology in the field and will not be elaborated here.

[0056] Through the above step S40, each image is sequentially subjected to image fusion to obtain a panoramic image of the candidate lesion area.

[0057] In one or more embodiments, as Figure 2 shown, the stitching device for gastrointestinal magnetic control capsule endoscope images includes an image screening module 100, an image preprocessing module 200, a feature extraction module 300, an image registration module 400, and an image fusion module 500. The image screening module 100 is used to screen out the candidate lesion area image sequence from the original gastrointestinal image sequence collected by the magnetic control capsule endoscope; the image preprocessing module 200 is used to preprocess the candidate lesion area image sequence to obtain a preprocessed candidate lesion area image sequence; the feature extraction module 300 is used to extract features from the preprocessed candidate lesion area image sequence based on the pre-trained EfficientLoFTR improved model, and the EfficientLoFTR improved model is obtained by performing RepVGG reparameterized backbone network, fusing the feature pyramid network structure and the online attention structure on the basis of the original EfficientLoFTR model and combining depthwise separable convolution; the image registration module 400 is used to perform feature matching on the preprocessed candidate lesion area image sequence according to the extracted features to obtain matching feature point pairs, and perform image registration processing on each image of the preprocessed candidate lesion area image sequence according to the matching feature point pairs; the image fusion module 500 is used to perform image fusion on each image obtained by the image registration processing to obtain a panoramic image of the candidate lesion area.

[0058] Exemplarily, the image preprocessing module 200 includes an image cropping unit, a distortion correction unit, an image denoising unit, and a data augmentation unit. The image cropping unit is used to crop and remove the irrelevant black background areas and edge areas in each image of the candidate lesion area image sequence; the distortion correction unit uses a checkerboard image and Zhang Zhengyou's distortion correction for camera calibration to obtain the camera internal parameters and distortion coefficients, and corrects the radial distortion and tangential distortion of each cropped image; the image denoising unit performs denoising processing on each image after distortion correction based on a preset denoising algorithm; the data augmentation unit performs image enhancement and normalization operations on each image after denoising processing.

[0059] Exemplarily, the image registration module 400 is used to perform image registration processing on each image of the candidate lesion area image sequence based on the matching feature points, using global homography transformation and local thin plate spline transformation. Specifically, it includes: replacing the weighted homography model in the original APAP stitching algorithm with a thin plate spline transformation; using the replaced APAP stitching algorithm to perform global homography transformation on each image to obtain a preliminary registered image; dividing the preliminary registered image into several grid areas, and using the replaced APAP stitching algorithm to perform local optimization on each grid area.

[0060] Exemplarily, the image fusion module 500 is used to: perform projective transformation on each image obtained by image registration processing, splice the images after projective transformation using an image fusion strategy to obtain a spliced image, perform splicing quality evaluation on the spliced image, and if the splicing quality evaluation is qualified, use the spliced image as the panoramic image of the final candidate lesion area.

[0061] Among them, the more detailed working processes of each module of the splicing device for gastrointestinal magnetic control capsule endoscope images can refer to the relevant descriptions of the splicing method in the previous embodiment, and will not be elaborated here.

[0062] In one or more embodiments, a computer-readable storage medium stores a splicing program for gastrointestinal magnetic control capsule endoscope images. When the splicing program for gastrointestinal magnetic control capsule endoscope images is executed by a processor, it implements the splicing method for gastrointestinal magnetic control capsule endoscope images in Embodiment 1.

[0063] This embodiment also discloses a computer device. At the hardware level, such as Figure 3As shown, the computer device includes a processor 12, an internal bus 13, a network interface 14, and a computer-readable storage medium 11. The processor 12 reads the corresponding computer program from the computer-readable storage medium and then runs it, forming a request processing device at the logical level. Of course, in addition to the software implementation, one or more embodiments of this specification do not exclude other implementation manners, such as logic devices or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logical unit, but can also be hardware or a logic device. A splicing program for gastrointestinal magnetically controlled capsule endoscope images is stored on the computer-readable storage medium 11. When the splicing program for gastrointestinal magnetically controlled capsule endoscope images is executed by the processor, the above-mentioned method for splicing gastrointestinal magnetically controlled capsule endoscope images is implemented.

[0064] A computer-readable storage medium includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer-readable storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory, or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD), or other optical storage, magnetic cassette tapes, disk storage, quantum memory, graphene-based storage media, or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0065] The specific embodiments of the present application have been described in detail above. Although some embodiments have been shown and described, those skilled in the art should understand that these embodiments can be modified and perfected without departing from the principles and spirit of the present application as defined by the claims and their equivalents, and these modifications and perfections should also be within the protection scope of the present application.

Claims

1. A method for stitching images of a magnetically controlled capsule endoscope for the digestive tract, characterized in that, The splicing method includes: Screening out a candidate lesion area image sequence from the original gastrointestinal tract image sequence collected by the magnetic control capsule endoscope; Preprocessing the candidate lesion area image sequence to obtain a preprocessed candidate lesion area image sequence; Performing feature extraction on the preprocessed candidate lesion area image sequence based on a pre-trained Improved EfficientLoFTR model, where the Improved EfficientLoFTR model is obtained by performing RepVGG reparameterized backbone network, fusing feature pyramid network structure and online attention structure on the original EfficientLoFTR model and combining depthwise separable convolution; Performing feature matching on the preprocessed candidate lesion area image sequence according to the extracted features to obtain matching feature point pairs, and performing image registration processing on each image of the preprocessed candidate lesion area image sequence according to the matching feature point pairs; Performing image fusion on each image obtained by the image registration processing to obtain a panoramic image of the candidate lesion area.

2. The splicing method for the images of the magnetically controlled capsule endoscope for the digestive tract according to claim 1, wherein The method for preprocessing the candidate lesion area image sequence includes: Image cropping: Cropping and removing the irrelevant black background area and edge area in each image of the candidate lesion area image sequence; Distortion correction: Using a checkerboard image and Zhang Zhengyou's distortion correction for camera calibration to obtain the camera internal parameters and distortion coefficients, and correcting the radial distortion and tangential distortion of each cropped image; Image denoising: Performing denoising processing on each image after distortion correction based on a preset denoising algorithm; Data augmentation: Performing image enhancement and normalization operations on each image after denoising processing.

3. The method for stitching images of a magnetically controlled capsule endoscope for the digestive tract according to claim 1, wherein The method for performing image registration processing on each image of the preprocessed candidate lesion area image sequence according to the matching feature point pairs includes: Performing image registration processing on each image of the candidate lesion area image sequence based on global homography transformation and local thin plate spline transformation according to the matching feature point pairs.

4. The method for stitching images of a magnetically controlled capsule endoscope for the digestive tract according to claim 3, wherein Before performing the image registration processing, the splicing method includes: Removing the incorrect matching points in the matching feature point pairs.

5. The method for stitching images of a magnetically controlled capsule endoscope for the digestive tract according to claim 3, characterized in that, The method for performing image registration processing on each image of the candidate lesion area image sequence based on global homography transformation and local thin plate spline transformation according to the matching feature point pairs includes: Replacing the weighted homography model in the original APAP stitching algorithm with a thin plate spline transformation; Performing global homography transformation on each image using the replaced APAP stitching algorithm to obtain a preliminary registered image; Dividing the preliminary registered image into several grid regions, and performing local optimization on each grid region using the replaced APAP stitching algorithm.

6. The method for stitching images of a magnetically controlled capsule endoscope for the digestive tract according to claim 1, wherein, The method for performing image fusion on each image obtained by the image registration processing to obtain a panoramic image of the candidate lesion area includes: Performing projective transformation on each image obtained by the image registration processing; Stitching each image after projective transformation using an image fusion strategy to obtain a stitched image; Performing stitching quality evaluation on the stitched image, and if the stitching quality evaluation is qualified, using the stitched image as the final panoramic image of the candidate lesion area.

7. The method for stitching images of a magnetically controlled capsule endoscope for the digestive tract according to claim 1, wherein The splicing method further includes: Train the improved EfficientLoFTR model using a publicly available endoscopic dataset with depth information.

8. A splicing device for images of a magnetically controlled capsule endoscope for the digestive tract, characterized in that The splicing device includes: An image screening module for screening out a candidate lesion area image sequence from the original digestive tract image sequence collected by the magnetically controlled capsule endoscope; An image preprocessing module for preprocessing the candidate lesion area image sequence to obtain a preprocessed candidate lesion area image sequence; A feature extraction module for extracting features from the preprocessed candidate lesion area image sequence based on the pre-trained improved EfficientLoFTR model, where the improved EfficientLoFTR model is obtained by performing RepVGG reparameterized backbone network, fusing feature pyramid network structure and online linear attention structure on the original EfficientLoFTR model and combining depthwise separable convolution; An image registration module for performing feature matching on the preprocessed candidate lesion area image sequence according to the extracted features to obtain matching feature point pairs, and performing image registration processing on each image of the preprocessed candidate lesion area image sequence according to the matching feature point pairs; An image fusion module for performing image fusion on each image obtained by the image registration processing to obtain a panoramic image of the candidate lesion area.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a splicing program for digestive tract magnetically controlled capsule endoscope images, and when the splicing program for digestive tract magnetically controlled capsule endoscope images is executed by a processor, it implements the splicing method for digestive tract magnetically controlled capsule endoscope images according to any one of claims 1 to 7.

10. A computer device, characterized in that, The computer device includes a computer-readable storage medium, a processor, and a splicing program for digestive tract magnetically controlled capsule endoscope images stored in the computer-readable storage medium. When the splicing program for digestive tract magnetically controlled capsule endoscope images is executed by the processor, it implements the splicing method for digestive tract magnetically controlled capsule endoscope images according to any one of claims 1 to 7.

Images