Method, device and equipment for three-dimensional reconstruction of capsule endoscope cavity images

Abstract

The present application provides a method, device and equipment for three-dimensional reconstruction of images in a capsule endoscope cavity, which belongs to the field of image processing technology. The method includes: obtaining a first image set taken by a capsule endoscope in a cavity; enhancing the frame rate of the first image set based on a pre-trained depth perception model to form a second image set; eliminating unqualified images in the second image set based on corner point detection of each second image in the second image set to form a third image set; identifying the three-dimensional scene information of the capsule endoscope based on the third image set, and reconstructing the cavity in three dimensions based on the three-dimensional scene information. The present application can improve the quality of three-dimensional image reconstruction, and the three-dimensional reconstruction process does not need to rely on additional sensor equipment.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method, device and equipment for three-dimensional reconstruction of images in a capsule endoscope cavity. Background Art

[0002] In the image detection of human tissue cavities (such as the esophagus, stomach, intestines and other digestive tract cavities), although the traditional endoscopic inspection method can provide intuitive intracavitary images, it has limitations such as strong invasiveness, patient discomfort and inspection blind spots. As a non-invasive inspection method, capsule endoscopy aims to address the limitations of traditional endoscopic inspection. It allows patients to swallow capsules containing cameras to automatically capture images of the inside of the tissue cavity, greatly improving the comfort and range of intracavitary image detection.

[0003] However, capsule endoscopy technology still faces some key technical challenges in practical applications, the most prominent of which is the three-dimensional reconstruction of images in tissue cavities. Since capsule endoscopes rely on the natural peristalsis of the digestive tract to move, current technology cannot accurately track their position and posture in the digestive tract; in addition, capsule endoscopes can only provide two-dimensional image information during the shooting process and lack depth data; in addition, capsule endoscopes are limited by factors such as their size, battery capacity, and shooting frame rate, and the image quality obtained is often not ideal, further increasing the difficulty of three-dimensional reconstruction.

[0004] Existing 3D reconstruction methods such as CN109448041B and CN112261399B mostly rely on depth sensors or additional image acquisition devices to obtain depth information, but these methods not only increase the complexity and cost of the equipment, but may also cause additional discomfort to patients. In addition, although some 3D reconstruction algorithms based on image processing can solve the problem of missing depth information to a certain extent, due to the complexity and variability of the internal environment of the tissue cavity, these algorithms often find it difficult to achieve ideal reconstruction effects.

[0005] Therefore, it is of great application value to develop a method that can reconstruct the three-dimensional images in the capsule endoscope cavity using only the two-dimensional images taken by the capsule endoscope without relying on additional sensor equipment. Summary of the invention

[0006] The purpose of the present application is to provide a method, device and equipment for three-dimensional reconstruction of images in the cavity of a capsule endoscope, so as to solve at least one of the technical problems mentioned above.

[0007] In a first aspect, the present application provides a method for three-dimensional reconstruction of an image in a capsule endoscope cavity, the method comprising:

[0008] Acquire a first image set taken by the capsule endoscope in the cavity;

[0009] Performing frame rate enhancement on the first image set based on a pre-trained depth perception model to form a second image set;

[0010] Eliminate unqualified images in the second image set based on corner point detection of each second image in the second image set to form a third image set;

[0011] The three-dimensional scene information of the capsule endoscope is identified based on the third image set, and the cavity is three-dimensionally reconstructed based on the three-dimensional scene information.

[0012] Optionally, the frame rate of the first image set is enhanced based on the pre-trained depth perception model to form a second image set, including: identifying fusion feature information of each two previous and next frames of the first image in the first image set based on the depth perception model; generating an intermediate frame image between each two previous and next frames of the first image based on the fusion feature information, and inserting the intermediate frame image into the first image set to obtain the second image set.

[0013] Optionally, the depth perception model includes a convolutional neural network model, a depth estimation network model and an optical flow estimation network model. The step of identifying fusion feature information of each two preceding and succeeding first frames of the first image set based on the depth perception model includes:

[0014] Generate a corresponding feature map from each of the two preceding and following first images through the convolutional neural network model;

[0015] Generating a corresponding depth map and an optical flow vector from the feature map through the depth estimation network model and the optical flow estimation network model respectively;

[0016] The fusion feature information of each of the two preceding and succeeding frames of the first image is generated based on the feature map, the depth map and the optical flow vector.

[0017] Optionally, generating fused feature information of each two frames of the first image based on the feature map, the depth map and the optical flow vector includes: spatially transforming the feature map and the depth map of each two frames of the first image through the optical flow vector; and performing feature fusion on the spatially transformed feature map and depth map to generate the fused feature information.

[0018] Optionally, the depth perception model also includes a generative network model, which generates an intermediate frame image between each two previous and next first frames based on the fused feature information, including: performing upsampling and convolution operations on the fused feature information through the generative network model to generate an intermediate frame image with the same size as the each two previous and next first frames.

[0019] Optionally, the generative network model is formed by iteratively training a model loss function based on a weighted sum of a structural similarity loss function and an edge loss function, wherein the edge loss function is used to characterize the edge feature loss of the intermediate frame image and the first image of each of the two previous and next frames.

[0020] Optionally, the removing unqualified images in the second image set based on the corner point detection of each second image in the second image set to form the third image set includes:

[0021] Generate a structure tensor containing gradient information in horizontal and vertical directions for the second image in the second image set;

[0022] Identifying corner points of the second image based on a feature value of each pixel in the structure tensor;

[0023] The number of corner points in the second image is counted, and the second image with the number of corner points below a preset threshold is eliminated to form a third image set.

[0024] Optionally, identifying the corner points of the second image based on the eigenvalue of each pixel in the structure tensor includes: calculating a response function based on the eigenvalue of each pixel in the structure tensor, and determining whether the corresponding pixel is a corner point according to the size of each response function in the local area.

[0025] Optionally, the identifying the three-dimensional scene information of the capsule endoscope based on the third image set includes:

[0026] extracting feature points in a continuous third image sequence in the third image set by using a first real-time positioning and mapping model;

[0027] Matching feature points of the third image sequence to form a matching relationship;

[0028] Identifying pose data of a key image in the third image sequence;

[0029] Corresponding three-dimensional scene information is constructed based on the posture data, feature points and matching relationships.

[0030] Optionally, the three-dimensional scene information includes motion trajectory and scene point cloud data, and the three-dimensional reconstruction of the cavity based on the three-dimensional scene information includes: obtaining the correspondence between the three-dimensional scene information and each third image in the third image set; and reconstructing the third image set in three dimensions based on a second real-time positioning and map construction model and the correspondence between the third image and the three-dimensional scene information.

[0031] In a second aspect of the present application, a three-dimensional reconstruction device for capsule endoscope in-cavity images is provided, the device comprising:

[0032] A first image acquisition module, used to acquire a first image set taken by the capsule endoscope in the cavity;

[0033] A frame enhancement module, configured to enhance the frame rate of the first image set based on a pre-trained depth perception model to form a second image set;

[0034] A corner point processing module, configured to remove unqualified images in the second image set based on corner point detection of each second image in the second image set, so as to form a third image set;

[0035] A three-dimensional reconstruction module is used to identify the three-dimensional scene information of the capsule endoscope based on the third image set, and to perform three-dimensional reconstruction of the cavity based on the three-dimensional scene information.

[0036] In a third aspect of the present application, an electronic device is provided, comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute any of the methods described in the present application.

[0037] The method, device and equipment for three-dimensional reconstruction of images inside the capsule endoscope cavity in the present application, in view of the situation that the images captured by the capsule endoscope are limited by the size and power of the camera and result in a low shooting frame rate, the present application performs frame rate enhancement processing on the first image, so as to improve the frame rate of the three-dimensional reconstructed image; in addition, for the second image set formed after the frame rate enhancement, corner point detection is used to identify whether the second image is qualified, and unqualified images are eliminated to improve the overall image quality and the richness of feature information; finally, three-dimensional scene information is recognized based on the retained image, and the cavity is three-dimensionally reconstructed based on the three-dimensional scene information and the third image set, so that the three-dimensional reconstruction of the cavity can be achieved without relying on additional sensor equipment, and the quality of the three-dimensional reconstruction can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope of the present application.

[0039] Figure 1 A schematic diagram of a scene of a method for three-dimensional reconstruction of capsule endoscope in-cavity images in one embodiment;

[0040] Figure 2 is a flow chart of a method for three-dimensional reconstruction of capsule endoscope in-cavity images in one embodiment;

[0041] Figure 3 A flowchart of identifying fusion feature information of each of two preceding and succeeding first image frames in a first image set based on a depth perception model in one embodiment;

[0042] Figure 4 A flowchart of forming a third image set by removing unqualified images in the second image set based on corner point detection of each second image in the second image set in one embodiment;

[0043] Figure 5 A flowchart of identifying three-dimensional scene information of a capsule endoscope based on a third image set in one embodiment;

[0044] Figure 6 is a schematic structural diagram of a three-dimensional reconstruction device for images in a tissue cavity in one embodiment;

[0045] Figure 7 FIG. 1 is a schematic diagram of the structure of an electronic device in an embodiment. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0047] All terms (including technical and scientific terms) used in this application have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used here should be interpreted as having a meaning consistent with the context of this specification, and should not be interpreted in an idealized or overly rigid manner.

[0048] For example, the terms "first", "second", etc. used in this application may be used herein to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish a first element from another element. For example, without departing from the scope of this application, a first image may be referred to as a second image, and similarly, a second image may be referred to as a first image. Both the first image and the second image are images, but they are not necessarily the same image.

[0049] For another example, the terms “include”, “comprising”, etc. used in this application indicate the existence of features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0050] The three-dimensional reconstruction method of capsule endoscope cavity images in this application can be applied to Figure 1 In the scenario shown. Figure 1As shown, the user can take the capsule endoscope orally. When the capsule endoscope enters the user's tissue cavity, it will continuously capture images to form a first image set. The capsule endoscope transmits the captured first image set to an external receiver, and the external receiver transmits it to a local host computer. The local host computer uses a cloud computing cluster to perform frame rate enhancement processing and unqualified image elimination processing on the first image set, and finally forms a third image set. The third image set is subjected to three-dimensional scene information recognition, and finally completes three-dimensional reconstruction.

[0051] In one embodiment, Figure 2 As shown, a method for three-dimensional reconstruction of capsule endoscope cavity images is provided, the method comprising:

[0052] Step 210: Acquire a first image set captured by the capsule endoscope in the cavity.

[0053] In this embodiment, the cavity is an organ tissue structure of the human body, which may include one or more of the digestive tract cavities such as the esophagus, stomach, intestines, etc. The first image set includes a plurality of first image sequences captured at a preset frame rate, and at least one first image contains the corresponding information in the tissue cavity. The first image is an image captured by a shooting device that integrates relevant cameras for capturing information in the tissue cavity, and the shooting device may be a capsule endoscope. The first image may be an original image captured by a capsule endoscope, or it may be an image formed after the original image is pre-processed by image cropping, image enhancement, normalization, etc.

[0054] Specifically, the first image in the first image set may be the original image received by the capsule endoscope receiver from the capsule endoscope that has completed the shooting in the tissue cavity without deduplication and other operations. Taking the tissue cavity as the digestive tract as an example, under the premise of protecting privacy, these original images fully record the entire process of the capsule endoscope from entering the mouth to being discharged from the body, covering all image information. These original images also remove the initial image after the capsule endoscope is started to avoid interference caused by the static state during the startup process; at the same time, the images taken by the capsule after being discharged from the human body that are not related to the motion trajectory are also removed to ensure the validity and relevance of the final data set. The retained original images form the corresponding first image set. The first image set finally obtained will provide rich original materials for subsequent tissue cavity analysis and processing, and help to achieve a comprehensive evaluation and research of the tissue cavity.

[0055] Step 220: Perform frame rate enhancement on the first image set based on the pre-trained depth perception model to form a second image set.

[0056] In this embodiment, due to the limitation of the shooting frame rate of the capsule endoscope, it is necessary to enhance the frame rate before performing three-dimensional reconstruction to improve the quality of three-dimensional reconstruction. Among them, the electronic device pre-sets a relevant depth perception model, and enhances the frame rate of the first image set based on the depth perception model. The depth perception model can be a model formed by an interpolation algorithm, image replication, or image prediction based on a time series. The model can specifically be a combination of one or more models such as a perceptron model, a multi-layer perceptron model, a recurrent neural network model, a long short-term memory network model, a convolutional neural network model, and a radial basis function network model.

[0057] Specifically, a temporal interpolation method is used to insert one or more predicted images between each two first images (referred to as "previous and next frame images") in the first image set, thereby increasing the total number of frames in the image set to form a second image set. The predicted image can be generated based on the previous frame image and / or the next frame image. The number of frames of the predicted image inserted between each previous and next frame image can be determined according to the frame rate of the first image, so that the frame rate of the image finally generated can meet the frame rate requirement of 3D reconstruction.

[0058] The second image set includes the first image and the generated predicted image. The first image and the generated predicted image are both used as the second image, and are arranged in time sequence according to the corresponding shooting time of the first image and the interpolation sequence of the predicted image to form a second image sequence. The generated second image set contains more image information, which is helpful for subsequent feature extraction and 3D reconstruction.

[0059] In one embodiment, an intermediate frame image is generated based on every two preceding and succeeding first image frames in the first image set, and the intermediate frame image is inserted into the first image set to obtain the second image set.

[0060] In this embodiment, an electronic device (such as the local host computer mentioned above) can detect the image frame rate in the first image set. When the image frame rate exceeds the preset minimum frame rate, an intermediate frame image is generated based on each previous and next frame image, so that the frame rate of the second image set is doubled relative to the frame rate of the first image set.

[0061] The minimum frame rate is not less than half of the required frame rate for 3D reconstruction, so that the frame rate after frame rate enhancement can reach the required frame rate for 3D reconstruction. For example, the 3D reconstruction frame rate can be 30 frames per second (fps), and the minimum frame rate can be set to 15fps. When the image frame rate in the first image set exceeds 15fps, an intermediate frame image is generated based on each previous and next frame image.

[0062] Step 230 , based on the corner point detection of each second image in the second image set, unqualified images in the second image set are eliminated to form a third image set.

[0063] In this embodiment, the corner points of each second image in the second image set are identified, unqualified images in the second image set are eliminated based on the identified corner points, and a third image set is formed based on the retained second images. Corner points represent points in an image that change dramatically, usually located at the edge or junction of an object, and are of great significance for image matching and three-dimensional reconstruction.

[0064] For each second image in the second image set, the corners therein can be identified based on relevant image processing technology (such as corner detection algorithm). The magnitude of the pixel values ​​of each pixel point in the second image is used to identify the degree of change of each pixel value in a local area of ​​the second image, and the pixel points corresponding to the pixel values ​​that meet the degree of change are taken as corner points.

[0065] Based on the identified corner points, it is determined whether the corresponding second image is suitable for three-dimensional reconstruction. If it is determined that it is not suitable for three-dimensional image reconstruction, it is said that the second image is unqualified and is deleted. Among them, the set of retained second images constitutes a third image set. The retained second images are used as third images, and each third image is arranged according to its time sequence in the second image set to form a third image sequence.

[0066] Specifically, the electronic device pre-constructs corner point determination conditions and image qualification conditions, identifies corner points in the second image according to the corner point determination conditions, detects whether the corner points in the second image meet the qualification conditions, deletes them if they do not meet the qualification conditions, and retains them if they meet the qualification conditions. The corner point determination conditions and image qualification conditions can be set according to the characteristics of the tissue cavity. The image qualification conditions can include one or more of the number of corner points, the position distribution characteristics of the corner points in the image, and the like.

[0067] In one embodiment, the number of corner points in the unqualified image is lower than a preset threshold, and the preset threshold may be any set quantity threshold suitable for judging whether the second image is qualified.

[0068] Step 240: Identify the three-dimensional scene information of the capsule endoscope based on the third image set, and reconstruct the cavity in three dimensions based on the three-dimensional scene information.

[0069] The three-dimensional scene information is used to characterize the motion state and related parameters of the capsule endoscope in the tissue cavity, which constitutes a comprehensive description of the motion state of the capsule endoscope in the three-dimensional space of the tissue cavity. The three-dimensional scene information may specifically include one or more of the motion trajectory of the capsule endoscope, the camera's position data, and the scene point cloud data. The electronic device can extract the three-dimensional scene information corresponding to each frame of the third image from the third image set through a preset three-dimensional reconstruction model, and perform three-dimensional reconstruction to generate a corresponding three-dimensional reconstructed video. The three-dimensional reconstructed video can be a video used to reflect the three-dimensional data of the corresponding tissue cavity, and the video format can be MP4 format.

[0070] Specifically, the electronic device is pre-set with a corresponding 3D reconstruction model, which may be a combination of one or more of a SLAM model, a surface reconstruction algorithm, a volume reconstruction algorithm or a hybrid algorithm. After generating 3D scene information through the 3D reconstruction model, the 3D scene information and the third image set are further combined, and the 3D reconstruction model is run to obtain a 3D reconstruction result, and a corresponding 3D reconstruction video is output.

[0071] The three-dimensional reconstruction method of the image inside the capsule endoscope cavity in the present application, in view of the situation that the image captured by the capsule endoscope is limited by the camera size and power and leads to a low shooting frame rate, the present application performs frame rate enhancement processing on the first image, so as to improve the frame rate of the three-dimensional reconstructed image; in addition, for the second image set formed after the frame rate enhancement, corner point detection is used to identify whether the second image is qualified, and the unqualified image is eliminated to improve the overall image quality and the richness of feature information; finally, the three-dimensional scene information is recognized based on the retained image, and the cavity is three-dimensionally reconstructed based on the three-dimensional scene information and the third image set, so that the three-dimensional reconstruction of the cavity can be realized without relying on additional sensor equipment, and the quality of the three-dimensional reconstruction can be improved.

[0072] In one embodiment, step 220 includes: identifying fusion feature information of each two frames of the first image in the first image set based on the depth perception model; generating an intermediate frame image between each two frames of the first image based on the fusion feature information, and inserting the intermediate frame image into the first image set to obtain a second image set.

[0073] In this embodiment, the fused feature information integrates the feature information of the previous and next frame images to form a more comprehensive and accurate feature representation. The fused feature information reflects one or more features such as motion, depth and scene in the corresponding previous and next two frame images.

[0074] Optionally, the depth perception model can be used to identify each first image frame and extract features to obtain individual feature information of each frame image. Feature matching is then continued for the individual feature information between the previous and next frame images, and feature fusion is performed according to the individual feature information between the previous and next frame images that have a matching relationship, to obtain corresponding fused feature information.

[0075] After obtaining the fusion feature information, the depth perception model is used to generate intermediate frame images based on the fusion feature information and the corresponding front and back frame images. After all intermediate frame images are generated, the generated intermediate frame images are inserted between the corresponding front and back frame images to form a new image sequence, and each frame image in the image sequence is used as the second image to constitute a second image set.

[0076] Optionally, the depth perception model may include a combination of one or more models. In one or more of the above-mentioned feature extraction, feature matching, feature fusion and intermediate frame generation, the same model may be used for processing, or different models may be used for processing. For example, the depth perception model includes a convolutional neural network model, a depth estimation network model and an optical flow estimation network model, and may also include a generation network model. The model used in the above-mentioned feature extraction process is a convolutional neural network model; the model used in the feature matching and fusion process is a depth estimation network model and an optical flow estimation network model; the model used in the intermediate frame generation process is a generation network model. By using different models in different links, the quality of the feature information generated in each link can be improved, thereby ultimately improving the quality of the intermediate frame generation.

[0077] In one embodiment, Figure 3 As shown, identifying fusion feature information of each of the two preceding and succeeding first image frames in the first image set based on the depth perception model includes:

[0078] Step 310: Generate corresponding feature maps from each of the two previous and next frames of the first image through a convolutional neural network model.

[0079] In this embodiment, a feature map is extracted for each frame of the first image. The convolutional neural network can be any pre-set network model suitable for performing image frame processing, such as a VGG convolutional neural network, a ResNet convolutional neural network, etc. The feature map contains spatial and contextual information, which can effectively describe the changes in the scene.

[0080] Optionally, the convolutional neural network specifically uses a 3×3 convolutional layer as a basic component. Such a design can effectively capture local features of the image while maintaining a relatively small number of parameters, which is conducive to rapid training and deployment of the model.

[0081] When the first image is sent to the network as input data, the first 3×3 convolutional layer begins to perform preliminary feature extraction on the image. This layer traverses the entire image by sliding the convolution kernel, calculating the dot product between the convolution kernel and the local area of ​​the image, generating a series of feature maps, and producing a preliminary feature map.

[0082] Next, in order to improve the stability of model training and accelerate the convergence process, a batch normalization layer is immediately connected after each convolution layer. This layer normalizes the output of each neuron in the preliminary feature map so that the output normalized feature map maintains a stable distribution during the training process, effectively alleviating the problems of gradient vanishing and gradient exploding, and also making the model more robust to the selection of initialization parameters.

[0083] Then, the ReLU activation function (Rectified Linear Unit) is applied to the normalized feature map. After completing the above operations, the normalized feature map will be sent to the downsampling layer for spatial dimension reduction. You can choose the maximum pooling layer or the stride convolution layer as the downsampling method. The maximum pooling layer represents the features of the area by selecting the maximum value in the pooling window, thereby reducing the size of the feature map while retaining the most significant feature information. The stride convolution layer achieves downsampling by increasing the step size of the convolution operation, which can also reduce the resolution of the feature map while extracting more critical feature representations.

[0084] After multiple cycles of convolution, normalization, activation, and downsampling, the network gradually extracts higher-level semantic features from low-level edge and texture features. Ultimately, the feature map generated after this series of operations is not only significantly reduced in size, but also more concentrated and refined in information, providing good feature support for subsequent image analysis, recognition, or other tasks (such as depth estimation and optical flow calculation).

[0085] Step 320, generating corresponding depth maps and optical flow vectors from the feature maps through the depth estimation network model and the optical flow estimation network model respectively.

[0086] After obtaining the feature map of each first image, it is input into a preset depth estimation network model to generate a depth map of each first image; it is also input into a preset optical flow estimation network model to generate an optical flow vector corresponding to the first image. The depth map provides the depth information of each pixel in the first image; the optical flow vector describes the motion information of the pixel in the image over time.

[0087] The depth estimation network model can be any suitable network model, such as a U-Net network model. In the U-Net network model, the input feature map is first processed layer by layer through a series of convolutional layers, and each layer is responsible for extracting features at different levels. At the same time, the high-level features of the encoder part are directly connected to the low-level features of the decoder part using jump connections to ensure that the network can take into account both global context information and local detail information during the depth estimation process, thereby improving the accuracy and detail expression of the depth map.

[0088] Furthermore, the depth estimation network is designed to be able to fuse the motion information in the feature map. By analyzing the temporal changes in the feature map, the features related to the object motion are extracted and combined with the static depth features. The generated depth map not only reflects the static structure of the scene, but also captures the depth changes caused by the object motion, further improving the accuracy and robustness of the depth estimation.

[0089] After the above processing steps, the network finally outputs a depth map, in which each pixel value represents the depth information of the corresponding position in the scene. The output depth map not only has high resolution and fine details, but also can accurately reflect the three-dimensional structure and dynamic changes of the scene, providing key information for the subsequent intermediate frame generation.

[0090] The optical flow estimation network can be a network based on the PWC-Net architecture. After the depth map of each previous and next frame image is input into the network, the network analyzes the motion information in the feature map through multiple convolutional layers and a special optical flow feature extraction module to extract the dynamic changes of each part of the scene. During the processing, the network generates an optical flow vector for each pixel. The optical flow vector describes the displacement of the corresponding pixels in each previous and next frame image in time, reflecting the motion relationship between the previous and next frame images. In the process of optical flow calculation, the previously generated depth information can be further combined to ensure that the motion in the scene can be accurately modeled when generating the interpolated frame. This combination makes the optical flow vector more realistic and improves the quality of interpolation.

[0091] Specifically, the extracted feature maps and depth maps of the previous and next frames are sent as input to the PWC-Net optical flow estimation network. The network first constructs a cost volume to preliminarily estimate the optical flow vector by comparing the similarities of the feature maps and depth maps of the previous and next frames under different displacements. Subsequently, the optical flow vector is refined layer by layer using a pyramid structure. Each layer uses the optical flow estimation results of the previous layer to distort the feature map and depth map, so that the feature map and depth map are more aligned, thereby improving the accuracy of the optical flow estimation. After the above processing, the PWC-Net optical flow estimation network finally outputs an optical flow vector, where each vector represents the displacement of the corresponding pixel in the image from the previous frame to the next frame. These optical flow vectors not only reflect the motion trajectory of the object, but also take into account the deep structure of the scene, making the optical flow vector more realistic and reliable.

[0092] Step 330: Generate fusion feature information of each of the two previous and next frames of the first image based on the feature map, the depth map, and the optical flow vector.

[0093] In this embodiment, after obtaining the feature map, depth map and optical flow vector of each frame image, the feature map, depth map and optical flow vector corresponding to the previous and next frame images can be feature fused to finally obtain the fused feature information of each two previous and next frames of the first image.

[0094] Specifically, the feature map and the depth map of each two frames of the first image are spatially transformed by using the optical flow vector; the feature map and the depth map after the spatial transformation are feature fused to generate fused feature information.

[0095] Spatial transformation can use affine transformation, bilinear interpolation or any other suitable transformation method to ensure that the transformed feature map and depth map are coherently aligned in space and depth, so as to better capture the motion information and pixel displacement in the scene. Specifically, the feature map and depth map are adjusted at the pixel level through the optical flow vector, so that the feature map and depth map of the previous and next frame images are matched as much as possible in space, providing a basis for subsequent feature fusion.

[0096] The transformed feature map and depth map are fused through the self-attention mechanism and / or convolutional layers. The self-attention mechanism can effectively capture the global contextual information in the scene and help the model better understand the correlation between various regions; while the convolutional layer helps to extract local detail features and retain high-resolution detail information. In this fusion process, the network combines the motion, depth, and scene features of the previous and next frames to generate a fused feature information that contains motion dynamics and scene structure. This process aims to capture the global dynamics of the scene while retaining local detail information, thereby laying a solid foundation for generating high-quality interpolated frames. Finally, the fused feature information will be used as the input of the intermediate frame generation network to ensure that the subsequent intermediate frame generation can smoothly and naturally transition the changes between the previous and next frames, improving the effect and visual experience of video interpolation.

[0097] In one embodiment, the depth perception model also includes a generative network model, which generates an intermediate frame image between each two previous and next frames of the first image based on the fused feature information, including: upsampling and convolution operations are performed on the fused feature information through the generative network model to generate an intermediate frame image of the same size as each two previous and next frames of the first image.

[0098] The generative network model can be a network model trained based on any suitable architecture such as U-Net, GAN, etc. The generative network model gradually reconstructs the intermediate frames through a series of upsampling and convolution layers. In this process, the generative network model first performs spatial expansion through a deconvolution layer (transposed convolution) to gradually restore the size of the image so that the generated intermediate frame is consistent with the size of the previous and next frames. At the same time, the generative network model uses skip connections to combine features from different layers. This strategy ensures that important detail information is retained during the upsampling process. The skip connection combines the high-resolution features of the low layer with the abstract features of the high layer, so that the generated interpolated frame can better match the previous and next frames in texture and structure. In this way, the generative network model can not only reconstruct the basic structure of the image, but also carefully preserve the texture and details of the image, thereby generating natural and smooth intermediate frames. The final output intermediate frame image will help improve the coherence and visual quality of the video and enhance the viewing experience.

[0099] In one embodiment, the network model is generated by iteratively training the model loss function based on the weighted sum of the structural similarity loss function and the edge loss function, wherein the edge loss function is used to characterize the edge feature loss of the intermediate frame image and each of the first two frames before and after.

[0100] Specifically, the intermediate frame generation network model is formed by iterative training according to the model loss function Capsule_Loss=w1×SSIM_Loss+w2×Edge_Loss, where w1 and w2 represent weight coefficients, SSIM_Loss represents the structural similarity loss function, and Edge_Loss represents the edge loss function.

[0101] In generating an intermediate frame image between each two frames of the first image based on the fusion feature information, the method further includes: training the generation network model to generate a trained intermediate frame generation network model. The training process includes: obtaining a sample set, the sample set includes multiple frames of real images and real intermediate frame images corresponding to each two frames of real images; iteratively training the generation network model using the real images in the sample set, and adjusting the model parameters during each iterative training process to obtain the intermediate frame generation network model after the iterative training is completed.

[0102] Before starting iterative training, the relevant parameters in the generative network are initialized. The process of updating the parameters in each iteration is as follows: input the real images of the previous and next two frames into the generative network, perform forward propagation, and obtain the generated intermediate frame image; calculate the difference between the generated intermediate frame image and the real intermediate frame image based on the above-designed model loss function to obtain the loss value; perform back propagation based on the loss value, calculate the gradient of each layer in the network, and use the optimization algorithm (such as Adam, SGD, etc.) to update the parameters in the generative network according to the gradient.

[0103] The parameters in the generated network in the current iteration process are the latest parameters after the most recent update. Repeat the above iteration process. When the number of iterations reaches the preset iteration threshold, the iteration is terminated, and the generated network after the iterative training is used as the above generated network model. The iteration threshold can be any appropriate number threshold set in advance. When the number of iterations reaches the preset number threshold, the relevant parameters in the generated network are basically stable, and the calculated loss value can be less than the preset value threshold.

[0104] In the above model loss function Capsule_Loss, the structural similarity loss function SSIM_Loss=1-SSIM(I,G).

[0105] .

[0106] I represents the real intermediate frame image, and G represents the generated intermediate frame image. and are the average pixel values ​​of image I and image G respectively. and are the variances of image I and image G respectively. is the covariance of image I and image G. C1 and C2 are preset constant values ​​to avoid the denominator being zero.

[0107] In the model loss function Capsule_Loss, the edge loss function Edge_Loss is calculated according to the following formula:

[0108] .

[0109] Among them, Edges I =Canny(I),Edges G =Canny(G). Canny(I) and Canny(G) represent the edge image generated by extracting the real intermediate frame image I using Canny edge detection, and the edge image generated by extracting the generated intermediate frame image G, respectively. N represents the total number of pixels in the edge image. I [j] and Edges G [j] are the pixel values ​​of the j-th pixel in the edge image generated by the real intermediate frame image I and the edge image generated by the generated intermediate frame image G, respectively.

[0110] Unlike most other frame rate improvement tasks, for the frame supplementation task of the first image in the tissue cavity captured by devices such as capsule endoscopes, in addition to maintaining the continuity of the video, it is also necessary to restore the boundaries of the organs and the structure of the lesions as much as possible. Based on this, the present application starts from the edge features of the image, the fidelity of the medical structure, and the detail recovery of the biological tissue, while taking into account that retaining the global structural information of the image in medical images is more important than simple pixel values. On the basis of the original loss function, an edge loss function that is very effective for the first image frame supplementation is added, and the model loss function Capsule_Loss is constructed by combining the structural similarity loss function with the edge loss function. By using the above-mentioned model loss function Capsule_Loss to generate iterative training of the network, the key medical structures and detail information in the intermediate frame images can be retained and restored to the maximum extent during the frame supplementation process, thereby significantly improving the quality and diagnostic value of the intermediate frame images.

[0111] In one embodiment, Figure 4 As shown, step 230 includes:

[0112] Step 410 : Generate a structure tensor containing gradient information in the horizontal direction and the vertical direction for the second image in the second image set.

[0113] In this embodiment, the structure tensor is a second-order matrix for organizing image pixels. For each frame of the second image, the gradient (partial derivative) of each pixel in the horizontal and vertical directions can be calculated, and the gradient can be calculated by the Sobel operator, the Prewitt operator or other gradient algorithms. After calculating the gradient of each pixel, a structure tensor containing gradient information can be formed based on the gradient. The structure tensor represents the gradient feature of the corresponding pixel in the second image, which includes the gradient direction and the gradient magnitude.

[0114] Specifically, step 410 includes: converting the second image into a grayscale image; calculating the gradient I of the grayscale image in the horizontal direction; x and the vertical gradient I y ; Based on the formula The structure tensor M is calculated.

[0115] Among them, the structure tensor M can be expressed as a Harris matrix. x and I y Energy-efficiently highlight edges and corners in images. x and I y They represent the gradients of the pixels in the image in the x and y directions, respectively, reflecting the brightness changes of the image; x 2 Indicates the degree of change in the x direction, I y 2 Indicates the degree of change in the y direction; I x ×I y Represents the correlation between the x and y directions, reflecting the common changes in the two directions.

[0116] Step 420: identifying corner points of the second image based on the feature value of each pixel in the structure tensor.

[0117] Specifically, a Gaussian filter is used to smooth each element of the structure tensor to reduce the impact of noise on corner point detection. The eigenvalues ​​of the structure tensor (i.e., the eigenvalues ​​of the corresponding pixels) are calculated by eigenvalue decomposition or directly. There may be two eigenvalues, denoted as r1 and r2, which reflect the degree of change in the local area.

[0118] In one embodiment, a response function is calculated based on the eigenvalue of each pixel in the structure tensor, and whether the corresponding pixel belongs to a corner point is determined according to the size of each response function in the local area.

[0119] Among them, the response function R can be calculated according to the following formula:

[0120] .

[0121] k is a preset coefficient. After obtaining the response function of each structure tensor, the size of the response function corresponding to all pixels of the second image in the local area can be detected, and based on the size of the response function in the local area, it is determined whether the corresponding pixel belongs to a corner point.

[0122] Specifically, a preset intensity threshold can be set to detect whether the value of each response function exceeds the intensity threshold. For the response function that exceeds the intensity threshold, it is further detected whether the response function belongs to a local maximum in its corresponding local area. If the response function that exceeds the intensity threshold also belongs to a local maximum, the pixel point corresponding to the response function is used as the corner point of the second image.

[0123] Among them, the local maximum value indicates that there are significant corner point features around the point (i.e., the local area), and the local area where the response function is located can be divided into regions with the pixel point X0 corresponding to the response function as the center, and the area surrounded by the region division is the corresponding local area. For example, all pixels whose pixel interval from the pixel point X0 is less than the preset pixel interval threshold can be regarded as local areas. The pixel interval threshold can be a preset value of any suitable size, such as 5, 8, 10, etc. Taking the pixel interval threshold as 10 as an example, for a certain response function R0, with the pixel point X0 where the response function is located as the center, in the local area composed of multiple pixels whose pixel interval from the pixel point X0 does not exceed 10, the response function R0 is the maximum value in the local area, and its value exceeds the preset intensity threshold, then the pixel point X0 can be regarded as a corner point.

[0124] Step 430 , counting the number of corner points in the second image, and eliminating the second image with a number of corner points below a preset threshold, to form a third image set.

[0125] In this embodiment, for the identified corner points, the number of corner points in the second image of the same frame can be counted to detect whether the number of corner points is sufficient. If the number of corner points is too low, the second image is marked as an unqualified image and eliminated. If the number is sufficient, it is retained. A number threshold is pre-set in the electronic device, and the size of the number threshold of the corner points can be determined according to the characteristics of the tissue cavity. Compare whether the number of corner points in the second image exceeds the number threshold. If not, it means that the number of corner points is insufficient and it is eliminated as an unqualified image.

[0126] The second images retained in the second image set are the third images, and all the retained third images form a third image set to improve the overall image quality and the richness of feature information.

[0127] In one embodiment, Figure 5 As shown, the three-dimensional scene information of the capsule endoscope is identified based on the third image set, including:

[0128] Step 510: extract feature points in a continuous third image sequence in a third image set using a first real-time positioning and mapping model.

[0129] In this embodiment, the feature point is a pixel point or area in the image that is unique and stable, and can maintain relatively consistent characteristics in different images. The first real-time positioning and mapping model (first SLAM model) can be any suitable model for three-dimensional scene information recognition. Optionally, the first SLAM model can be an ORB-SLAM3 (third generation real-time SLAM based on ORB features) model, and the ORB (Oriented FAST and RotatedBRIEF) algorithm in the first SLAM model can be used to identify feature points of each frame of the third image.

[0130] In one embodiment, before step 510, the method further includes: acquiring system parameters corresponding to the first image set.

[0131] Among them, the system parameters include model parameters of relevant models used to generate three-dimensional scene information (such as the first real-time positioning and map construction model, the second real-time positioning and map construction model), parameters related to feature points, the video type of the first image set (such as monocular, stereo or with IMU), and camera parameters.

[0132] The camera parameters include the camera's internal and external parameters. The internal parameters may include focal length, principal point position, etc. Specifically, a set of calibration images (such as checkerboard images) for calibration can be taken in advance using the capsule endoscope to be tested, and processed through the OpenCV library to obtain the internal parameters of the capsule endoscope, which include but are not limited to one or more of the camera focal length, principal point coordinates, and distortion coefficients.

[0133] Parameters related to feature points may include descriptor types to meet the feature extraction requirements of different scenarios. Different video types can choose different system parameters. For example, if the video is in monocular mode shot by a capsule endoscope, only the parameters of a single camera need to be set, while in stereo mode, the relative position and calibration information of the two cameras need to be configured to achieve depth perception. If an IMU is used, the relevant parameters of the IMU need to be configured to enhance the system's ability to track motion. By configuring different system parameters according to different scenarios, the positioning and mapping accuracy of the relevant models can be improved.

[0134] Step 520: Match the feature points of the third image sequence to form a matching relationship.

[0135] Step 530 : Identify the pose data of the key image in the third image sequence.

[0136] Step 540, construct corresponding three-dimensional scene information based on the posture data, feature points and matching relationships.

[0137] Among them, for each frame of the third image that is identified, the feature points representing the same features in each of the two frames of the third image before and after are matched to obtain the matching relationship between the feature points. Specifically, the descriptor for the feature point is calculated, and the descriptor is used for feature point matching; according to the descriptor, the feature points of adjacent third images are matched through the FLANN (Fast Library for ApproximateNearest Neighbors) algorithm to form a matching relationship.

[0138] A descriptor is a high-dimensional vector that represents the image information around a feature point. Descriptors can be generated using information such as grayscale values ​​and gradient direction histograms around feature points. A matching algorithm is used to compare the descriptors of two sets of feature points to find matching feature point pairs and form a matching relationship. The matching algorithm can be based on different distance metrics, such as Euclidean distance, Hamming distance, FLANN algorithm, etc. The FLANN algorithm can quickly find the nearest neighbor by establishing an efficient index structure, thereby significantly improving the speed and accuracy of feature point matching. This algorithm is particularly suitable for large-scale feature sets and can reduce calculation time while ensuring matching quality. The key feature points extracted by the ORB algorithm are combined with the FLANN algorithm to not only improve the real-time performance of the system, but also improve the accuracy of feature matching, laying a solid foundation for subsequent pose estimation and map construction.

[0139] For pose data: For key images, using the matched feature point pairs and the camera's external and internal parameters, the PnP (Perspective-n-Point) algorithm or the ICP (Iterative Closest Point) algorithm is used to estimate the pose (i.e., position and posture) of the image relative to a global reference coordinate system (such as the world coordinate system). Key images represent third images that contain enough feature points and have a good match with other images, and they play a key role in 3D reconstruction. Pose data includes rotation data and translation data. Specifically, the system can determine the matching relationship between each third image and each key image. The 3D point information in the world coordinate system is extracted from the matched feature points and associated with the corresponding feature points in the current frame. Then, this information is input into the PnP algorithm, solved using optimization methods (such as EPnP or UPnP), and finally the rotation matrix and translation vector of the camera are output. Through this estimation, ORB-SLAM3 can update the pose information in real time, providing the necessary spatial reference for subsequent 3D reconstruction.

[0140] Regarding the motion trajectory: by accumulating the changes in the camera's posture data between each pair of adjacent third images, the motion trajectory of the camera during the shooting process of the entire third image set is obtained, that is, the motion trajectory of the camera is obtained.

[0141] Scene point cloud data construction: By detecting and tracking feature points, the ORB-SLAM3 model can determine the matching relationship between the third image of the current frame and the third image of the previous frame, and select a new key frame third image (i.e., key image). In each newly selected key image, feature points are extracted and corresponding three-dimensional scene point cloud data is generated. The generation process of these scene point cloud data depends on the camera pose, and this information is used to reconstruct the feature points into the world coordinate system to achieve position determination in three-dimensional space. Subsequently, the ORB-SLAM3 model continuously updates and optimizes the map through local optimization and loop detection technology to ensure global consistency. Local optimization improves the accuracy of scene point cloud data and camera pose by minimizing reprojection errors; while loop detection helps correct possible drift by identifying previously visited locations, thereby enhancing the stability and accuracy of the map.

[0142] Taking the camera as a capsule endoscope as an example, during this process, the ORB-SLAM3 model will output the 3D scene information of the capsule and record it in the "trajectory.txt" file, which contains the camera pose data of each third image. At the same time, the scene point cloud data will also be generated and saved in the "point cloud.ply" file.

[0143] In one embodiment, the three-dimensional scene information includes a motion trajectory and scene point cloud data. The cavity is reconstructed in three dimensions based on the three-dimensional scene information, including: obtaining a correspondence between the three-dimensional scene information and each third image in the third image set; and reconstructing the third image set in three dimensions based on the second real-time positioning and map building model and the correspondence between the third image and the three-dimensional scene information.

[0144] In this embodiment, the second real-time positioning and mapping model (second SLAM model) can be any model suitable for three-dimensional reconstruction. Optionally, the second SLAM model can be a GO-SLAM (Graph-based Online Simultaneous Localization and Mapping) model.

[0145] For the generated three-dimensional scene information, the corresponding relationship between the generated motion trajectory and each frame of the third image is constructed. Based on the formed corresponding relationship, the motion trajectory and the third image set are input into the GO-SLAM (Graph-basedOnline Simultaneous Localization and Mapping) algorithm for three-dimensional reconstruction, and the corresponding three-dimensional reconstruction results are output. The GO-SLAM algorithm is an algorithm based on the dense visual SLAM framework of deep learning, which can realize real-time reconstruction of postures and dense 3D scenes.

[0146] Specifically, each frame of the third image and the motion trajectory are matched one by one to construct a dictionary file containing timestamps, third images, and motion trajectories. In this process, the motion trajectory data is first parsed to extract the timestamps and corresponding camera pose information of each third image. Then, the third image set is traversed, the timestamps of each frame of the third image are recorded, and matched with the corresponding motion trajectory data. In this way, the generated dictionary file will use the timestamp as a keyword to map each timestamp to the corresponding third image and motion trajectory. This structured data preparation not only facilitates subsequent analysis and visualization, but also provides a basis for further video reconstruction. Finally, the dictionary file will be used as the input of the GO-SLAM algorithm to fully prepare for subsequent steps and ensure the efficiency and accuracy of the processing flow. When performing three-dimensional reconstruction, some third images in the third image set of the part that needs to be reconstructed in three dimensions can be selected (for example, the video frames of the capsule entering the colon and exiting the colon are selected), and after processing the corresponding parameters, the GO-SLAM project is started to reconstruct the three-dimensional scene of the tissue cavity part, and the corresponding three-dimensional reconstruction scene and the corresponding MP4 video are output.

[0147] The three-dimensional reconstruction method of the capsule endoscope cavity image proposed in the present application uses relevant technologies in the fields of computer vision and deep learning. It can calculate the camera's movement trajectory in the body without using any sensors, only through the original image (first image) in the tissue cavity (such as the digestive tract) taken by the capsule endoscope camera. At the same time, the calculated movement trajectory and the original image are used to perform three-dimensional reconstruction of the scene of a partial area of ​​the tissue cavity without using any depth information. This solves the problem that the capsule endoscope camera cannot load the corresponding module to capture images with depth information due to limitations such as its own size and battery capacity, and thus cannot be modeled using traditional three-dimensional reconstruction methods.

[0148] In one embodiment, Figure 6 As shown, a three-dimensional reconstruction device for images in a tissue cavity is provided, the device comprising:

[0149] A first image acquisition module 610 is used to acquire a first image set taken by the capsule endoscope in the cavity;

[0150] A frame enhancement module 620, configured to enhance the frame rate of the first image set based on a pre-trained depth perception model to form a second image set;

[0151] A corner point processing module 630, configured to remove unqualified images in the second image set based on corner point detection of each second image in the second image set, so as to form a third image set;

[0152] The three-dimensional reconstruction module 640 is used to identify the three-dimensional scene information of the capsule endoscope based on the third image set, and perform three-dimensional reconstruction of the cavity based on the three-dimensional scene information.

[0153] In one embodiment, the frame adding module 620 is also used to identify the fusion feature information of each two frames of the first image in the first image set based on the depth perception model; generate an intermediate frame image between each two frames of the first image based on the fusion feature information, insert the intermediate frame image into the first image set, and obtain the second image set.

[0154] In one embodiment, the frame adding module 620 is also used to generate a corresponding feature map from each of the two previous and next frames of the first image through a convolutional neural network model; generate a corresponding depth map and optical flow vector from the feature map through a depth estimation network model and an optical flow estimation network model, respectively; and generate fused feature information of each of the two previous and next frames of the first image based on the feature map, the depth map and the optical flow vector.

[0155] In one embodiment, the frame adding module 620 is further used to perform spatial transformation on the feature map and depth map of each two frames of the first image through optical flow vectors; perform feature fusion on the feature map and depth map after spatial transformation to generate fusion feature information.

[0156] In one embodiment, the frame adding module 620 is further used to perform upsampling and convolution operations on the fused feature information by generating a network model to generate an intermediate frame image with the same size as the first image of each of the two frames before and after.

[0157] In one embodiment, the frame adding module 620 is also used to perform iterative training on the generated network model. The generated network model is formed by iterative training of the model loss function based on the weighted sum of the structural similarity loss function and the edge loss function, wherein the edge loss function is used to characterize the edge feature loss of the intermediate frame image and each of the first two frames before and after.

[0158] In one embodiment, the corner point processing module 630 is also used to generate a structure tensor containing gradient information in the horizontal and vertical directions for the second image in the second image set; identify the corner points of the second image based on the eigenvalue of each pixel in the structure tensor; count the number of corner points in the second image, and eliminate the second image whose number of corner points is lower than a preset threshold to form a third image set.

[0159] In one embodiment, the corner point processing module 630 is further used to calculate a response function based on the eigenvalue of each pixel point in the structure tensor, and determine whether the corresponding pixel point is a corner point according to the size of each response function in the local area.

[0160] In one embodiment, the three-dimensional scene information includes motion trajectory and scene point cloud data.

[0161] The three-dimensional reconstruction module 640 is also used to extract feature points in a continuous third image sequence in a third image set through the first real-time positioning and map construction model; match the feature points of the third image sequence to form a matching relationship; identify the posture data of the key image in the third image sequence; and construct corresponding three-dimensional scene information based on the posture data, feature points and matching relationship.

[0162] In one embodiment, the three-dimensional reconstruction module 640 is also used to obtain the correspondence between the three-dimensional scene information and each third image in the third image set; based on the second real-time positioning and map construction model and the correspondence between the third image and the three-dimensional scene information, perform three-dimensional reconstruction on the third image set.

[0163] The three-dimensional reconstruction device of the capsule endoscope cavity image proposed in the present application uses relevant technologies in the fields of computer vision and deep learning. It can calculate the camera's movement trajectory in the body without using any sensors, only through the original image (first image) in the tissue cavity (such as the digestive tract) taken by the capsule endoscope camera. At the same time, the calculated movement trajectory and the original image are used to perform three-dimensional reconstruction of the scene of a partial area of ​​the tissue cavity without using any depth information. This solves the problem that the capsule endoscope camera cannot load the corresponding module to capture images with depth information due to limitations such as its own size and battery capacity, and thus cannot be modeled using traditional three-dimensional reconstruction methods.

[0164] In one embodiment, a computer-readable storage medium is provided, on which executable instructions are stored. When the instructions are executed by a processor, the processor executes the steps in the above-mentioned method embodiments.

[0165] In one embodiment, an electronic device is also provided, including one or more processors; a memory, wherein one or more programs are stored in the memory, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the steps in the above-mentioned method embodiments. The electronic device may be the above-mentioned host computer.

[0166] In one embodiment, Figure 7As shown, it shows a schematic diagram of the structure of an electronic device for implementing an embodiment of the present application. The electronic device 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage part 708 to a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 are also stored. The CPU 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0167] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage section 708 as needed.

[0168] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, including a computer-readable medium carrying instructions, in such an embodiment, the instructions can be downloaded and installed from a network through a communication part 709, and / or installed from a removable medium 711. When the instructions are executed by a central processing unit (CPU) 701, the various method steps described in the present application are executed.

[0169] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0170] In addition, those skilled in the art will appreciate that, although some embodiments herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present application and form different embodiments. For example, all of the above embodiments may be used in any combination. The information disclosed in this background technology section is intended only to deepen the understanding of the overall background technology of the present application and should not be regarded as an admission or in any form to imply that the information constitutes prior art already known to those skilled in the art.

Claims

1. A method for three-dimensional reconstruction of capsule endoscope cavity images, characterized in that: The method comprises: Acquire a first image set taken by the capsule endoscope in the cavity; Performing frame rate enhancement on the first image set based on a pre-trained depth perception model to form a second image set, including: identifying fusion feature information of each of the two preceding and succeeding first image frames in the first image set based on the depth perception model, generating an intermediate frame image between each of the two preceding and succeeding first image frames based on the fusion feature information, inserting the intermediate frame image into the first image set to obtain the second image set, wherein the depth perception model includes a convolutional neural network model, a depth estimation network model, and an optical flow estimation network model; Eliminate unqualified images in the second image set based on corner point detection of each second image in the second image set to form a third image set, wherein the number of corner points in the unqualified images is lower than a preset threshold; The three-dimensional scene information of the capsule endoscope is identified based on the third image set, and the cavity is three-dimensionally reconstructed based on the three-dimensional scene information.

2. The three-dimensional reconstruction method according to claim 1, characterized in that: The identifying, based on the depth perception model, fusion feature information of each of the two preceding and succeeding first image frames in the first image set includes: Generate a corresponding feature map from each of the two preceding and following first image frames by using the convolutional neural network model; Generating a corresponding depth map and an optical flow vector from the feature map through the depth estimation network model and the optical flow estimation network model respectively; The fusion feature information of each of the two preceding and succeeding frames of the first image is generated based on the feature map, the depth map and the optical flow vector.

3. The three-dimensional reconstruction method according to claim 2, characterized in that: The generating the fusion feature information of each of the two preceding and succeeding frames of the first image based on the feature map, the depth map and the optical flow vector comprises: Performing spatial transformation on the feature map and the depth map of each of the two preceding and succeeding first frames of the image by using the optical flow vector; The feature map after spatial transformation and the depth map are subjected to feature fusion to generate the fused feature information.

4. The three-dimensional reconstruction method according to claim 1, characterized in that: The depth perception model further includes a generation network model, wherein the generation of the intermediate frame image between each two preceding and succeeding first frames of the image based on the fusion feature information includes: The fused feature information is upsampled and convolved through the generative network model to generate an intermediate frame image with the same size as the first image of each of the two frames before and after. The generative network model is formed by iteratively training the model loss function based on the weighted sum of the structural similarity loss function and the edge loss function, wherein the edge loss function is used to characterize the edge feature loss of the intermediate frame image and the first image of each of the two frames before and after.

5. The three-dimensional reconstruction method according to claim 1, characterized in that: The step of removing unqualified images in the second image set based on the corner point detection of each second image in the second image set to form a third image set includes: Generate a structure tensor containing gradient information in horizontal and vertical directions for the second image in the second image set; Identifying corner points of the second image based on a feature value of each pixel in the structure tensor; The number of corner points in the second image is counted, and the second image with the number of corner points lower than the preset threshold is eliminated to form a third image set.

6. The three-dimensional reconstruction method according to claim 5, characterized in that: The identifying the corner points of the second image based on the feature value of each pixel point in the structure tensor comprises: A response function is calculated based on the characteristic value of each pixel in the structure tensor, and whether the corresponding pixel belongs to a corner point is determined according to the size of each response function in the local area.

7. The three-dimensional reconstruction method according to claim 1, characterized in that: The identifying the three-dimensional scene information of the capsule endoscope based on the third image set includes: extracting feature points in a continuous third image sequence in the third image set by using a first real-time positioning and mapping model; Matching feature points of the third image sequence to form a matching relationship; Identifying pose data of a key image in the third image sequence; Constructing corresponding three-dimensional scene information based on the posture data, feature points and matching relationships; The three-dimensional scene information includes motion trajectory and scene point cloud data, and the three-dimensional reconstruction of the cavity based on the three-dimensional scene information includes: Acquire a correspondence between the three-dimensional scene information and each third image in the third image set; Based on the second real-time positioning and mapping model and the correspondence between the three-dimensional scene information and each third image in the third image set, the third image set is three-dimensionally reconstructed.

8. A three-dimensional reconstruction device for capsule endoscope cavity images, characterized in that: The device comprises: A first image acquisition module, used to acquire a first image set taken by the capsule endoscope in the cavity; a frame enhancement module, configured to enhance the frame rate of the first image set based on a pre-trained depth perception model to form a second image set, identify fusion feature information of each of the two preceding and succeeding first image frames in the first image set based on the depth perception model, generate an intermediate frame image between each of the two preceding and succeeding first image frames based on the fusion feature information, insert the intermediate frame image into the first image set, and obtain the second image set, wherein the depth perception model includes a convolutional neural network model, a depth estimation network model, and an optical flow estimation network model; A corner point processing module, configured to remove unqualified images in the second image set based on corner point detection of each second image in the second image set to form a third image set, wherein the number of corner points in the unqualified images is lower than a preset threshold; A three-dimensional reconstruction module is used to identify the three-dimensional scene information of the capsule endoscope based on the third image set, and to perform three-dimensional reconstruction of the cavity based on the three-dimensional scene information.

9. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs, which, when executed by the one or more processors, causes the one or more processors to perform the method according to any one of claims 1 to 7.

Images