Three-dimensional reconstruction method and device for capsule endoscope image and storage medium

Abstract

The invention discloses a three-dimensional reconstruction method and device for a capsule endoscope image and a storage medium. The three-dimensional reconstruction method comprises the steps that a capsule endoscope collects an image; judging whether the image is clear; if the image meets the definition requirement, the image is preprocessed, image feature points are further extracted, image feature matching is carried out, key frames are further extracted, meanwhile, deep training is carried out on the key frames, then a new view angle is synthesized, and finally a three-dimensional reconstruction image of the intragastric environment is generated through fusion. According to the monocular vision-based three-dimensional reconstruction method for the human gastric environment, external huge magnetic control equipment is not needed to guide the capsule, the medical equipment cost is reduced, the equipment land occupation pressure of a medical institution is reduced, the medical cost of a patient is reduced, a doctor can be assisted to observe the gastric condition of the patient from any angle, the three-dimensional reconstruction effect is good, and the method is suitable for popularization and application. Images are smooth and clear, and doctors can be assisted to make accurate clinical diagnosis.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a method, device and storage medium for three-dimensional reconstruction of capsule endoscope images. Background Art

[0002] During gastrointestinal auxiliary examinations, the images captured by capsule endoscopy may be discontinuous or have a narrow field of view. Professional doctors need to review tens of thousands of images to assist in identifying suspicious lesions. If the captured images themselves contain omissions, the doctor's auxiliary diagnosis will be limited.

[0003] Chinese Patent No. 2018112662754 discloses a method and system for three-dimensional reconstruction of capsule endoscopy images. This solution uses a monocular image sequence to generate a three-dimensional, accurate 3D dense point cloud model to complete the three-dimensional reconstruction. However, the three-dimensional reconstructed images of the cavity generated by this technical solution may have incomplete parts of the cavity tissue, and its reconstruction effect needs to be improved.

[0004] Therefore, it is necessary to improve the existing monocular capsule endoscope image three-dimensional reconstruction technology, introduce a deep training model, and then generate three-dimensional reconstructed images with better accuracy and higher precision. Summary of the Invention

[0005] In a first aspect, the present invention provides a method for three-dimensional reconstruction of capsule endoscopy images, comprising the following steps: S101: The capsule endoscope acquires images. Specifically, the monocular camera of the capsule endoscope captures the gastrointestinal environment in the body and performs distortion correction on the camera before capturing the images. S102: Determine whether the image is clear and remove blurry or smeared images; S103: Image preprocessing, specifically including filtering and enhancing the image; S104: Extract image feature points using an improved ORB algorithm. Scale the image into an image pyramid. The original image is L1, and the subsequent image layers L2 to L4 are 1 / 2, 1 / 4, and 1 / 8 of the original image, respectively. Image feature point extraction is performed on each layer of the image. While extracting feature points, rotation feature extraction is performed on the feature points of each layer of the image. The rotation feature is defined by the centroid and direction of each feature point. S105: Performing image feature matching. Specifically, after the image feature points are extracted, feature matching is performed on the images of the previous and next frames. The similarity between the two features is calculated using the Hamming distance. A fixed threshold T is set. When the Hamming distance between the two frames is less than the threshold T, the two features are considered similar, i.e., the feature matching is successful. S106: Extract key frames, specifically: perform feature matching between the current frame and the key frame. For example, when the system is initialized, the first frame is selected as the key frame by default. Then, each frame thereafter performs feature matching with the nearest key frame. When the feature points of the current frame and the key frame are within a certain range [n1, n2], the current frame is added to the key frame sequence. S107: Deep training key frames, performing deep learning neural network training on the extracted key frames, and inferring images of other perspectives of non-key frames; S108: New perspective synthesis.

[0006] S109: Fusion generates a three-dimensional image of the intragastric environment, specifically, fusing the key frames and the 3D model of the image from the new perspective to generate a 3D model of the entire intragastric environment; Furthermore, the extraction of image feature points specifically includes: S1041: selecting a pixel P in the image, setting the brightness of the pixel P to B p ; S1042: Setting brightness threshold thresh; S1043: Taking pixel P as the center, select 16 pixels on a circle with a radius of 3; S1044: Assume that on the selected circle, there are N consecutive points with brightness greater than B p +thresh or B p -thresh, then the pixel P can be identified as a feature point, where N is a positive integer; S1045: Loop the above steps and perform the same operation on each pixel until all pixels are traversed.

[0007] Furthermore, the image scale scaling L1~L2 layer N=9, and L3~L4 layer N=11.

[0008] Furthermore, the brightness threshold thresh=0.2*B p Furthermore, the centroid selection method is: take the 8-neighborhood of the current feature point as a block Q, and the calculation formula is shown in Formula 1 and Formula 2: , Formula 1; , Formula 2; The direction θ is expressed by Formula 3: , formula three.

[0009] Furthermore, the depth training key frame specifically uses the mapping relationship formula of Formula 4 to define the mapping relationship between the pixel points of the image and the 3D model: , Formula 4; The input is the x, y, z coordinates of each pixel and the viewing directions α and β of the pixel in space, and the output is the RGB color and transparency σ of the pixel.

[0010] Furthermore, the threshold T of the Hamming distance ranges from 40 to 60.

[0011] Furthermore, the feature point matching range is n1=50, n2=80.

[0012] In a second aspect, the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the three-dimensional reconstruction method as described in any one of the first aspects.

[0013] In a third aspect, the present invention provides a three-dimensional reconstruction device, comprising: one or more processors; Memory; and One or more computer programs, wherein the one or more computer programs are stored in the memory and are configured to be executed by the one or more processors, and when the processor executes the computer program, the steps of the three-dimensional reconstruction method as described in any one of the first aspects are implemented.

[0014] The present invention proposes a three-dimensional reconstruction method of the human gastric environment based on monocular vision. It does not require a large external magnetic control device to guide the capsule, which reduces the cost of medical equipment and the equipment space pressure of medical institutions, thereby reducing the medical costs of patients. It can assist doctors in observing the patient's stomach condition from any angle and making accurate clinical diagnoses. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 : Flowchart of the three-dimensional reconstruction method of the present invention.

[0016] Figure 2 : Schematic diagram of image feature point extraction.

[0017] Figure 3 : Schematic diagram of image scaling.

[0018] Figure 4 : Schematic diagram of the three-dimensional reconstruction device. DETAILED DESCRIPTION

[0019] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0020] Please refer to Figure 1Flowchart of the 3D reconstruction method of the present invention, Figure 2 Image feature point extraction diagram and Figure 3 Image zooming diagram, the 3D reconstruction method of the present invention comprises the following steps: S101: The capsule endoscope acquires images. Specifically, the monocular camera of the capsule endoscope captures the gastrointestinal environment in the body and performs distortion correction on the camera before capturing the images. S102: Determine whether the image is clear, specifically including removing blurry and smeared images. If so, proceed to step 103; if the image is not clear, return to the previous step and retake the image; S103: Image preprocessing, specifically including filtering and enhancing the image; S104: Extract image feature points using the improved ORB (Oriented FAST and Rotated BRIEF) algorithm. The image is scaled to form an image pyramid, with the original image being L1 and the subsequent image layers L2 to L4 being 1 / 2, 1 / 4, and 1 / 8 of the original image, respectively. Feature point extraction is performed on each layer of the image. While extracting feature points, rotation features are extracted for the feature points of each layer. The rotation features are defined by the centroid and direction of each feature point. Furthermore, extracting image feature points specifically includes the following steps: S1041: Select pixel P in the image, such as Figure 2 As shown, let the brightness of pixel P be B p ; S1042: Setting the brightness threshold thresh; S1043: Taking pixel P as the center, select 16 pixels on a circle with a radius of 3; S1044: Assume that on the selected circle, there are N consecutive points with brightness greater than B p +thresh or B p -thresh, the pixel P can be identified as a feature point, where N is a positive integer, and the brightness threshold thresh=0.2*B p ; S1045: Loop the above steps and perform the same operation on each pixel until all pixels are traversed.

[0021] Further, the above steps S1041 to S1045 are performed on each layer of image, where N=9 for L1 to L2 layers and N=11 for L3 to L4 layers; Furthermore, the centroid is selected as follows: the 8-neighborhood of the current feature point is taken as a block Q, and the calculation formula is shown in Formula 1 and Formula 2: , Formula 1; , Formula 2; The direction θ of the feature point is expressed as formula 3: , Formula 3; S105: Perform image feature matching. After the image feature points are extracted, feature matching is performed on the images of the previous and next frames. The similarity between the two features is calculated using the Hamming distance. A fixed threshold T is set. When the Hamming distance between the two frames is less than the threshold T, the two features are considered similar, i.e., the feature matching is successful. The threshold T of the Hamming distance ranges from 40 to 60. S106: Extract key frames. To improve processing efficiency, feature matching is not performed on each preceding and following frame. Specifically, the current frame is matched against the key frame. For example, when the system is initialized, the first frame is selected as the key frame by default. Then, each frame thereafter is matched against the nearest key frame. When the feature points of the current frame and the key frame are within a certain range [n1, n2], the current frame is added to the key frame sequence. The feature point matching range is n1=50 and n2=80. If there are too few feature points for feature matching, it means that the two frames are different and tracking is likely to be lost; if there are too many feature points for feature matching, it means that the two frames are too repetitive, which is likely to cause redundant calculations and reduce system performance.

[0022] S107: Deep training key frames, deep learning neural network training is performed on the extracted key frames, and images of non-key frames from other perspectives are inferred. Regarding deep learning neural network training models, there are many mature algorithms, and the present invention is not limited here. Specifically, the mapping relationship formula of Formula 4 is used to define the mapping relationship between the pixels of the image and the 3D model: , Formula 4; The input is the x, y, z coordinates of each pixel and the viewing direction α, β of the pixel in space, and the output is the RGB color and transparency σ of the pixel, which is the representation of the 3D model.

[0023] S108: New perspective synthesis.

[0024] S109: Fusion generates a three-dimensional image of the intragastric environment, specifically: fusing the key frames and the 3D model of the image from the new perspective to generate a 3D model of the entire intragastric environment.

[0025] refer to Figure 4Schematic diagram of a three-dimensional reconstruction device, wherein the three-dimensional reconstruction device 10 further includes one or more memories 20 and one or more processors 30, wherein the one or more computer programs are stored in the memories 20 and are configured to be executed by the one or more processors 30, and the processors 30 implement the steps of the three-dimensional reconstruction method when executing the computer programs.

[0026] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, which may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0027] The present invention's monocular vision-based three-dimensional reconstruction method for capsule endoscope images does not require a large external magnetic control device to guide the capsule, thereby reducing the cost of medical equipment and the equipment footprint of medical institutions, thereby reducing patient medical costs. It can assist doctors in observing the patient's stomach condition from any angle, thereby making accurate clinical diagnoses.

[0028] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for three-dimensional reconstruction of capsule endoscopy images, characterized in that: The following steps are involved: S101: The capsule endoscope acquires images. Specifically, the monocular camera of the capsule endoscope captures the gastrointestinal environment in the body and performs distortion correction on the camera before capturing the images. S102: Determine whether the image is clear and remove blurry or smeared images; S103: Image preprocessing, specifically including filtering and enhancing the image; S104: Extracting image feature points: Using an improved ORB algorithm to extract image feature points, the image is scaled into an image pyramid. The original image is L1, and the subsequent image layers L2 to L4 are 1 / 2, 1 / 4, and 1 / 8 of the original image, respectively. Image feature point extraction is performed on each layer of the image. While extracting the feature points, rotation features are extracted for the feature points of each layer of the image. The rotation features are defined by the centroid and direction of each feature point. S105: Perform image feature matching. After the image feature points are extracted, feature matching is performed on the images of the previous and next frames. The similarity between the two features is calculated using the Hamming distance. A fixed threshold T is set. When the Hamming distance between the two frames is less than the threshold T, the two features are considered similar, i.e., the feature matching is successful. S106: Extract key frames, specifically: perform feature matching between the current frame and the key frame. For example, when the system is initialized, the first frame is selected as the key frame by default. Then, each frame thereafter performs feature matching with the nearest key frame. When the feature points of the current frame and the key frame are within a certain range [n1, n2], the current frame is added to the key frame sequence. S107: Deep training key frames, performing deep learning neural network training on the extracted key frames, and inferring images of other perspectives of non-key frames; S108: New perspective synthesis; S109: Fusion generates a three-dimensional image of the intragastric environment, specifically, fuses the key frames and the 3D model of the image from the new perspective to generate a 3D model of the entire intragastric environment.

2. The method for three-dimensional reconstruction of capsule endoscopy images according to claim 1, wherein: The extracting of image feature points specifically includes: S1041: Select pixel P in the image and set the brightness of pixel P to B p ; S1042: Setting brightness threshold thresh; S1043: Taking pixel P as the center, select 16 pixels on a circle with a radius of 3; S1044: Assume that on the selected circle, there are N consecutive points with brightness greater than B p +thresh or B p -thresh, then the pixel P can be identified as a feature point, where N is a positive integer; S1045: Loop the above steps and perform the same operation on each pixel until all pixels are traversed.

3. The method for 3D reconstruction of capsule endoscopy images according to claim 2, wherein: The image scale scaling has L1~L2 layers N=9 and L3~L4 layers N=11.

4. The method for 3D reconstruction of capsule endoscopy images according to claim 2, wherein: The brightness threshold thresh=0.2*B p .

5. The method for 3D reconstruction of capsule endoscopy images according to claim 1, wherein: The method for selecting the centroid is: take the 8-neighborhood of the current feature point as a block Q, and the calculation formula is shown in Formula 1 and Formula 2: , Formula 1; , Formula 2; The direction θ is expressed by Formula 3: , formula three.

6. The method for 3D reconstruction of capsule endoscopy images according to claim 1, wherein: The depth training key frame specifically uses the mapping relationship formula of Formula 4 to define the mapping relationship between the pixel points of the image and the 3D model: , Formula 4; The input is the x, y, z coordinates of each pixel and the viewing directions α and β of the pixel in space, and the output is the RGB color and transparency σ of the pixel.

7. The method for 3D reconstruction of capsule endoscopy images according to claim 1, wherein: The threshold T of the Hamming distance ranges from 40 to 60.

8. The method for 3D reconstruction of capsule endoscopy images according to claim 1, wherein: The feature point matching range is n1=50, n2=80.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method for three-dimensional reconstruction of capsule endoscopy images according to any one of claims 1 to 8 are implemented.

10. A three-dimensional reconstruction device comprising: one or more processors; Memory; as well as One or more computer programs, wherein the one or more computer programs are stored in the memory and are configured to be executed by the one or more processors, characterized in that when the processor executes the computer program, the steps of the three-dimensional reconstruction method of capsule endoscopy images according to any one of claims 1 to 8 are implemented.

Images