A three-dimensional reconstruction method for the stomach working environment of a magnetorheological medical capsule robot

Through the three-dimensional reconstruction method of the stomach working environment of magnetorheological medical capsule robot, the image acquisition, processing and three-dimensional model reconstruction system is used to solve the problem of image acquisition and three-dimensional reconstruction of capsule robots in the stomach environment, and achieve high-precision stomach Three-dimensional model reconstruction supports medical staff's diagnosis.

CN114299250BActive Publication Date: 2024-12-06CHINA UNIV OF MINING & TECH
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202111629456.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-12-28
Publication Date
2024-12-06
Estimated Expiration
2041-12-28

AI Technical Summary

Technical Problem

The existing three-dimensional reconstruction technology is difficult to effectively obtain clear images and determine the location of the disease in the stomach working environment of medical capsule robots. Moreover, due to the light weight and small size of the capsule robot, it cannot meet the three-dimensional reconstruction requirements of the stomach environment.

Method used

The three-dimensional reconstruction method of the stomach working environment of magnetorheological medical capsule robots is adopted, including image acquisition system, image processing system and three-dimensional model reconstruction system. Images are collected through the capsule robot vision module and wireless transmission module, and distortion correction, bilateral filtering and characteristics are performed. Point detection, feature points are extracted using SIFT algorithm, combined with FLANN and Deepflow algorithms for matching, and three-dimensional model reconstruction is carried out through phase correlation algorithm and beam adjustment method.

Benefits of technology

It realizes the acquisition of stomach images in medical capsule robots and reconstructs a high-precision three-dimensional model, helping medical staff better examine and diagnose stomach diseases, and provides a visualized and high surface accuracy three-dimensional model of the stomach.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114299250B_ABST
    Figure CN114299250B_ABST
Patent Text Reader

Abstract

The present invention discloses a three-dimensional reconstruction method of the stomach working environment of a magnetorheological medical capsule robot, comprising a stomach image acquisition system, an image processing system and a three-dimensional model reconstruction system; wherein, the stomach image acquisition system works, the medical capsule robot acquires images and transmits them to the upper computer software, then enters the image processing system, and finally enters the three-dimensional model reconstruction system. The present invention uses the multi-view images acquired by the medical capsule robot, improves the SIFT algorithm to extract feature descriptors, then performs feature matching, and uses the motion recovery structure technology to establish a sparse point cloud model, then uses the PMVS algorithm to calculate the depth map of the image, performs dense reconstruction, and then performs mesh optimization and texture mapping to reconstruct the three-dimensional model of the stomach, so as to obtain a stomach image with a wider field of view, clearer and more comprehensive.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to a folding mechanism, in particular to a three-dimensional reconstruction method for the stomach working environment of a magnetorheological medical capsule robot, and belongs to the technical field of mechanization in new energy photovoltaic power generation. Background Art

[0002] Due to changes in social environment and people's lifestyles, the gastrointestinal tract of most people is in a sub-healthy state. According to the World Health Organization's latest "World Cancer Report 2020", China ranks first in the world in the number of new cancer cases, and the incidence of gastrointestinal malignancies (colorectal cancer and gastric cancer) ranks among the top five. Therefore, it is particularly important to carry out extensive early clinical examinations of the gastrointestinal tract, which is the key to avoiding the development of gastric cancer and other gastrointestinal diseases.

[0003] According to current research, there are few methods for reconstructing the three-dimensional environment of capsule robot visual images. The existing three-dimensional reconstruction technologies are mainly divided into reconstruction technologies based on active vision, mainly including structured light, TOF (Time of Flight), lidar, etc., and reconstruction technologies based on passive vision, mainly including monocular or multi-camera cameras.

[0004] However, due to the narrow working environment of the stomach of the medical capsule robot, and the narrow field of view of the pictures taken during the movement, the complex changes in the inclination and rotation angles, and the unknown overlapping areas, it is not easy to determine the specific location of the disease. In addition, other 3D reconstruction methods do not meet the requirements of 3D reconstruction of the stomach environment due to the characteristics of the capsule robot itself, such as light weight and small size. Summary of the invention

[0005] The purpose of the present invention is to provide a three-dimensional reconstruction method for the stomach working environment of a magnetorheological medical capsule robot in order to solve the problem, so as to obtain stomach images and reconstruct a three-dimensional stomach model during the working process of the medical capsule robot, thereby helping medical staff to better examine and diagnose stomach diseases.

[0006] The present invention achieves the above-mentioned purpose through the following technical solutions: A three-dimensional reconstruction method of the stomach working environment of a magnetorheological medical capsule robot, comprising:

[0007] The image acquisition system is used to acquire stomach images and is composed of a capsule robot vision module and a capsule robot wireless transmission module. Specifically, the system includes the following steps:

[0008] ① Turn on the capsule robot vision module of the capsule robot to check whether the stomach image taken by the capsule robot meets the clarity condition. If so, collect stomach images Img1, Img2, Img3, ..., ImgN. Otherwise, adjust the focal length parameters of the capsule robot micro-camera to re-collect the image;

[0009] ② The wireless transmission module of the capsule robot transmits the collected images to the PC. The PC numbers and classifies the images and determines whether the images are continuous and whether two consecutive images overlap. If they meet the requirements, the next step is carried out. Otherwise, the reason is that the capsule robot moves too fast or the motion trajectory is different. By adjusting the capsule robot's motion rate and motion trajectory, the capsule robot parameters are adjusted to meet the judgment conditions. Finally, the image set is output to the stomach image processing system.

[0010] The image processing system is used to process the collected images, and is composed of an image smoothing processing module and an image feature point detection and matching module, and specifically includes the following steps:

[0011] ① Input the stomach images collected by the capsule robot, correct the distorted images in the collected image set, perform distortion correction, and then perform bilateral filtering on the image set to remove salt and pepper noise and retain the edge information of the image;

[0012] ② Improve the SIFT operator to extract image feature points, generate feature point descriptors, and then calculate the spatial transformation matrix H between two images;

[0013] ③ Use the neighbor ratio algorithm to eliminate mismatched points and calculate the nearest neighbor distance D(a,b*) and the next nearest neighbor distance D(a,b**) of feature point a. Calculate the threshold δ. If δ<5, it is an exact match point, which is retained and stored in the exact match point list. Otherwise, if 5≤δ<8 is satisfied, it is a rough match point, which is retained and stored in the rough match point list. If δ≥8, it is considered as an incorrect match point and deleted.

[0014] ④Finally, the phase correlation algorithm is used to fuse the precise matching points and coarse matching points of each pair of images, and the matching points are stored in descending order according to the number of matching points of each pair of images, and then the fused matching points are output to the 3D model reconstruction system.

[0015] The 3D model reconstruction system calculates the image depth map through PMVS, then fuses the reconstructed sparse point cloud based on the image depth map, and finally performs registration to obtain a 3D densely reconstructed model. The system is composed of a sparse point cloud reconstruction module, a dense reconstruction module, a 3D model mesh optimization module, and a texture mapping module, and specifically includes the following steps:

[0016] ① The image feature point detection and matching module inputs the matching points of the two images, finds the best basic perspective for 3D reconstruction, and then reconstructs based on the best dual perspective. Enters the outer loop judgment box to determine whether there are any unadded perspectives. If there are any, add them to the 3D reconstruction system. Enters the inner loop judgment box to determine whether the Mesh matching points are not empty. If not, execute the next step.

[0017] ② The external parameters of the camera, namely the rotation matrix R and the translation matrix T, are calculated by the principle of epipolar geometry, and the coordinates of the three-dimensional points in the camera coordinate system are obtained by the triangulation method. However, due to the continuous iteration of adding new perspectives and matching points, the iteratively calculated three-dimensional point cloud coordinates have cumulative errors. Therefore, the bundle adjustment method (BA optimization) is used to optimize and adjust them, that is, to reconstruct a new three-dimensional sparse point cloud model, and then return to the loop body, iteratively calculate until the Mesh matching point is empty, then immediately reconstruct a new three-dimensional point cloud model, perform global BA optimization, and input the dense reconstruction module;

[0018] ③ After all perspectives are added, perform global BA optimization directly to obtain a sparse point cloud model and input it into the dense reconstruction module. At this point, you can view the densely reconstructed model. First, optimize the triangular mesh composed of pixels to obtain a pixel mesh model with more obvious image features. Then, perform texture mapping and color calculation on the three-dimensional model. Finally, obtain a complete, visual, and high-surface-precision three-dimensional model and output it.

[0019] As a further solution of the present invention: in step ① of the image acquisition system, the magnetorheological capsule robot vision module acquires stomach images Img1, Img2, Img3, ..., ImgN, transmits them to the PC through the micro-electromechanical module, and uses the image frame difference method to determine whether the acquired images are continuous, forming a judgment loop until the acquired images can meet the requirements.

[0020] As a further solution of the present invention: in step ① of the image processing system, the image smoothing processing module uses a bilateral filtering algorithm to smooth the collected stomach image, which can ensure the edge information of the image, eliminate image noise, and facilitate the generation of image edge key point descriptors. The calculation formula is as follows:

[0021]

[0022]

[0023] Among them, ||pq|| is the Euclidean distance between points p and q, |I p -I q | is the difference between the pixel and its neighboring pixels, G s is the spatial distance weight value, G r is the pixel weight value, W p bf is the weight value after fusion.

[0024] As a further solution of the present invention: in step ② of the image processing system, each feature point of the image extracted by the image feature point detection and matching module generates a 128-dimensional feature vector, which can accurately perform image registration and reduce the time of image registration.

[0025] As a further solution of the present invention: in step ② of the image processing system, the spatial transformation matrix H is calculated. If the value of the H matrix is ​​large, the FLANN algorithm is executed to match the feature points between the two images and save the Euclidean distance values ​​of the matching points; if the value of the H matrix is ​​small, the Deepflow algorithm is executed to match the feature points and save the Euclidean distance values ​​of the matching points.

[0026] As a further solution of the present invention: in step ③ of the image processing system, the neighbor ratio algorithm removes the mismatched points, and the threshold calculation formula is as follows:

[0027]

[0028]

[0029] In the formula, D(a,b * ) is the matching point a and the nearest point b * The Euclidean distance, D(a,b ** ) is the matching point a and the next adjacent point b ** The Euclidean distance of , δ is its ratio, P(a,b) is the value of the judgment box, and the purpose is to execute the steps. If P(a,b)=0, it is regarded as an accurate matching point and stored; if P(a,b)=1, it is regarded as a rough matching point and stored; if P(a,b)=2, it is regarded as an incorrect matching point and the matching point is removed.

[0030] As a further solution of the present invention: in step ① of the three-dimensional model reconstruction system, a double loop structure is used to add image perspective and image matching points, and finally a global BA optimization is performed, which specifically includes: firstly traversing the image without added perspective and adding it to the reconstruction system; secondly, adding the pairwise image matching points in the Mesh to the reconstruction system, calculating the rotation matrix R and translation matrix T of the camera extrinsic parameter camera, and obtaining the three-dimensional coordinates of the matching points in the camera coordinate system by triangulation.

[0031] As a further solution of the present invention: in step ② of the three-dimensional model reconstruction system, the bundle adjustment method bundles and adjusts the accumulated pairwise images, takes the camera posture and the three-dimensional coordinates of the measurement points as unknown parameters, and takes the coordinates of the feature points detected on the image for forward intersection as observation data to perform adjustment to obtain the optimal camera parameters and world point coordinates, and finally reconstructs a new three-dimensional sparse point cloud.

[0032] As a further solution of the present invention: in step ③ of the three-dimensional model reconstruction system, PMVS is used to calculate the depth map of the image, and dense reconstruction based on depth fusion is obtained to obtain a complete dense point cloud three-dimensional model, and the triangular mesh composed of pixel points is optimized to make the surface features of the model smoother and more prominent.

[0033] The beneficial effects of the present invention are:

[0034] 1. This 3D reconstruction method uses stomach images collected by a medical capsule robot as input and outputs a stomach reconstruction model, which can be zoomed in, rotated, transformed, and visualized;

[0035] 2. The 3D reconstruction method includes three aspects: gastric image acquisition, image processing, and 3D model reconstruction;

[0036] 3. The 3D reconstruction method uses the SIFT algorithm to extract image feature points, uses the FLANN algorithm and the Deepflow algorithm to match feature points, and the neighbor ratio algorithm removes mismatched points by setting a threshold;

[0037] 4. The 3D reconstruction method sets up a double loop nested structure, where the first loop adds pictures from different perspectives, and the second loop adds matching points stored in the Mesh, adding all pictures and matching points to the reconstruction system;

[0038] 5. The 3D reconstruction method calculates the coordinates of the 3D points, obtains the 3D sparse point cloud model by bundle adjustment (BA optimization), and adds the above process into a loop nested structure for multiple iterations;

[0039] 6. This 3D reconstruction method uses the PMVS algorithm to calculate the depth map of the image and obtains dense reconstruction based on depth fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a schematic diagram of the overall framework structure of the present invention;

[0041] Figure 2 This is a working flow chart of the stomach image acquisition system of the present invention;

[0042] Figure 3 This is a workflow diagram of the image processing system of the present invention;

[0043] Figure 4 This is a workflow diagram of the three-dimensional model reconstruction system of the present invention. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] Embodiment 1

[0046] like Figures 1 to 4 As shown: A three-dimensional reconstruction method for the stomach working environment of a magnetorheological medical capsule robot, including a stomach image acquisition system, an image processing system, and a three-dimensional model reconstruction system. When this system is turned on, the stomach image acquisition system works, and the medical capsule robot collects images and transmits them to the upper computer software, then enters the image processing system, and finally enters the three-dimensional model reconstruction system.

[0047] Turn on the micro-camera unit of the medical capsule robot to check whether the image is clear. If it is not clear, adjust the parameters of the medical capsule robot's visual module. Then put the medical capsule robot into the stomach working environment, start collecting stomach images and transmit the images to the PC, number the collected stomach images on the PC, and normalize the image size. Enter to check whether the image is continuous. If the images do not overlap, it will affect the accuracy of its three-dimensional reconstruction. Therefore, it is necessary to judge whether the image is continuous. If it is continuous, it will be output to the image processing system. If it is not continuous, adjust the capsule robot's movement rate and trajectory, indirectly adjust the capsule robot parameters, and re-collect the image.

[0048] Enter the image processing system, as shown in the attached Figure 3 As shown, firstly, the stomach images Img1, Img2, Img3, ..., ImgN collected by the magnetorheological capsule robot are input, and the distorted images in the collected image set are corrected. Then, the image set is processed by bilateral filtering to remove salt and pepper noise and retain the edge information of the image. The calculation formula is as follows:

[0049]

[0050]

[0051] Among them, ||pq|| is the Euclidean distance between points p and q, |I p -I q | is the difference between the pixel and its neighboring pixels, G s is the spatial distance weight value, G r is the pixel weight value, W p bf is the weight value after fusion.

[0052] Secondly, the improved SIFT operator is used to extract image feature points and generate feature point descriptors. Taking the image Img n as an example, the spatial transformation matrix H of Img n+1 is calculated. If the value of the determinant H is large, the FLANN algorithm is used for matching; if the value of the determinant H is small, the Deepflow algorithm is used for matching. The FLANN algorithm has good robustness for images with large perspective changes, while the Deepflow algorithm has high matching accuracy and matching rate for images with strong continuity and small perspective changes. The neighbor ratio algorithm is used to eliminate mismatched points, and the threshold calculation formula is as follows:

[0053]

[0054]

[0055] In the formula, D(a,b * ) is the matching point a and the nearest point b * The Euclidean distance, D(a,b ** ) is the matching point a and the next adjacent point b ** The Euclidean distance of , δ is its ratio, P(a,b) is the value of the judgment box, and the purpose is to execute the steps. If P(a,b)=0, it is regarded as an accurate matching point and stored; if P(a,b)=1, it is regarded as a rough matching point and stored; if P(a,b)=2, it is regarded as an incorrect matching point and the matching point is removed.

[0056] Finally, the phase correlation algorithm is used to fuse the precise matching points and coarse matching points of each pair of images, and the matching points are stored in descending order according to the number of matching points in each pair of images, and then the fused matching points are output to the 3D model reconstruction system.

[0057] Enter the 3D model reconstruction system, as shown in the attached Figure 4 As shown, first, the image feature point detection and matching module inputs the matching points of the two images to find the best basic perspective for 3D reconstruction, and then reconstructs based on the best dual perspective. Set a double loop nested structure, enter the outer loop judgment box, and determine whether there are any unadded perspectives? If there are still perspectives, add them to the 3D reconstruction system, enter the inner loop judgment box, and determine whether the Mesh matching points are not empty? If not, execute the next step, calculate the external parameters of the magnetorheological capsule robot micro camera, namely the rotation matrix R and the translation matrix T, through the principle of epipolar geometry, and obtain the coordinates of the 3D points in the camera coordinate system by the triangulation method. The bundle adjustment method (BA optimization) is used to optimize and adjust them, that is, to reconstruct a new 3D sparse point cloud model, and then return to the loop body, execute the loop structure iterative calculation until the Mesh matching points are empty, then immediately reconstruct a new 3D point cloud model, perform global BA optimization again, and input the value dense reconstruction module.

[0058] After all the perspectives are added, global BA optimization is directly performed to obtain a sparse point cloud model which is input into the dense reconstruction module. In the dense reconstruction process of the 3D model, PMVS first calculates the image depth map, and then reconstructs the dense model based on the image depth map fusion.

[0059] At this point, you can view the densely reconstructed model. If you want to get a three-dimensional model with high surface accuracy, you need to optimize the triangular mesh composed of pixels to get a pixel mesh model with more obvious image features. Then, perform texture mapping and color calculation on the three-dimensional model. Finally, you can get a complete, visual, and high-surface-accuracy three-dimensional model and output it.

[0060] Embodiment 2

[0061] like Figures 2 to 4 As shown, a three-dimensional reconstruction method for the stomach working environment of a magnetorheological medical capsule robot is

[0062] The image acquisition system is used to acquire stomach images and is composed of a capsule robot vision module and a capsule robot wireless transmission module. Specifically, the system includes the following steps:

[0063] ① Turn on the capsule robot vision module of the capsule robot to check whether the stomach image taken by the capsule robot meets the clarity condition. If so, collect stomach images Img1, Img2, Img3, ..., ImgN. Otherwise, adjust the focal length parameters of the capsule robot micro-camera to re-collect the image;

[0064] ② The wireless transmission module of the capsule robot transmits the collected images to the PC. The PC numbers and classifies the images and determines whether the images are continuous and whether two consecutive images overlap. If they meet the requirements, the next step is carried out. Otherwise, the reason is that the capsule robot moves too fast or the motion trajectory is different. By adjusting the capsule robot's motion rate and motion trajectory, the capsule robot parameters are adjusted to meet the judgment conditions. Finally, the image set is output to the stomach image processing system.

[0065] The image processing system is used to process the collected images, and is composed of an image smoothing processing module and an image feature point detection and matching module, and specifically includes the following steps:

[0066] ① Input the stomach images collected by the capsule robot, correct the distorted images in the collected image set, perform distortion correction, and then perform bilateral filtering on the image set to remove salt and pepper noise and retain the edge information of the image;

[0067] ② Improve the SIFT operator to extract image feature points, generate feature point descriptors, and then calculate the spatial transformation matrix H between two images;

[0068] ③ Use the neighbor ratio algorithm to eliminate mismatched points and calculate the nearest neighbor distance D(a,b*) and the next nearest neighbor distance D(a,b**) of feature point a. Calculate the threshold δ. If δ<5, it is an exact match point, which is retained and stored in the exact match point list. Otherwise, if 5≤δ<8 is satisfied, it is a rough match point, which is retained and stored in the rough match point list. If δ≥8, it is considered as an incorrect match point and deleted.

[0069] ④Finally, the phase correlation algorithm is used to fuse the precise matching points and the rough matching points of each pair of images, and the matching points of each pair of images are stored in descending order, and then the fused matching points are output to the 3D model reconstruction system;

[0070] The 3D model reconstruction system calculates the image depth map through PMVS, then fuses the reconstructed sparse point cloud based on the image depth map, and finally performs registration to obtain a 3D densely reconstructed model. The system is composed of a sparse point cloud reconstruction module, a dense reconstruction module, a 3D model mesh optimization module, and a texture mapping module, and specifically includes the following steps:

[0071] ① The image feature point detection and matching module inputs the matching points of the two images, finds the best basic perspective for 3D reconstruction, and then reconstructs based on the best dual perspective. Enters the outer loop judgment box to determine whether there are any unadded perspectives. If there are any, add them to the 3D reconstruction system. Enters the inner loop judgment box to determine whether the Mesh matching points are not empty. If not, execute the next step.

[0072] ② The external parameters of the camera, namely the rotation matrix R and the translation matrix T, are calculated by the principle of epipolar geometry, and the coordinates of the three-dimensional points in the camera coordinate system are obtained by the triangulation method. However, due to the continuous iteration of adding new perspectives and matching points, the iteratively calculated three-dimensional point cloud coordinates have cumulative errors. Therefore, the bundle adjustment method (BA optimization) is used to optimize and adjust them, that is, to reconstruct a new three-dimensional sparse point cloud model, and then return to the loop body, iteratively calculate until the Mesh matching point is empty, then immediately reconstruct a new three-dimensional point cloud model, perform global BA optimization, and input the dense reconstruction module;

[0073] ③ After all perspectives are added, perform global BA optimization directly to obtain a sparse point cloud model and input it into the dense reconstruction module. At this point, you can view the densely reconstructed model. First, optimize the triangular mesh composed of pixels to obtain a pixel mesh model with more obvious image features. Then, perform texture mapping and color calculation on the three-dimensional model. Finally, obtain a complete, visual, and high-surface-precision three-dimensional model and output it.

[0074] In an embodiment of the present invention, in step ① of the image acquisition system, the magnetorheological capsule robot vision module acquires stomach images Img1, Img2, Img3, ..., ImgN, transmits them to the PC through the micro-electromechanical module, and uses the image frame difference method to determine whether the acquired images are continuous, forming a judgment loop until the acquired images can meet the requirements.

[0075] In the embodiment of the present invention, in step ① of the image processing system, the image smoothing processing module uses a bilateral filtering algorithm to smooth the collected stomach image, which can ensure the edge information of the image, remove the image noise, and facilitate the generation of the image edge key point descriptor. The calculation formula is as follows:

[0076]

[0077]

[0078] Among them, ||pq|| is the Euclidean distance between points p and q, |I p -I q | is the difference between the pixel and its neighboring pixels, G s is the spatial distance weight value, G r is the pixel weight value, W p bf is the weight value after fusion.

[0079] In the embodiment of the present invention, in step ② of the image processing system, each feature point of the image extracted by the image feature point detection and matching module generates a 128-dimensional feature vector, which can accurately perform image registration and reduce the time of image registration.

[0080] In the embodiment of the present invention, in step ② of the image processing system, the spatial transformation matrix H is calculated. If the value of the H matrix is ​​large, the FLANN algorithm is executed to match the feature points between the two images and save the Euclidean distance values ​​of the matching points. If the value of the H matrix is ​​small, the Deepflow algorithm is executed to match the feature points and save the Euclidean distance values ​​of the matching points.

[0081] In the embodiment of the present invention, in step ③ of the image processing system, the neighbor ratio algorithm removes the mismatched points, and the threshold calculation formula is as follows:

[0082]

[0083]

[0084] In the formula, D(a,b * ) is the matching point a and the nearest point b * The Euclidean distance, D(a,b **) is the matching point a and the next adjacent point b ** The Euclidean distance of , δ is its ratio, P(a,b) is the value of the judgment box, and the purpose is to execute the steps. If P(a,b)=0, it is regarded as an accurate matching point and stored; if P(a,b)=1, it is regarded as a rough matching point and stored; if P(a,b)=2, it is regarded as an incorrect matching point and the matching point is removed.

[0085] In the embodiment of the present invention, in step ① of the three-dimensional model reconstruction system, a double loop structure is used to add image perspective and image matching points, and finally a global BA optimization is performed, which specifically includes: firstly traversing the image without added perspective and adding it to the reconstruction system; secondly, adding the pairwise image matching points in the Mesh to the reconstruction system, calculating the rotation matrix R and translation matrix T of the camera extrinsic camera, and obtaining the three-dimensional coordinates of the matching points in the camera coordinate system by triangulation.

[0086] In an embodiment of the present invention, in step ② of the three-dimensional model reconstruction system, the bundle adjustment method (BA optimization) bundles and adjusts the accumulated pairwise images, takes the camera posture and the three-dimensional coordinates of the measurement points as unknown parameters, and takes the coordinates of the feature points detected on the image for forward intersection as observation data to perform adjustment to obtain the optimal camera parameters and world point coordinates, and finally reconstructs a new three-dimensional sparse point cloud.

[0087] In an embodiment of the present invention, in step ③ of the three-dimensional model reconstruction system, PMVS is used to calculate the depth map of the image, and dense reconstruction based on depth fusion is performed to obtain a complete dense point cloud three-dimensional model, and the triangular mesh composed of pixel points is optimized to make the surface features of the model smoother and more prominent.

[0088] Working principle: The multi-view images collected by the medical capsule robot are used to improve the SIFT algorithm to extract feature descriptors, followed by feature matching. The motion recovery structure technology is used to establish a sparse point cloud model. The PMVS algorithm is then used to calculate the depth map of the image and perform dense reconstruction. This is followed by mesh optimization and texture mapping to reconstruct the three-dimensional model of the stomach in order to obtain a wider, clearer, and more comprehensive stomach image.

[0089] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and range of equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

[0090] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A three-dimensional reconstruction method for the stomach working environment of a magnetorheological medical capsule robot, characterized in that: include The image acquisition system is used to acquire stomach images and is composed of a capsule robot vision module and a capsule robot wireless transmission module. Specifically, the following steps are included: ① Turn on the capsule robot vision module of the capsule robot to check whether the stomach image taken by the capsule robot meets the clarity condition. If so, collect stomach images Img1, Img2, Img3, ..., ImgN. Otherwise, adjust the focal length parameters of the capsule robot micro-camera to re-collect the image; ② The wireless transmission module of the capsule robot transmits the collected images to the PC. The PC numbers and classifies the images and determines whether the images are continuous and whether two consecutive images overlap. If they meet the requirements, the next step is carried out. Otherwise, the reason is that the capsule robot moves too fast and the motion trajectory is different. By adjusting the capsule robot's motion rate and motion trajectory, the capsule robot parameters are adjusted to meet the judgment conditions. Finally, the image set is output to the stomach image processing system. The image processing system is used to process the collected images, and is composed of an image smoothing processing module and an image feature point detection and matching module, and specifically includes the following steps: ① Input the stomach images collected by the capsule robot, correct the distorted images in the collected image set, perform distortion correction, and then perform bilateral filtering on the image set to remove salt and pepper noise and retain the edge information of the image; ② Improve the SIFT operator to extract image feature points, generate feature point descriptors, and then calculate the spatial transformation matrix H between two images; ③ Use the neighbor ratio algorithm to eliminate mismatched points and calculate the nearest neighbor distance D(a,b*) and the next nearest neighbor distance D(a,b**) of feature point a. Calculate the threshold δ. If δ<5, it is an exact match point, which is retained and stored in the exact match point list. Otherwise, if 5≤δ<8 is satisfied, it is a rough match point, which is retained and stored in the rough match point list. If δ≥8, it is considered as an incorrect match point and deleted. ④ Use the phase correlation algorithm to fuse the precise matching points and coarse matching points of each pair of images, and store them in descending order according to the number of matching points of each pair of images, and then output the fused matching points to the 3D model reconstruction system; The 3D model reconstruction system calculates the image depth map through PMVS, then fuses the reconstructed sparse point cloud based on the image depth map, and finally performs registration to obtain a 3D densely reconstructed model. The system is composed of a sparse point cloud reconstruction module, a dense reconstruction module, a 3D model mesh optimization module, and a texture mapping module, and specifically includes the following steps: ① The image feature point detection and matching module inputs the matching points of the two images, finds the best basic perspective for 3D reconstruction, and then reconstructs based on the best dual perspective. Enters the outer loop judgment box to determine whether there are any unadded perspectives. If there are any, add them to the 3D reconstruction system. Enters the inner loop judgment box to determine whether the Mesh matching points are not empty. If not, execute the next step. ② The external parameters of the camera, namely the rotation matrix R and the translation matrix T, are calculated through the principle of epipolar geometry, and the coordinates of the three-dimensional points in the camera coordinate system are obtained by the triangulation method. However, due to the continuous iteration of adding new perspectives and matching points, the iteratively calculated three-dimensional point cloud coordinates have cumulative errors. Therefore, the bundling adjustment method is used to optimize and adjust them, that is, to reconstruct a new three-dimensional sparse point cloud model, and then return to the loop body, iteratively calculate until the Mesh matching point is empty, then immediately reconstruct a new three-dimensional point cloud model, perform global BA optimization, and input the dense reconstruction module; ③ After all perspectives are added, perform global BA optimization directly to obtain a sparse point cloud model and input it into the dense reconstruction module. At this point, you can view the densely reconstructed model. First, optimize the triangular mesh composed of pixels to obtain a pixel mesh model with more obvious image features. Then, perform texture mapping and color calculation on the three-dimensional model. Finally, obtain a complete, visual, and high-surface-precision three-dimensional model and output it.

2. A three-dimensional reconstruction method of the stomach working environment of a magnetorheological medical capsule robot according to claim 1, characterized in that: In step ① of the image acquisition system, the magnetorheological capsule robot vision module acquires stomach images Img1, Img2, Img3, ..., ImgN, transmits them to the PC through the micro-electromechanical module, and uses the image frame difference method to determine whether the acquired images are continuous, forming a judgment loop until the acquired images meet the requirements.

3. A three-dimensional reconstruction method for the stomach working environment of a magnetorheological medical capsule robot according to claim 1 or 2, characterized in that: In step ① of the image processing system, the image smoothing processing module uses a bilateral filtering algorithm to smooth the collected stomach image, and the calculation formula is as follows: Among them, ||pq|| is the Euclidean distance between points p and q, |I p -I q | is the difference between the pixel and its neighboring pixels, G s is the spatial distance weight value, G r is the pixel weight value, W p bf is the weight value after fusion.

4. The method for three-dimensional reconstruction of the stomach working environment of a magnetorheological medical capsule robot according to claim 1, characterized in that: In step ② of the image processing system, each feature point of the image extracted by the image feature point detection and matching module generates a 128-dimensional feature vector.

5. The method for three-dimensional reconstruction of the stomach working environment of a magnetorheological medical capsule robot according to claim 1, characterized in that: In step ② of the image processing system, the spatial transformation matrix H is calculated. If the value of the H matrix is ​​large, the FLANN algorithm is executed to match the feature points between the two images and save the Euclidean distance values ​​of the matching points. If the value of the H matrix is ​​small, the Deepflow algorithm is executed to match the feature points and save the Euclidean distance values ​​of the matching points.

6. The method for three-dimensional reconstruction of the stomach working environment of a magnetorheological medical capsule robot according to claim 1, characterized in that: In step ③ of the image processing system, the neighbor ratio algorithm removes the mismatched points, and the threshold calculation formula is as follows: In the formula, D(a,b * ) is the matching point a and the nearest point b * The Euclidean distance, D(a,b ** ) is the matching point a and the next adjacent point b ** The Euclidean distance of , δ is its ratio, P(a,b) is the value of the judgment box, and the purpose is to execute the steps. If P(a,b)=0, it is regarded as an accurate matching point and stored; if P(a,b)=1, it is regarded as a rough matching point and stored; if P(a,b)=2, it is regarded as an incorrect matching point and the matching point is removed.

7. The method for three-dimensional reconstruction of the stomach working environment of a magnetorheological medical capsule robot according to claim 1, characterized in that: In step ① of the 3D model reconstruction system, a double loop structure is used to add image perspective and image matching points, and finally a global BA optimization is performed, which specifically includes: 1) Traverse the images without added viewpoints and add them into the reconstruction system; 2) Add the pairwise image matching points in the Mesh to the reconstruction system, calculate the rotation matrix R and translation matrix T of the camera extrinsic camera, and obtain the three-dimensional coordinates of the matching points in the camera coordinate system through triangulation.

8. The method for three-dimensional reconstruction of the stomach working environment of a magnetorheological medical capsule robot according to claim 1, characterized in that: In step ② of the three-dimensional model reconstruction system, the bundle adjustment method bundles and adjusts the accumulated pairwise images, takes the camera posture and the three-dimensional coordinates of the measurement points as unknown parameters, and takes the coordinates of the feature points detected on the image for forward intersection as observation data to perform adjustment to obtain the optimal camera parameters and world point coordinates, and finally reconstructs a new three-dimensional sparse point cloud.

9. The method for three-dimensional reconstruction of the stomach working environment of a magnetorheological medical capsule robot according to claim 1, characterized in that: In step ③ of the three-dimensional model reconstruction system, PMVS is used to calculate the depth map of the image, and dense reconstruction based on depth fusion is performed to obtain a complete dense point cloud three-dimensional model, and the triangular mesh composed of pixel points is optimized.

Citation Information

Patent Citations

  • Double-hemispheric capsule robot attitude detection method based on magnetic vector error calibration image

    CN110327011A

  • Unmanned aerial vehicle navigation map construction system and method based on image three-dimensional reconstruction technology

    CN111599001A

  • Unmanned aerial vehicle image rapid three-dimensional reconstruction method for field unknown environment

    CN112085844A

  • Soft medical capsule robot based on magnetorheological fluid

    CN214965292U

  • Unmanned aerial vehicle navigation map construction system and method based on three-dimensional image reconstruction technology

    US20210358206A1