Endoscopic-image-based three-dimensional reconstruction method and apparatus for pediatric adenoid situation

Abstract

The present application relates to the technical field of medical three-dimensional reconstruction, and particularly relates to an endoscopic-image-based three-dimensional reconstruction method and apparatus for a pediatric adenoid situation, which method and apparatus can solve, to a certain extent, the problems whereby existing three-dimensional reconstruction methods in the medical field still lack research on three-dimensional reconstruction of adenoid regions, and reconstruction results in existing technology have low completeness and cannot meet clinical requirements. The method comprises: by means of local plane parameter initialization, establishing relationships between corresponding points in different images; by means of local plane parameter optimization, accurately updating local plane parameters of each pixel point in a reference image; on the basis of a filtering strategy using relative depth difference, implementing depth filtering; by means of a spatial grid surface fitting method, restoring missing depth information of some regions in a depth map after the depth filtering, so as to densify a reconstruction result; and by taking into consideration two metrics, i.e., relative depth difference and reprojection error, converting into a point cloud the depth map which has been subjected to surface fitting, so as to reconstruct a three-dimensional model of an adenoid region.

Description

Endoscope image based three-dimensional reconstruction method and device for adenoid scene of children TECHNICAL FIELD

[0001] The present application relates to the technical field of medical three-dimensional reconstruction, in particular to an endoscope image based three-dimensional reconstruction method and device for adenoid scene of children. BACKGROUND

[0002] Research shows that adenoid hypertrophy is one of the main causes of obstructive sleep apnea syndrome (OSAS) in children, and evaluating the condition of adenoid plays a very important role in the treatment of OSAS. The current diagnostic method mainly relies on electronic nasopharyngeal endoscopy, which evaluates adenoid through two-dimensional images. Due to the lack of three-dimensional information, the diagnostic results are highly dependent on the observation angle and personal experience of the doctor, which may lead to different diagnoses for the same case. This dependency not only affects the accuracy of the diagnosis, but also may confuse doctors and parents, thereby affecting the effective management and treatment of the disease.

[0003] In the past two decades, computer vision technology has undergone significant development, especially in the medical field. Three-dimensional reconstruction technology based on endoscope images has been widely used. Currently, three-dimensional reconstruction of endoscope images mainly relies on three technologies: shape from shading (SFS), multi-view stereo (MVS), and three-dimensional reconstruction methods based on special camera technology. The SFS method uses the relationship between illumination and imaging model to obtain the depth information of the object by solving partial differential equations, but the accuracy of this method is highly dependent on the lighting conditions and optical parameters, which limits its application in three-dimensional reconstruction of endoscope images. The MVS method reconstructs three-dimensional structures by matching image blocks at different angles, but when dealing with weak texture or smooth areas, image block matching becomes difficult, affecting the effectiveness of the reconstruction. Adenoid, as a lymphatic tissue, has the characteristics of strong surface reflection, unclear features, and smooth scene, making it difficult for existing MVS methods to accurately reconstruct its three-dimensional structure. In addition, although some studies have attempted to use endoscopes based on Time of Flight (ToF) technology or binocular stereo cameras for three-dimensional reconstruction to directly obtain three-dimensional information of the inner cavity, these endoscopes are usually large in size and are not suitable for endoscopy of the nasopharyngeal cavity of children.

[0004] In summary, although there are a large number of three-dimensional reconstruction methods in the field of medical images, the three-dimensional reconstruction research of adenoid region is still blank, and the existing three-dimensional reconstruction methods cannot well solve the three-dimensional reconstruction problem of adenoid scene, the completeness of the reconstruction results of these methods is low, and the clinical needs cannot be met.

[0005] SUMMARY

[0006] In order to solve the problem that the existing three-dimensional reconstruction method in the medical field, the three-dimensional reconstruction research of adenoid region is still blank, and the completeness of the reconstruction results of the existing technology is low, and the clinical needs cannot be met, the application provides a three-dimensional reconstruction method and device of adenoid scene of children based on endoscopic images.

[0007] Embodiments of the application are implemented as follows:

[0008] In a first aspect, the application provides a three-dimensional reconstruction method of adenoid scene of children based on endoscopic images, comprising:

[0009] By local plane parameter initialization, the relationship between the corresponding points of different images is established;

[0010] By local plane parameter optimization, the local plane parameters of each pixel point in the reference image are accurately updated;

[0011] A filtering strategy based on depth relative difference is used to realize depth filtering;

[0012] By a spatial grid surface fitting method, the depth information of the part region information loss of the depth map after depth filtering is recovered to reconstruct the dense reconstruction result;

[0013] Considering the depth relative difference and the re-projection error, the depth map after surface fitting is converted into point cloud, and the three-dimensional model of the adenoid region is reconstructed.

[0014] In a possible implementation manner, before the step of establishing the relationship between the corresponding points of different images by local plane parameter initialization, the method further comprises:

[0015] It is assumed that the surface of the adenoid is a smooth surface, and the local neighborhood of any point x on the surface can be regarded as a plane, which can be represented by the depth value and the normal vector of point x in the camera coordinate system. The depth value and the normal vector are the local plane parameters of point x.

[0016] In a possible implementation manner, before the step of establishing the relationship between the corresponding points of different images by local plane parameter initialization, the method further comprises:

[0017] An input image is obtained, and each image in the input image is sequentially taken as a reference image, and the remaining images are taken as source images of the reference image.

[0018] In a possible implementation, the establishing the relationship between the corresponding points of different images by the local plane parameter initialization further includes:

[0019] Randomly initializing each image, that is, randomly assigning a depth value and a normal vector to each pixel point, to construct the local plane parameter;

[0020] Mapping the pixel point and its neighborhood pixels on the reference image to the corresponding positions in the source image through the plane homography transformation;

[0021] Calculating the zero-mean normalized cross-correlation coefficient of each point relative to each source image to estimate the initial photometric cost.

[0022] In a possible implementation, the updating the local plane parameter of each pixel point in the reference image by the local plane parameter optimization further includes:

[0023] For a certain local plane parameter at a certain pixel of the reference image, the weight of each source image relative to the local plane parameter is calculated by calculating the photometric cost of the neighborhood pixels of the pixel relative to each source image;

[0024] The aggregate cost of the local plane parameter at the pixel is calculated according to the weights provided by all source images, and the information of multiple views is balanced using the aggregate cost;

[0025] More local plane parameters are explored by slightly disturbing the existing parameters and randomly generating new parameters, and finally the parameter with the lowest photometric cost is selected as the final solution.

[0026] In a possible implementation, the depth filtering based on the depth relative difference further includes:

[0027] Unreliable points are identified and removed by comparing the depth difference between each point in the reference image and the corresponding point in the source image.

[0028] In a possible implementation, the dense reconstruction result is obtained by restoring the depth information of the partial region information lost in the depth map after the depth filtering through the spatial grid surface fitting method, and the method further includes:

[0029] The depth values of the filtered reliable point pixels are fitted using a smooth triangular patch in the depth space to obtain a smooth estimation of the image depth values.

[0030] In a possible implementation, the three-dimensional model of the adenoid region is reconstructed by converting the depth map after the surface fitting into a point cloud by considering the two indexes of the depth relative difference and the re-projection error, and the method further includes:

[0031] Based on depth consistency and re-projection consistency, the depth information of all input images is fused to obtain a three-dimensional point cloud of the adenoid region.

[0032] A three-dimensional model of the adenoid region is reconstructed according to the three-dimensional point cloud, and a three-dimensional reconstruction result is obtained.

[0033] In a possible implementation, after the step of recovering the depth information of the partial region of the depth map that loses information after depth filtering by the spatial grid surface fitting method, the method further includes:

[0034] The fitted local plane parameters are updated using a local plane parameter optimization algorithm, and the iteration is performed for three times.

[0035] In a second aspect, the application provides a three-dimensional reconstruction device for a child adenoid scene based on an endoscope image, which includes:

[0036] A parameter initialization module is configured to establish a relationship between corresponding points of different images by local plane parameter initialization.

[0037] A parameter optimization module is configured to accurately update the local plane parameters of each pixel point in the reference image by local plane parameter optimization.

[0038] A depth filtering module is configured to implement depth filtering based on a filtering strategy of depth relative difference.

[0039] A surface fitting module is configured to recover the depth information of the partial region of the depth map that loses information after depth filtering by a spatial grid surface fitting method to densify the reconstruction result.

[0040] A depth fusion module is configured to convert the depth map after surface fitting into a point cloud and reconstruct a three-dimensional model of the adenoid region by considering two indexes of depth relative difference and re-projection error.

[0041] The technical scheme provided by the application can achieve at least the following beneficial effects:

[0042] The three-dimensional reconstruction method for a child adenoid scene based on an endoscope image provided by the application proposes a complete algorithm framework for the task of adenoid reconstruction, effectively reconstructs the adenoid region, opens the application of the multi-view stereo matching method in the field of adenoid reconstruction, and through the proposed local plane parameter optimization algorithm, more efficient local plane parameter propagation can be performed in the adenoid region, and further search for better local plane parameters can be performed to provide a good initial estimation result for subsequent grid surface fitting. In addition, the grid surface fitting algorithm is proposed for the smooth features of the adenoid, an optimization function for surface fitting is constructed, and a sparse matrix is used for rapid solution. Compared with the traditional method, the reconstruction result of the adenoid region is more complete and has better robustness. BRIEF DESCRIPTION OF DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort.

[0044] Fig. 1 is a flow diagram of a method for three-dimensional reconstruction of a child adenoid scene based on an endoscopic image according to an example embodiment of the present application;

[0045] Fig. 2 is a schematic diagram of a chessboard sampling pattern according to an example embodiment of the present application;

[0046] Fig. 3 is a schematic diagram of a device for three-dimensional reconstruction of a child adenoid scene based on an endoscopic image according to an example embodiment of the present application;

[0047] Fig. 4 is a flow diagram of a working process of a three-dimensional reconstruction method according to an example embodiment of the present application;

[0048] Fig. 5 is a schematic diagram of a comparison of adenoid point cloud results reconstructed by different methods in a real experiment. DETAILED DESCRIPTION

[0049] In order to make the purpose, implementation and advantages of the present application more clear, the example embodiments of the present application will be described clearly and completely below with reference to the drawings of the example embodiments of the present application. Obviously, the described example embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not intended to limit the present application.

[0050] It should be noted that the brief description of the terms in the present application is only for the convenience of understanding the following described embodiments, and is not intended to limit the embodiments of the present application. Unless otherwise specified, these terms should be understood according to their ordinary and general meanings.

[0051] The terms "first", "second", "third", etc. in the specification and claims of the present application and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit the specific order or sequence, unless otherwise specified. It should be understood that the terms used in this way can be interchanged under appropriate circumstances.

[0052] The terms "comprise" and "comprising", and any variations thereof, such as "comprises" and "comprising", are intended to cover both the singular and the plural, i.e., "comprising a" means "comprising one or more" and "comprising" means "comprising one or more of".

[0053] Before the method for three-dimensional reconstruction of adenoid scene of children based on endoscopic images provided by the embodiments of the present application is explained and described, the application scenarios and implementation environments of the embodiments of the present application are introduced.

[0054] Research shows that adenoid hypertrophy is one of the main causes of obstructive sleep apnea syndrome (OSAS) in children, and the evaluation of adenoid condition plays a very important role in the treatment of OSAS. The current diagnostic method mainly relies on electronic nasopharyngeal endoscopy, which evaluates the adenoid through two-dimensional images. Due to the lack of three-dimensional information, the diagnostic results are highly dependent on the observation angle and personal experience of the doctor, which may lead to differences in the diagnosis of the same case. This dependence not only affects the accuracy of the diagnosis, but also may confuse the doctors and parents of the children, thereby affecting the effective management and treatment of the disease.

[0055] In the past two decades, computer vision technology has undergone significant development, especially in the medical field, three-dimensional reconstruction technology based on endoscopic images has been widely used. At present, three-dimensional reconstruction of endoscopic images mainly relies on several technologies: shape from shading (SFS), multi-view stereo (MVS), and three-dimensional reconstruction method based on special camera technology. The SFS method uses the relationship between illumination and imaging model to obtain the depth information of the object by solving partial differential equations, but the accuracy of this method is highly dependent on the lighting conditions and optical parameters, which limits its application in three-dimensional reconstruction of endoscopic images. The MVS method reconstructs the three-dimensional structure by matching image blocks at different angles, but when dealing with weak texture or smooth areas, image block matching becomes difficult, affecting the effectiveness of the reconstruction. Adenoids, as a lymphatic tissue, have the characteristics of strong surface reflection, unclear features, and smooth scene, making it difficult for existing MVS methods to accurately reconstruct its three-dimensional structure. In addition, although some studies have tried to use endoscopes based on Time of Flight (ToF) technology or binocular stereo cameras for three-dimensional reconstruction to directly obtain the three-dimensional information of the inner cavity, the volume of these endoscopes is usually large, which is not suitable for endoscopic examination of the nasopharyngeal cavity of children.

[0056] Although there have been many studies on three-dimensional reconstruction of endoscopic images, research specifically on adenoid regions has not been fully developed. After analyzing the above reconstruction methods, it is considered that the MVS method is the most suitable method for the reconstruction of adenoid regions. The MVS method makes full use of image information from multiple perspectives to complete three-dimensional reconstruction. Shen first established a multi-view reconstruction framework based on block matching. The matching relationship between different images is established through homography transformation, and then the optimal depth of the pixel points is searched based on the block matching method to achieve the effect of dense reconstruction. However, the accuracy and running efficiency of this algorithm are relatively low. Subsequently, Zheng et al. optimized the MVS framework through a graph model. The speed of reconstruction is improved by using parallel computing, and the accuracy of reconstruction is improved by using more detailed view information. However, this algorithm has a huge amount of calculation, and cannot reconstruct regions with few features. On this basis, Galliani et al. added the consideration of depth, normal vector and geometric constraints between image pixels, which improved the accuracy of reconstruction. However, the efficiency of this method decreases when dealing with high-resolution images. To solve this problem, Galliani proposed a red and black chessboard depth information propagation strategy, which can use GPU to simultaneously process half of the pixels in the image, greatly improving the efficiency of the algorithm. However, the simplified cost calculation affects the reconstruction quality. Xu et al. subsequently proposed an adaptive depth information propagation scheme, which uses a larger range of depth information for propagation, and proposed a multi-depth hypothesis cost aggregation strategy, effectively utilizing multi-view information, and significantly improving the reconstruction effect. However, the Xu method still cannot provide satisfactory three-dimensional reconstruction of flat areas.

[0057] In summary, although there have been a large number of three-dimensional reconstruction methods in the field of medical images, research on three-dimensional reconstruction of adenoid regions is still in the blank stage, and existing three-dimensional reconstruction methods cannot well solve the problem of three-dimensional reconstruction of adenoid scenes. The reconstruction results of these methods are low in completeness and cannot meet the clinical needs.

[0058] Based on this, the present application provides a three-dimensional reconstruction method and device for adenoid scenes of children based on endoscopic images. The multi-view stereo matching technology is used to obtain a rough estimate of the depth map corresponding to the image, and then the rough depth information is fitted in the depth space using a grid surface, thereby obtaining a smooth and fine depth map. Finally, a point cloud fusion algorithm is used to obtain a dense and accurate three-dimensional reconstruction of the adenoid region.

[0059] Next, the technical solutions of the present application and how the technical solutions of the present application solve the above technical problems will be described in detail through embodiments and in conjunction with the drawings. Each embodiment can be combined with each other, and the same or similar concepts or processes can not be described again in some embodiments. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments.

[0060] FIG. 1 is a flowchart of a method for three-dimensional reconstruction of a child adenoid scene based on endoscopic images according to an example embodiment of the present application.

[0061] In one example embodiment, as shown in FIG. 1, a method for three-dimensional reconstruction of a child adenoid scene based on endoscopic images is provided, which can include the following steps:

[0062] Step 100: Establish the relationship between the corresponding points of different images by local plane parameter initialization.

[0063] Step 200: Accurately update the local plane parameters of each pixel point in the reference image by local plane parameter optimization.

[0064] Step 300: Realize depth filtering based on the filtering strategy of depth relative difference.

[0065] Step 400: Recover the depth information of the partial region information loss of the depth map after depth filtering by the spatial grid surface fitting method to reconstruct the dense result.

[0066] Step 500: Convert the depth map after surface fitting into a point cloud by considering the two indicators of depth relative difference and re-projection error to reconstruct the three-dimensional model of the adenoid region.

[0067] FIG. 2 is a schematic diagram of a chessboard sampling pattern according to an example embodiment of the present application.

[0068] In one possible implementation, the specific implementation of the present application is as follows:

[0069] 1) Establish the relationship between the corresponding points of different images by local plane parameter initialization;

[0070] Given an image sequence {Is, s = 1,.., N}, the camera coordinate system corresponding to Is is denoted as Cs.

[0071] For any reference image Ii, some embodiments of the present application randomly assign a depth value d and a normal vector n to each pixel point of Ii, thereby constructing the local plane parameters of the pixel point.

[0072] The depth d is the Z component of the spatial point corresponding to the pixel point in the camera coordinate system Ci, and the normal vector n is the normal vector of the spatial plane where the spatial point corresponding to the pixel point is located.

[0073] Some embodiments of the present application assume that the camera parameters of the reference image Ii are P i = [K i ,R i ,t i ], and the camera matrix of the source image Ij is P j = [Kj ,R j ,t j ], where Ki and Kj are the intrinsic parameters of the camera, Ri and Rj are the rotation matrices from camera coordinate system C1 to camera coordinate systems Ci and Cj, respectively, and ti and tj are the translation vectors from C1 to Ci and Cj, respectively.

[0074] For any pixel p in Ii, assume its corresponding homogeneous coordinates are p = [u, v, 1] T The corresponding depth values ​​and normal vectors are di and ni, respectively. Their corresponding points q = H in the source image Ij can be obtained through planar homography. ji p, where the expression for Hji is as follows:

[0075] The local plane parameter initialization method is as follows:

[0076] Some embodiments of this application are in the depth range [d] min ,d max d is obtained by random sampling in ], where dmin and dmax are the preset maximum and minimum depth values, respectively.

[0077] For the normal vector, some embodiments of this application let n = [cosθsinφsinθsinφcosφ]. T , where θ ranges from [0, 2π]; φ is the angle between the back projection line of the pixel and the Z-axis. Some embodiments of this application believe that if a point on the adenoid is visible, then the φ value corresponding to this point should fall within the interval [0, π / 3]. Therefore, some embodiments of this application randomly sample within this interval to obtain the initial value of φ.

[0078] Some embodiments of this application use formula (1) to calculate the zero-mean normalized cross-correlation (ZNCC) of each pixel in the reference image relative to each source image, and then use the average of the three smallest ZNCC values ​​as the initial photometric cost of that point.

[0079] 2) By optimizing local plane parameters, the local plane parameters of each pixel in the reference image are accurately updated.

[0080] Based on the photometric cost of each pixel, some embodiments of this application will use a parameter propagation and random perturbation strategy to update the local plane parameters of each pixel in the reference image.

[0081] The algorithm principle is explained as follows:

[0082] To improve the robustness and efficiency of the algorithm, some embodiments of the present application use the checkerboard pattern shown in FIG. 2 for parameter propagation and update.

[0083] For any pixel p (red dot in FIG. 2) in the reference image Ii, some embodiments of the present application define its eight neighbors according to the scope covered by the green box in FIG. 2.

[0084] Search for the pixel with the minimum photometric cost in each neighbor, and the pixel involved in the comparison in each neighbor is denoted by the blue dot in FIG. 2.

[0085] Define the set where θ0is the local plane parameter corresponding to pixel p, and θ1to θ8are the local plane parameters corresponding to the pixels with the minimum photometric cost in the first to eighth neighbors of pixel p, respectively.

[0086] For each source image Ij, the ZNCC value of pixel p with respect to Ij can be calculated by Obviously, for each Ij, pixel p has nine corresponding ZNCC values Therefore, pixel p has a total of 9 x (N-1) ZNCC values.

[0087] Next, some embodiments of the present application calculate the weight of source image Ij with respect to pixel p according to

[0088] For each source image Ij, some embodiments of the present application define the set

[0089] If the number of elements in Spj is greater than n, some embodiments of the present application consider that source image Ij contributes to the reference image, and therefore put the index j into the set .

[0090] Then, the confidence of each in Spj is calculated by formula (2), which will be used to calculate the weight of source image Ij with respect to pixel p.

[0091] where β is a constant. The weight of source image Ij with respect to pixel p can be calculated by the following formula:

[0092] where median represents the median of the set. Finally, the photometric cost of pixel p with respect to Ii is

[0093] ​​​Next, some embodiments of the present application use the local plane parameter updated by the minimum photometric cost to update the local plane parameter of the current pixel p.

[0094] 3) Depth filtering is implemented based on the depth relative difference filtering strategy.

[0095] The above optimization process only considers the photometric consistency between different images. Since the adenoid image has few features and a single color, there are many obvious noise points and error points in the depth map obtained.

[0096] Therefore, after depth optimization, some embodiments of the present application filter out the obvious noise points and error points in the depth map corresponding to all views based on the depth relative difference.

[0097] For any point p in the reference image Ii, some embodiments of the present application find the corresponding point q of p in the source image Ij through formula (1), and the depth value of q is dq.

[0098] The depth value of the spatial point corresponding to point p in the camera coordinate system of the source image Ij is dp, and the depth relative difference is If at least n source images satisfy drj≤0.01, some embodiments of the present application consider that point p is reliable, otherwise the local plane parameter of p is deleted. The filtering algorithm removes most of the obvious noise points and error points, thereby providing reliable data for subsequent mesh surface fitting.

[0099] 4) The depth information lost in some regions of the depth map after depth filtering is recovered by a spatial mesh surface fitting method to reconstruct the dense result.

[0100] The filtering algorithm in the last section removes unreliable depth information in the depth map, thereby causing the depth loss in some regions of the depth map.

[0101] To recover the depth information, some embodiments of the present application propose a spatial mesh surface fitting method. Specifically, some embodiments of the present application consider that the neighborhood of each point on the nasopharyngeal cavity surface can be approximated as a small plane, and the normal vectors of the neighborhoods of two nearby points are similar.

[0102] Therefore, some embodiments of the present application use a smooth triangular patch surface to fit the depth values of the filtered reliable point pixels in the depth space, thereby obtaining a smooth estimation of the image depth values.

[0103] First, some embodiments of the present application define that the xy plane of the depth space coincides with the image plane. In the depth space, for any triangular patch Ti, the points inside Ti satisfy the plane equation d=ax+by+c (5)

[0104] Where d is the depth value of a point on Ti, [x,y] are the pixel coordinates of the point on Ti, and [a,b,c] are the plane parameters. Let the vertices of the triangular patch Ti be derived from... It consists of three points, among which and for The x and y coordinates, For point The depth value.

[0105] Therefore, Substituting into formula (5) yields

[0106] Substituting formula (6) into formula (5), we can see that the depth value of any point within the triangular facet Ti can be calculated using the following formula.

[0107] in, It can be obtained by triangulation of the xy plane (image plane) in depth space, and therefore is a constant value.

[0108] For the variable to be determined, different This determines the different positions of the triangular facets in depth space. Some embodiments of this application approximate the 3D information of the scene using triangular facets, striving to make the curved surfaces of the triangular facets fit the depth of reliable pixels as closely as possible. Simultaneously, to smooth the curved surfaces of the triangular facets, some embodiments of this application require that the parameters of two adjacent triangular facets be as consistent as possible. Ultimately, some embodiments of this application construct the following equation:

[0109] In the above formula, T represents the set of all triangular facets; pro(T) i ) represents the orthogonal projection of the triangular facet Ti onto the xy plane. It means pro(T) i Reliable pixel coordinates within ) for The homogeneous coordinates of the plane; adj is the set of adjacent plane pairs; for The corresponding depth value; λ is the smoothing parameter.

[0110] In the above formula, the data term represents the difference between the fitted depth value and the depth value of the reliable pixel at the reliable pixel, and the smoothing term represents the difference in parameters between two adjacent triangular faces.

[0111] The above equation is a quadratic programming problem. Some embodiments of this application use sparse matrix theory to quickly obtain the depth values ​​corresponding to the vertices of all triangular faces. Thus, the plane equation of each triangular facet is obtained. Then, some embodiments of the present application estimate the depth value of all pixel points in the xy plane (image plane) using the plane equation of all triangular facets, which can optimize the original reliable depth value while filling the depth map hole. After completing the depth fitting, some embodiments of the present application update the fitting result using the local plane parameter optimization algorithm, and iterate 3 times.

[0112] 5) Considering the depth relative difference and the re-projection error, the depth map after surface fitting is converted into a point cloud, and a three-dimensional model of the adenoid region is reconstructed.

[0113] Finally, some embodiments of the present application fuse the fitted depth map into a point cloud. The fusion conditions include the depth relative difference and the re-projection error. The calculation method of the depth relative difference is as described in the depth filtering.

[0114] The homography matrix Hji of the neighborhood of a point xi in the reference image Ii to the source image Ij can be determined by (1).

[0115] Assuming that the corresponding point of the point xi in the source image Ij is xj = Hjixi, the homography matrix Hij of the neighborhood of xj to the image Ii is easily obtained from the local plane parameters at xj.

[0116] Then, the corresponding point of xj in Ii is The re-projection error of xi with respect to Ij is calculated as follows:

[0117] wherein represents the non-homogeneous coordinates of the point x. If at least m source images satisfy: ereproj≤2, drj≤0.01, the corresponding three-dimensional point of xi is retained, and the corresponding point of xi in all source images satisfying the above conditions is deleted to avoid repeated calculation.

[0118] The final obtained three-dimensional point cloud is the three-dimensional reconstruction result of the adenoid region.

[0119] It should be understood that, although each step in the flowchart involved in the above embodiments is displayed in sequence as indicated, these steps are not necessarily executed in sequence as indicated. Unless explicitly stated herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other sequences. Moreover, at least part of the steps in the flowchart involved in the above embodiments can include multiple steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or steps or stages in other steps.

[0120] Corresponding to the foregoing embodiments of the method for three-dimensional reconstruction of a child adenoid scene based on endoscopic images, the same technical concept is adopted, and the application further provides embodiments of a device for three-dimensional reconstruction of a child adenoid scene based on endoscopic images.

[0121] Fig. 3 is a structural schematic diagram of a device for three-dimensional reconstruction of a child adenoid scene based on endoscopic images according to an example embodiment of the application.

[0122] In an example embodiment, as shown in Fig. 3, the device for three-dimensional reconstruction of a child adenoid scene based on endoscopic images comprises:

[0123] a parameter initialization module configured to establish a relationship between corresponding points of different images through local plane parameter initialization;

[0124] a parameter optimization module configured to accurately update the local plane parameters of each pixel point in the reference image through local plane parameter optimization;

[0125] a depth filtering module configured to implement depth filtering based on a filtering strategy of depth relative difference;

[0126] a curved surface fitting module configured to recover the depth information of a partial region of the depth map after depth filtering, which loses information, through a spatial grid curved surface fitting method, to densify the reconstruction result;

[0127] a depth fusion module configured to convert the depth map after curved surface fitting into a point cloud to reconstruct a three-dimensional model of the adenoid region by considering two indexes of depth relative difference and re-projection error.

[0128] The specific limitations of the device for three-dimensional reconstruction of a child adenoid scene based on endoscopic images can be referred to the limitations of the method for three-dimensional reconstruction of a child adenoid scene based on endoscopic images described above, which will not be repeated here. The various modules in the device for three-dimensional reconstruction of a child adenoid scene based on endoscopic images described above can be realized by software, hardware and combinations thereof in whole or in part. The various modules described above can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to the various modules.

[0129] Fig. 4 is a flow schematic diagram of the working process of the three-dimensional reconstruction method according to an example embodiment of the application.

[0130] The working process is shown in Fig. 4, and specifically includes the following steps:

[0131] First, input images are acquired, and each image in the input images is sequentially taken as a reference image, and the remaining images are taken as source images of the reference image.

[0132] Then, the local plane parameters corresponding to each pixel point in the reference image are randomly initialized, and based on the local plane parameters corresponding to each point in the reference image, homographic transformation between the neighborhood of the point and all source images is established, and the photometric cost of the point is constructed.

[0133] According to the photometric cost of each pixel, the local plane parameters of each pixel point in the reference image are updated through parameter propagation and random disturbance strategy.

[0134] Then, the obviously incorrect local plane parameters are filtered out based on depth consistency, and the depth map of the reference image is fitted in depth space using mesh surface to obtain smooth local plane parameters.

[0135] The fitted local plane parameters are updated according to the local plane parameter optimization algorithm, and iteration is performed for 3 times.

[0136] Finally, based on the depth consistency and the reprojection consistency, the three-dimensional point cloud of the adenoid region is obtained by fusing the depth information of all input images, and the reconstruction result is obtained.

[0137] The technical features of the above embodiments can be combined in any way. To make the description concise, not all possible combinations of the technical features in the above embodiments are described, but as long as the combinations of the technical features do not exist contradictions, they should be considered as the scope of the present application.

[0138] Fig. 5 is a comparison diagram of adenoid point cloud results reconstructed by different methods in real experiments.

[0139] In order to verify the feasibility of the present application, as shown in Fig. 5, in real experiments, the point cloud result of the present method is obviously more dense and smoother than the existing algorithm, and more consistent with the structure characteristics of adenoid.

[0140] The above embodiments only express several embodiments of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, some modifications and improvements can be made, which are all within the scope of the present application. Therefore, the protection scope of the present application patent should be subject to the appended claims.

Claims

1. A method for three-dimensional reconstruction of a child adenoid scene based on endoscopic images, characterized in that, The method comprises the following steps: establishing the relationship between the corresponding points of different images by local plane parameter initialization; accurately updating the local plane parameters of each pixel point in the reference image by local plane parameter optimization; implementing depth filtering based on a filtering strategy of depth relative difference; restoring the depth information of the partial region information loss of the depth map after depth filtering by a spatial grid surface fitting method to reconstruct the dense reconstruction result; considering two indexes of depth relative difference and re-projection error, converting the depth map after surface fitting into point cloud to reconstruct the three-dimensional model of the adenoid region.

2. The endoscopic image-based three-dimensional reconstruction method of a child adenoid scene according to claim 1, wherein, Before the step of establishing the relationship between the corresponding points of different images by local plane parameter initialization, the method further comprises the following steps: assuming that the surface of the adenoid is a smooth surface, and the local neighborhood of any point x on the surface can be regarded as a plane, which can be represented by the depth value and normal vector of the point x in the camera coordinate system, and the depth value and normal vector are the local plane parameters of the point x.

3. The endoscopic image-based three-dimensional reconstruction method of a child adenoid scene according to claim 1, wherein, Before the step of establishing the relationship between the corresponding points of different images by local plane parameter initialization, the method further comprises the following steps: obtaining input images, and sequentially taking each image in the input images as a reference image and taking the remaining images as source images of the reference image.

4. The endoscopic image-based three-dimensional reconstruction method of a child adenoid scene according to claim 1, wherein, The step of establishing the relationship between the corresponding points of different images by local plane parameter initialization further comprises the following steps: randomly initializing each image, that is, randomly assigning a depth value and a normal vector to each pixel point to construct the local plane parameters; mapping the pixel point and its neighborhood pixels on the reference image to the corresponding positions in the source images by a plane homography transformation; calculating the zero-mean normalized cross-correlation coefficient of each point relative to each source image to estimate the initial photometric cost.

5. The endoscopic image-based three-dimensional reconstruction method of a child adenoid field according to claim 1, wherein, The step of accurately updating the local plane parameters of each pixel point in the reference image by local plane parameter optimization further comprises the following steps: for a certain local plane parameter at a certain pixel of the reference image, calculating the weight of each source image relative to the local plane parameter by calculating the photometric cost of the neighborhood pixels of the pixel relative to each source image; calculating the aggregated cost of the local plane parameter at the pixel according to the weights provided by all the source images, and balancing the information of multiple views by using the aggregated cost; exploring more local plane parameters by slightly perturbing the existing parameters and randomly generating new parameters, and finally selecting the parameter with the lowest photometric cost as the final solution.

6. The endoscopic image-based three-dimensional reconstruction method of a child adenoid scene according to claim 1, wherein, The step of implementing depth filtering based on a filtering strategy of depth relative difference further comprises the following steps: identifying and removing unreliable points by comparing the depth difference between each point in the reference image and the corresponding point in the source image.

7. The endoscopic image-based three-dimensional reconstruction method of a child adenoid field according to claim 1, wherein, The step of restoring the depth information of the partial region information loss of the depth map after depth filtering by a spatial grid surface fitting method to reconstruct the dense reconstruction result further comprises the following steps: using a smooth triangular patch surface to fit the depth values of the filtered reliable point pixels in the depth space to obtain a smooth estimation of the image depth values. The step of considering two indexes of depth relative difference and re-projection error, converting the depth map after surface fitting into point cloud to reconstruct the three-dimensional model of the adenoid region further comprises the following steps:

8. The endoscopic image-based three-dimensional reconstruction method of a child adenoid scene according to claim 1, wherein, ​ Based on depth consistency and re-projection consistency, three-dimensional point cloud of the adenoid region is obtained by fusing depth information of all input images; A three-dimensional model of the adenoid region is reconstructed according to the three-dimensional point cloud, and a three-dimensional reconstruction result is obtained.

9. The endoscopic image-based three-dimensional reconstruction method of a child adenoid scene according to claim 1, wherein, After the step of recovering the depth information of the partial region information loss of the depth map after the depth filtering by the spatial grid surface fitting method, the three-dimensional reconstruction result is densified. The fitted local plane parameters are updated by using a local plane parameter optimization algorithm, and the iteration is performed for 3 times.

10. An apparatus for three-dimensional reconstruction of a child adenoid scene based on endoscopic images, characterized by It comprises: A parameter initialization module is configured to establish a relationship between corresponding points of different images by local plane parameter initialization; A parameter optimization module is configured to accurately update the local plane parameters of each pixel point in the reference image by local plane parameter optimization; A depth filtering module is configured to implement depth filtering based on a filtering strategy of depth relative difference; A surface fitting module is configured to recover the depth information of the partial region information loss of the depth map after the depth filtering by the spatial grid surface fitting method, and to densify the reconstruction result; A depth fusion module is configured to consider two indexes of depth relative difference and re-projection error, convert the surface fitted depth map into a point cloud, and reconstruct a three-dimensional model of the adenoid region.

Images