Three-dimensional mapping method, terminal equipment and computer readable storage medium

Abstract

The invention is suitable for the technical field of image processing, and particularly relates to a three-dimensional mapping method, terminal equipment and a computer readable storage medium. The method comprises the following steps: acquiring a first shot image of a target object; wherein the first shot image is a two-dimensional image; generating a first point cloud picture according to the first shot image; performing densification processing according to the first point cloud image to obtain a second point cloud image; wherein the point cloud density of the second point cloud picture is greater than the point cloud density of the first point cloud picture; and constructing a three-dimensional image of the target object according to the second point cloud image. Through the method, the positioning precision and the real-time performance of three-dimensional mapping can be effectively improved.

Description

Technical Field

[0001] This application belongs to the technical field of image processing, and particularly relates to a three-dimensional mapping method, a terminal device, and a computer-readable storage medium. Background Art

[0002] Visual Simultaneous Localization and Mapping (V-SLAM) technology has been widely applied in various fields. For example, in the medical field, V-SLAM technology can help doctors perform accurate navigation and positioning during minimally invasive surgeries, enabling precise surgical operations in complex anatomical structures to reduce injuries. It can also be combined with medical imaging technologies, such as providing more accurate real-time positioning and navigation during endoscopic examinations, helping doctors better understand and handle patients' conditions and accurately reach the lesion location.

[0003] Currently, in the medical field, V-SLAM technology usually relies on manual calibration or external auxiliary devices for positioning. This method has low positioning accuracy and poor real-time performance, making it difficult to meet the requirements of real-time operations, thus affecting the processing efficiency during surgeries or diagnoses. Summary of the Invention

[0004] Embodiments of this application provide a three-dimensional mapping method, a terminal device, and a computer-readable storage medium, which can effectively improve the positioning accuracy and real-time performance of three-dimensional mapping.

[0005] In a first aspect, embodiments of this application provide a three-dimensional mapping method, including:

[0006] Obtain a first captured image of a target object; wherein, the first captured image is a two-dimensional image;

[0007] Generate a first point cloud map based on the first captured image;

[0008] Perform densification processing on the first point cloud map to obtain a second point cloud map; wherein, the point cloud density of the second point cloud map is greater than that of the first point cloud map;

[0009] Construct a three-dimensional image of the target object based on the second point cloud map.

[0010] In embodiments of this application, two-dimensional images can be processed in real time to meet the real-time requirements of three-dimensional mapping. Secondly, generating a dense three-dimensional point cloud from two-dimensional images greatly increases the number of map points and improves map details, thus facilitating the improvement of the positioning accuracy of three-dimensional mapping.

[0011] In a possible implementation of the first aspect, the densification process based on the first point cloud map to obtain a second point cloud map includes:

[0012] Represent the point cloud in the first point cloud map as a three-dimensional Gaussian distribution;

[0013] Perform densification processing on the three-dimensional Gaussian distribution to obtain a dense Gaussian distribution;

[0014] Obtain the second point cloud map according to the dense Gaussian distribution.

[0015] Through the above method, each three-dimensional point is no longer a simple coordinate, but is represented by a Gaussian distribution. In this way, each point cloud is given a distribution area, which can not only enhance the continuity of the point cloud, but also effectively reduce errors caused by perspective differences, lighting changes and noise.

[0016] In a possible implementation of the first aspect, the densification process on the three-dimensional Gaussian distribution to obtain a dense Gaussian distribution includes:

[0017] Perform multi-scale depth estimation based on the three-dimensional Gaussian distribution to obtain depth maps corresponding to each scale;

[0018] Perform detail correction on the depth maps corresponding to each scale to obtain corrected depth maps;

[0019] Obtain the dense Gaussian distribution according to the corrected depth map.

[0020] Through the above method, the obtained dense Gaussian distribution includes rich details, and these details can more accurately reflect the structure of the environment, providing a reliable data basis for the subsequent three-dimensional map.

[0021] In a possible implementation of the first aspect, the obtaining of the second point cloud map according to the dense Gaussian distribution includes:

[0022] Project the dense Gaussian distribution into a two-dimensional image;

[0023] Calculate the difference value between the two-dimensional image and the first captured image;

[0024] Adjust the parameters of the dense Gaussian distribution according to the difference value to obtain an optimized Gaussian distribution;

[0025] Obtain the second point cloud map according to the optimized Gaussian distribution.

[0026] In the above implementation, the difference value between the two-dimensional image obtained from the dense Gaussian distribution and the first captured image captured in reality is used to inversely adjust the parameters of the Gaussian distribution, thereby optimizing the dense Gaussian distribution and further improving the accuracy of 3D mapping.

[0027] In a possible implementation of the first aspect, adjusting the parameters of the dense Gaussian distribution according to the difference value to obtain an optimized Gaussian distribution includes:

[0028] Adjusting the attributes of the elements in the dense Gaussian distribution according to the difference value;

[0029] Adjusting the density of the elements in the dense Gaussian distribution according to the difference value;

[0030] Obtaining the optimized Gaussian distribution according to the attributes and density of the elements in the adjusted dense Gaussian distribution.

[0031] In a possible implementation of the first aspect, generating the first point cloud map according to the first captured image includes:

[0032] Estimating the camera pose according to the first captured image and the second captured image; wherein, the second captured image is the previous captured image of the first captured image, and the second captured image is a two-dimensional image;

[0033] Generating the first point cloud map according to the camera pose.

[0034] In a possible implementation of the first aspect, estimating the camera pose according to the first captured image includes:

[0035] Extracting the first feature points of the first captured image and the second feature points of the second captured image;

[0036] Performing feature matching on the first feature points and the second feature points to obtain multiple groups of matching points;

[0037] Estimating the camera pose according to multiple groups of the matching points.

[0038] In the embodiments of the present application, matching points between two consecutive frames of images are found through feature matching, and then the camera pose is estimated according to the matching points. This method can reduce the pose estimation error caused by feature mismatch and help improve the estimation accuracy of the camera pose.

[0039] In a possible implementation of the first aspect, estimating the camera pose according to multiple groups of the matching points includes:

[0040] Performing fitting of a data model according to multiple groups of the matching points to obtain a fitting model;

[0041] Estimate the camera pose based on the matching points that satisfy the fitting model.

[0042] In the above manner, it has strong robustness to the noise and outliers in the matching points, can accurately find the true model of the data in the presence of a large number of outliers, thereby obtaining the matching points that contribute to the camera pose estimation, and helps to improve the estimation accuracy of the camera pose. And it does not require preprocessing of the data to remove outliers, and the algorithm itself can automatically identify and process outliers.

[0043] In a second aspect, an embodiment of the present application provides a three-dimensional mapping device, including:

[0044] An acquisition unit, configured to acquire a first captured image of a target object; wherein, the first captured image is a two-dimensional image;

[0045] A generation unit, configured to generate a first point cloud map according to the first captured image;

[0046] A processing unit, configured to perform densification processing on the first point cloud map to obtain a second point cloud map; wherein, the point cloud density of the second point cloud map is greater than the point cloud density of the first point cloud map;

[0047] A construction unit, configured to construct a three-dimensional image of the target object according to the second point cloud map;

[0048] In a third aspect, an embodiment of the present application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the three-dimensional mapping method according to any one of the above first aspects.

[0049] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the three-dimensional mapping method according to any one of the above first aspects.

[0050] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product runs on a terminal device, it causes the terminal device to execute the three-dimensional mapping method according to any one of the above first aspects.

[0051] It can be understood that the beneficial effects of the above second aspect to fifth aspect can refer to the relevant descriptions in the above first aspect, and will not be repeated here. Description of the Drawings

[0052] To more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the accompanying drawings required for use in the embodiments or the description of the prior art. Obviously, the accompanying drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0053] Figure 1 is a schematic framework diagram of the three-dimensional mapping process provided by the embodiments of the present application;

[0054] Figure 2 is a schematic flowchart of the three-dimensional mapping method provided by the embodiments of the present application;

[0055] Figure 3 is a schematic flowchart of the generation process of the first point cloud map provided by the embodiments of the present application;

[0056] Figure 4 is a structural block diagram of the three-dimensional mapping device provided by the embodiments of the present application;

[0057] Figure 5 is a schematic structural diagram of the terminal device provided by the embodiments of the present application. Detailed implementation manners

[0058] In the following description, specific details such as specific system structures and technologies are presented for the purpose of illustration rather than limitation, so as to thoroughly understand the embodiments of the present application. However, those skilled in the art should clearly understand that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details from interfering with the description of the present application.

[0059] It should be understood that when used in the specification of the present application and the appended claims, the term "comprising" indicates the presence of the described features, wholes, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their combinations.

[0060] It should also be understood that the term " / and / " used in the specification of the present application and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0061] As used in the specification of this application and the appended claims, the term "if" may be construed contextually as "when" or "once" or "in response to determining" or "in response to detecting". Similarly, the phrase "if determined" or "if [the described condition or event] is detected" may be construed contextually to mean "once determined" or "in response to determining" or "once [the described condition or event] is detected" or "in response to detecting [the described condition or event]".

[0062] In addition, in the description of the specification of this application and the appended claims, the terms "first", "second", "third", etc. are only used for differential description and cannot be construed as indicating or implying relative importance.

[0063] Reference to "one embodiment" or "some embodiments" or the like described in the specification of this application means that a particular feature, structure, or characteristic described in connection with that embodiment is included in one or more embodiments of this application. Thus, statements such as "in one embodiment", "in some embodiments", "in other some embodiments", "in still other embodiments", etc. that appear in different places in this specification do not necessarily all refer to the same embodiment, but rather mean "one or more but not all embodiments", unless otherwise specifically emphasized in another way.

[0064] Visual Simultaneous Localization and Mapping (V-SLAM) technology has found wide applications in various fields. For example, in the medical field, V-SLAM technology can assist doctors in accurate navigation and positioning during minimally invasive surgery, helping doctors perform precise surgical operations in complex anatomical structures to reduce injuries. It can also be combined with medical imaging technologies, such as providing more accurate real-time positioning and navigation in endoscopic examinations, helping doctors better understand and handle patients' conditions and accurately reach the lesion location.

[0065] The medical field has relatively high requirements for real-time performance and accuracy. Especially during surgery, doctors need to obtain accurate environmental information in real time to assist in decision-making and operations. However, currently in the medical field, V-SLAM technology usually relies on manual calibration or external auxiliary devices for positioning. This method has low positioning accuracy and poor real-time performance, making it difficult to meet the requirements of real-time operations, thus affecting the processing efficiency during surgery or diagnosis.

[0066] Based on this, an embodiment of the present application provides a three-dimensional mapping method. In the embodiment of the present application, two-dimensional images can be processed in real time to meet the real-time requirements of three-dimensional mapping; in addition, dense three-dimensional point clouds are generated according to the two-dimensional images, greatly increasing the number of map points and improving the map details, thereby facilitating the improvement of the positioning accuracy of three-dimensional mapping.

[0067] See Figure 1 , which is a schematic framework diagram of the three-dimensional mapping process provided by the embodiment of the present application. As an example rather than a limitation, as Figure 1 shown, obtain a two-dimensional captured image, perform rough initialization on the captured image to obtain a sparse point cloud; then perform densification processing on the sparse point cloud to obtain a dense point cloud; finally, generate a three-dimensional map according to the dense point cloud.

[0068] Taking the endoscopic application scenario as an example, as the endoscope moves, the monocular camera on the endoscope can capture two-dimensional images of the target environment (such as the intestine, stomach, etc.) in real time to obtain the captured images. The endoscope transmits the captured images to the processing device for executing the three-dimensional mapping method provided by the embodiment of the present application in real time. After receiving the captured images, the processing device executes the three-dimensional mapping method according to the Figure 1 process shown, and obtains the three-dimensional map of the target environment in real time.

[0069] Through the above method, a three-dimensional map with high accuracy can be obtained by using monocular vision information (two-dimensional captured images), reducing the dependence of three-dimensional mapping on auxiliary devices, and helping to reduce the hardware cost while improving the accuracy of three-dimensional mapping.

[0070] Based on the Figure 1 framework shown, see Figure 2 , which is a schematic flowchart of the three-dimensional mapping method provided by the embodiment of the present application. As an example rather than a limitation, as Figure 2 shown, the three-dimensional mapping method may include the following steps:

[0071] S201, obtain a first captured image of the target object.

[0072] Wherein, the first captured image is a two-dimensional image.

[0073] Taking the endoscope as an example, the target object may be the object currently captured by the endoscope, such as the intestinal lumen, gastric cavity, etc. As described in the Figure 1 embodiment, the processing device for executing the three-dimensional mapping method can obtain the captured images from the endoscope in real time.

[0074] Optionally, the processing device executes the three-dimensional mapping method each time it receives a captured image. Optionally, the processing device executes the three-dimensional mapping method each time it receives several captured images.

[0075] It can be understood that the processing device can continue to construct the three-dimensional map corresponding to the next captured image based on the three-dimensional map corresponding to the previous captured image. In this way, as the endoscope moves, an overall three-dimensional map including all the positions captured by the endoscope can be obtained, such as an overall three-dimensional map of the intestinal lumen or an overall three-dimensional map of the gastric cavity, etc.

[0076] S202. Generate a first point cloud map according to the first captured image.

[0077] It can be understood that the first point cloud map is equivalent to Figure 1 the sparse point cloud shown in. S202 is equivalent to Figure 1 the process of rough initialization shown in.

[0078] In one embodiment, S202 may include:

[0079] Estimate the camera pose according to the first captured image and the second captured image; wherein, the second captured image is the previous captured image of the first captured image, and the second captured image is a two-dimensional image;

[0080] Generate the first point cloud map according to the camera pose.

[0081] Optionally, the captured image can be preprocessed first. Specifically, it may include image grayscale conversion and / or image scaling.

[0082] Among them, image grayscale conversion refers to the process of converting a color image into a grayscale image. Each pixel of a grayscale image is represented by only one value indicating its brightness, and the value range is usually from 0 (black) to 255 (white). The following image grayscale conversion methods can be used: average method, taking the average value of the R, G, and B components of each pixel in the color image as the grayscale value of the pixel. Weighted average method, considering that the human eye has different sensitivities to different colors, different weights are assigned to the R, G, and B components and weighted. The commonly used weights are: the weight of the R component is 0.299, the weight of the G component is 0.587, and the weight of the B component is 0.114. By performing image grayscale conversion on the input image to convert it into a grayscale image, the calculation amount can be reduced and the robustness of feature point detection can be improved.

[0083] Image scaling can change the size and resolution of the image, which helps to adapt to endoscope images with different resolutions. For example, interpolation method, transform-based method or up / down sampling method can be used to achieve image scaling. Optionally, different sliding windows can be set for different resolutions, such as setting a larger sliding window for a lower resolution and a smaller sliding window for a higher resolution, which helps to improve the motion estimation accuracy between consecutive frames.

[0084] In one embodiment, the step of estimating the camera pose may include:

[0085] Extract the first feature points of the first captured image and the second feature points of the second captured image;

[0086] Perform feature matching on the first feature points and the second feature points to obtain multiple sets of matching points;

[0087] Estimate the camera pose based on multiple sets of the matching points.

[0088] As described in the above embodiments, in the case of performing image preprocessing on the captured image, in this embodiment, correspondingly, extract the first feature points of the preprocessed first captured image and the second feature points of the preprocessed second captured image.

[0089] Optionally, the Scale-Invariant Feature Transform (SIFT) algorithm can be used to extract feature points in the image. Using the SIFT algorithm helps to achieve rotation and scale invariance, thus providing a reliable data basis for subsequent mapping. Exemplarily, the steps of the SIFT algorithm are as follows:

[0090] First, construct a Gaussian pyramid to generate images of different scales:

[0091] L(x, y, σ) = G(x, y, σ) * I(x, y);

[0092] where is the Gaussian kernel function, and σ is the scale parameter. The Gaussian pyramid is constructed through consecutive Gaussian blurring and downsampling operations. Each group contains s + 3 layers of images, where s is the number of layers in the scale space. The first layer image of each group is obtained by downsampling the third-to-last layer image of the previous group. Optionally, the sampling factor is 2.

[0093] Next, use the Difference of Gaussian (DoG) to detect key points. Calculate the difference between Gaussian images of adjacent scales to obtain the DoG image.

[0094] D(x, y, σ) = L(x, y, kσ) - L(x, y, σ);

[0095] Among them, k is the scale multiplication factor, and L(x, y, σ) represents the pixel at coordinates (x, y) with scale σ. Extreme points are searched in the DoG scale space. Each pixel is compared with its 26 neighboring regions (8 neighboring regions of the same scale + 9 upper-level neighboring regions + 9 lower-level neighboring regions). The position of the key point is accurately located by fitting a three-dimensional quadratic function, and low-contrast and edge response points are removed. Then, the gradient direction of the key point neighborhood is calculated. For each key point, the gradient magnitude m(x, y) and direction θ(x, y) of the image L(x, y, σ) at its corresponding scale are calculated:

[0096]

[0097] Finally, a 16×16 neighborhood window is taken with the key point as the center. The window is divided into 4×4 sub-regions, and each sub-region contains 4×4 pixels. The gradient directions in 8 directions are calculated within each sub-region to generate a 128-dimensional feature vector. A 128-dimensional feature descriptor is generated.

[0098] It should be noted that other feature extraction algorithms can be adopted according to the application scenario, and the embodiments of the present application do not make specific limitations on the feature extraction algorithm.

[0099] Optionally, the Fast Library for Approximate Nearest Neighbors (FLANN) can be used for feature matching. FLANN is an algorithm library for nearest neighbor search of high-dimensional data, especially suitable for large data sets. FLANN provides a variety of efficient search algorithms, including the k-Dimensional Tree (KD tree) and linear search, etc., and automatically selects the optimal search algorithm according to the data characteristics in practice. In feature matching, FLANN is usually used for fast nearest neighbor search of high-dimensional features (such as SIFT, SURF, etc. feature descriptors), and reliable matching points are further screened through the ratio test. Error matches are filtered by the Ratio Test method.

[0100] In the embodiments of the present application, matching points between two consecutive frames of images are found through feature matching, and then the camera pose is estimated based on the matching points. This method can reduce the pose estimation error caused by feature mismatching and help improve the estimation accuracy of the camera pose.

[0101] In one implementation, the method for estimating the camera pose based on multiple groups of matching points may include:

[0102] Fitting a data model based on multiple groups of the matching points to obtain a fitting model;

[0103] Estimating the camera pose based on the matching points that satisfy the fitting model.

[0104] Exemplarily, the Random Sample Consensus (RANSAC) algorithm can be used to fit the data model. This algorithm is based on an assumption: the data set contains two groups of data points, one group is the "Inliers" that conform to a certain specific mathematical model, and the other group is the "Outliers" or noise points that do not conform to the model. The goal of the algorithm is to iteratively exclude the interference of outliers as much as possible and find the model parameters that can best fit the inliers. The algorithm may include the following steps:

[0105] First, set the maximum number of iterations N, then set the inlier threshold ∈, and initialize the optimal inlier set During the iteration process, randomly select 8 matching point pairs as a random subset, and calculate E based on the data in the random subset. For example, for the matching point pair (x1, x2), it satisfies the epipolar constraint: where T1 and T2 are normalization matrices. Construct a linear equation system Ae = 0, where A is composed of normalized point coordinates, e is an element of E, perform SVD decomposition on A, and take the vector corresponding to the smallest singular value as the estimate of E.

[0106] Then calculate the symmetric epipolar distance of all points:

[0107]

[0108] Select the points that satisfy d(x1, x2) < ∈ as inliers. If the current inlier set I is larger than I best , update I best = I, and finally update the number of iterations N according to the inlier ratio.

[0109] After obtaining the fitting model, estimate E based on the matching points that satisfy the fitting model, and then perform SVD decomposition on E, which is decomposed into a rotation matrix R and a displacement matrix T; select the correct combination through triangulation, and for each combination, calculate the 3D point depth, and select the combination that makes all point depths positive, so as to estimate the camera pose.

[0110] In the above way, it has strong robustness to the noise and outliers in the matching points, can accurately find the true model of the data in the presence of a large number of outliers, so as to obtain the matching points that contribute to the camera pose estimation, which helps to improve the estimation accuracy of the camera pose. And it does not require preprocessing of the data to remove outliers, and the algorithm itself can automatically identify and process outliers.

[0111] In one implementation, after estimating the camera pose, triangulation can be performed on the matching points, that is, triangular patches are generated from the matching points in the first captured image, and a first point cloud map is generated according to the combination of the triangular patches of each matching point in the first captured image.

[0112] For example, the Bundle Adjustment (BA) method can be used to combine the triangular patches to generate the first point cloud map.

[0113] Optionally, outlier filtering can be performed on the first point cloud map to filter out the significantly abnormal point clouds during the change of the camera pose, so that the filtered first point cloud map is more accurate.

[0114] Exemplarily, refer to Figure 3 , which is a schematic diagram of the generation process of the first point cloud map provided by the embodiment of the present application. As an example rather than a limitation, as Figure 3 shown, first, feature extraction and matching are performed on the first captured image. Specifically, feature extraction is performed on the first captured image to obtain first feature points; the first feature points are feature-matched with the second feature points of the second captured image to obtain matching points; geometric reconstruction is performed on the matching points, that is, the matching points are marked in three-dimensional space.

[0115] Then, sparse reconstruction is performed according to the matching points. Specifically, the camera pose is estimated according to the matching points; then, triangulation is performed on the matching points according to the camera pose; the Bundle Adjustment (BA) method is used to combine the triangular patches to generate the first point cloud map; outlier filtering is performed on the first point cloud map to obtain the filtered first point cloud map.

[0116] S203, perform densification processing on the first point cloud map to obtain a second point cloud map.

[0117] Among them, the point cloud density of the second point cloud map is greater than that of the first point cloud map. It can be understood that the second point cloud map is equivalent to Figure 1 the dense point cloud shown.

[0118] In one embodiment, S203 may include:

[0119] Represent the point cloud in the first point cloud map as a three-dimensional Gaussian distribution;

[0120] Perform densification processing on the three-dimensional Gaussian distribution to obtain a dense Gaussian distribution;

[0121] Obtain the second point cloud map according to the dense Gaussian distribution.

[0122] For example, each Gaussian distribution can be represented as where x represents the position of the point cloud, and μ is the variance of the Gaussian distribution.

[0123] Optionally, the 3D Gaussian Splatting technique can be used to represent the point cloud in the first point cloud map as 3D Gaussian spheres. A 3D Gaussian sphere can be regarded as an element with attributes such as a specific position, color, opacity, size, and shape. For example, the position of a 3D Gaussian sphere can be determined by the position of its center point, usually represented by the three coordinate values (X, Y, Z) of the center point. The color of a 3D Gaussian sphere can be represented by spherical harmonics. The opacity of a 3D Gaussian sphere can be represented by a scalar value, usually ranging from 0 to 1, where 0 represents completely transparent and 1 represents completely opaque. It can be understood that 3D Gaussian spheres with high opacity can clearly define the boundaries and shapes of objects. In application scenarios such as 3D model generation, 3D Gaussian spheres with high opacity can clearly represent a part of a solid object and effectively contribute to the final shape construction. For example, when constructing a complex building model, opaque Gaussian spheres can accurately outline the shapes and positions of various parts of the building, such as walls and columns. The shape of a 3D Gaussian sphere can be spherical or ellipsoidal. The above parameters jointly define the shape, position, color, and transparency of the 3D Gaussian sphere in space, enabling the 3D Gaussian sphere to flexibly represent various geometric and lighting details in the scene.

[0124] In the above way, each 3D point is no longer a simple coordinate but is represented by a Gaussian distribution. In this way, each point cloud is given a distribution area, which can not only enhance the continuity of the point cloud but also effectively reduce errors caused by perspective differences, lighting changes, and noise.

[0125] In one implementation, the densification process may include:

[0126] Performing multi-scale depth estimation based on the 3D Gaussian distribution to obtain depth maps corresponding to each scale;

[0127] Performing detail correction on the depth maps corresponding to each scale to obtain corrected depth maps;

[0128] Obtaining the dense Gaussian distribution based on the corrected depth map.

[0129] Optionally, a dense Gaussian distribution can be obtained by using a Multi-Scale Geometric Consistency Guided Multi-View Stereo (ACMM). ACMM includes a multi-scale geometric consistency guiding mechanism and a detail restorer. Among them, the multi-scale geometric consistency guiding mechanism analyzes and processes images at different scales, obtains reliable depth estimates for low-texture regions at a coarser scale, and ensures that these estimates can be propagated to finer scales, enabling the algorithm to adaptively adjust the matching strategy and capture more complex and detailed scene details. The detail restorer can further improve the accuracy of depth estimation.

[0130] Specifically, the working principle of the multi-scale consistency guiding mechanism is as follows: at the coarse scale, depth estimates for low-texture regions are generated; then, the depth estimates for low-texture regions are upsampled layer by layer to obtain depth estimates at different finer scales in sequence, so as to propagate the depth estimates at the coarse scale to finer scales.

[0131] However, some errors may be introduced during this process, and details of depth information may be lost. To solve this problem, ACMM uses a detail restorer, and its working principle is as follows: the detail restorer calculates a difference map based on the illumination consistency between adjacent scales, that is, by analyzing and comparing the illumination changes between adjacent-scale images, it determines the illumination inconsistency differences caused by factors such as upsampling and scale changes. These differences can reflect the possibly lost or inaccurate detail information. Based on the calculated illumination consistency difference map, the detail restorer corrects the depth map at the current scale. It uses the information in the difference map to adjust the detail part of the depth map, compensates for the details lost during the upsampling process, corrects possible incorrect estimates, so that the depth map, while maintaining the reliable depth estimates propagated from the coarse scale, restores and enhances the detail information of the scene, making the depth map more accurately reflect the true geometric details of the scene.

[0132] In the above way, the obtained dense Gaussian distribution includes rich details, and these details can more accurately reflect the structure of the environment, providing a reliable data basis for the subsequent 3D map.

[0133] In one implementation, the step of generating the second point cloud map may include:

[0134] Project the dense Gaussian distribution into a two-dimensional image;

[0135] Calculate the difference value between the two-dimensional image and the first captured image;

[0136] Adjust the parameters of the dense Gaussian distribution according to the difference value to obtain an optimized Gaussian distribution;

[0137] The second point cloud map is obtained according to the optimized Gaussian distribution.

[0138] Optionally, the difference value can be calculated by the following formula:

[0139] Reprojection Error=||x ij -π(R i X j +T i )||;

[0140] where x ij represents the coordinates of the j-th 3D point (such as a 3D Gaussian sphere can be regarded as a 3D point) on the i-th 2D image, R i is the rotation matrix of the i-th 2D image, T i is the translation vector of the i-th 2D image, and X j is the 3D coordinates of the j-th 3D point.

[0141] Optionally, the Gaussian distribution can be optimized by minimizing the difference value, so that the Gaussian distribution is consistent with the captured image.

[0142] In the above implementation, the parameters of the Gaussian distribution are adjusted in the reverse direction of the difference value between the 2D image obtained by the dense Gaussian distribution and the first captured image, so as to optimize the dense Gaussian distribution and further improve the accuracy of 3D mapping.

[0143] Among them, obtaining the second point cloud map according to the optimized Gaussian distribution can mean regarding the 3D Gaussian sphere in the optimized Gaussian distribution as a point, and its position is the center point coordinates of the 3D Gaussian sphere, so as to convert the Gaussian distribution into a point cloud map.

[0144] It can be understood that the second point cloud map can also be optimized after it is generated.

[0145] Optionally, the method for adjusting the parameters of the dense Gaussian distribution can include:

[0146] Adjusting the attributes of the elements in the dense Gaussian distribution according to the difference value;

[0147] Adjusting the density of the elements in the dense Gaussian distribution according to the difference value;

[0148] Obtaining the optimized Gaussian distribution according to the attributes and / or density of the elements in the adjusted dense Gaussian distribution.

[0149] S204. Construct a 3D image of the target object according to the second point cloud map.

[0150] Optionally, the point cloud map can be converted into a three-dimensional model by the Marching cube method.

[0151] Optionally, the 3D Gaussian sphere can also be converted into a three-dimensional model by methods such as Sugar and Gaussian Opacity Fields. It is also possible to convert the 3D Gaussian sphere into a point cloud and then use the Marching cube method to convert it into a three-dimensional model. In the embodiments of the present application, the method for generating a three-dimensional model from the 3D Gaussian sphere is not specifically limited.

[0152] In the embodiments of the present application, two-dimensional images can be processed in real time to meet the real-time requirements of three-dimensional mapping. Secondly, dense three-dimensional point clouds are generated from the two-dimensional images, greatly increasing the number of map points and improving the map details, thereby facilitating the improvement of the positioning accuracy of three-dimensional mapping. In addition, the Gaussian distribution is used to represent the point cloud, so that each three-dimensional point is no longer a simple coordinate, but is represented by a Gaussian distribution; in this way, each point cloud is given a distribution area, which can not only enhance the continuity of the point cloud, but also effectively reduce the errors caused by perspective differences, lighting changes and noise.

[0153] It should be understood that the magnitudes of the sequence numbers of the steps in the above embodiments do not mean the order of execution, and the execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

[0154] Corresponding to the three-dimensional mapping method described in the above embodiments, Figure 4 is a structural block diagram of a three-dimensional mapping device provided by the embodiments of the present application. For the convenience of description, only the parts related to the embodiments of the present application are shown.

[0155] Referring to Figure 4 , the device 4 includes:

[0156] An acquisition unit 41, configured to acquire a first captured image of a target object; wherein, the first captured image is a two-dimensional image.

[0157] A generation unit 42, configured to generate a first point cloud map according to the first captured image.

[0158] A processing unit 43, configured to perform densification processing on the first point cloud map to obtain a second point cloud map; wherein, the point cloud density of the second point cloud map is greater than the point cloud density of the first point cloud map.

[0159] A construction unit 44, configured to construct a three-dimensional image of the target object according to the second point cloud map.

[0160] Optionally, the processing unit 43 is further configured to:

[0161] Represent the point cloud in the first point cloud map as a three-dimensional Gaussian distribution;

[0162] Perform densification processing on the three-dimensional Gaussian distribution to obtain a dense Gaussian distribution;

[0163] Obtain the second point cloud map according to the dense Gaussian distribution.

[0164] Optionally, the processing unit 43 is further configured to:

[0165] Perform multi-scale depth estimation according to the three-dimensional Gaussian distribution to obtain depth maps corresponding to respective multiple scales;

[0166] Perform detail correction on the depth maps corresponding to respective multiple scales to obtain a corrected depth map;

[0167] Obtain the dense Gaussian distribution according to the corrected depth map.

[0168] Optionally, the processing unit 43 is further configured to:

[0169] Project the dense Gaussian distribution into a two-dimensional image;

[0170] Calculate the difference value between the two-dimensional image and the first captured image;

[0171] Adjust the parameters of the dense Gaussian distribution according to the difference value to obtain an optimized Gaussian distribution;

[0172] Obtain the second point cloud map according to the optimized Gaussian distribution.

[0173] Optionally, the processing unit 43 is further configured to:

[0174] Adjust the attributes of the elements in the dense Gaussian distribution according to the difference value;

[0175] Adjust the density of the elements in the dense Gaussian distribution according to the difference value;

[0176] Obtain the optimized Gaussian distribution according to the attributes and density of the elements in the adjusted dense Gaussian distribution.

[0177] Optionally, the generating unit 42 is further configured to:

[0178] Estimate the camera pose according to the first captured image and the second captured image; wherein, the second captured image is the previous captured image of the first captured image, and the second captured image is a two-dimensional image;

[0179] Generate the first point cloud map according to the camera pose.

[0180] Optionally, the generating unit 42 is further configured to:

[0181] Extract the first feature points of the first captured image and the second feature points of the second captured image;

[0182] Perform feature matching on the first feature points and the second feature points to obtain multiple sets of matching points;

[0183] Estimate the camera pose based on multiple sets of the matching points.

[0184] Optionally, the generating unit 42 is further configured to:

[0185] Perform fitting of a data model based on multiple sets of the matching points to obtain a fitting model;

[0186] Estimate the camera pose based on the matching points that satisfy the fitting model.

[0187] It should be noted that for the information interaction, execution process, etc. between the above-mentioned device / units, since they are based on the same concept as the method embodiments of the present application, for their specific functions and the technical effects brought, reference can be specifically made to the method embodiment part, and details will not be elaborated here.

[0188] In addition, Figure 4 The shown 3D mapping device may be a software unit, a hardware unit, or a unit combining software and hardware built into an existing terminal device, may also be integrated into the terminal device as an independent attachment, or may exist as an independent terminal device.

[0189] Those skilled in the art can clearly understand that for the sake of convenience and conciseness of description, only the above division of each functional unit and module is used as an example for illustration. In actual applications, the above functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiments can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction and do not limit the protection scope of the present application. The specific working processes of the units and modules in the above system can refer to the corresponding processes in the foregoing method embodiments, and details will not be elaborated here.

[0190] Figure 5 It is a schematic structural diagram of a terminal device provided by an embodiment of the present application. As Figure 5 shown, the terminal device 5 in this embodiment includes: at least one processor 50( Figure 5Only one processor, a memory 51, and a computer program 52 stored in the memory 51 and executable on the at least one processor 50 are shown. When the processor 50 executes the computer program 52, the steps in any of the above-described three-dimensional mapping method embodiments are implemented.

[0191] The terminal device may be a computing device such as a desktop computer, a notebook, a handheld computer, and a cloud server. The terminal device may include, but is not limited to, a processor and a memory. Those skilled in the art can understand that Figure 5 merely examples of the terminal device 5, which do not constitute a limitation on the terminal device 5, may include more or fewer components than shown in the figure, or combine certain components, or different components. For example, it may also include input / output devices, network access devices, etc.

[0192] The processor 50 may be a central processing unit (CPU), and the processor 50 may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0193] In some embodiments, the memory 51 may be an internal storage unit of the terminal device 5, such as the hard disk or memory of the terminal device 5. In other embodiments, the memory 51 may also be an external storage device of the terminal device 5, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc., equipped on the terminal device 5. Further, the memory 51 may also include both the internal storage unit and the external storage device of the terminal device 5. The memory 51 is used to store an operating system, application programs, a boot loader, data, and other programs, such as the program code of the computer program, etc. The memory 51 may also be used to temporarily store data that has been output or will be output.

[0194] An embodiment of the present application also provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0195] An embodiment of the present application provides a computer program product, and when the computer program product runs on a terminal device, the terminal device can implement the steps in the above-mentioned method embodiments when executed.

[0196] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, to implement all or part of the processes in the above-mentioned method embodiments of the present application, a computer program can be used to instruct relevant hardware to complete. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file or some intermediate form, etc. The computer-readable medium can at least include: any entity or device capable of carrying the computer program code to the device / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal, and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk, or an optical disc, etc. In some jurisdictions, according to legislation and patent practice, the computer-readable medium may not be an electrical carrier signal and a telecommunication signal.

[0197] In the above embodiments, the descriptions of the respective embodiments have their own emphases. For parts not detailed or recorded in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0198] Those of ordinary skill in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0199] In the embodiments provided in the present application, it should be understood that the disclosed device / terminal device and method can be implemented in other ways. For example, the device / terminal device embodiments described above are merely illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some interfaces. The indirect coupling or communication connection of the device or unit can be in electrical, mechanical or other forms.

[0200] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0201] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A three-dimensional mapping method, characterized in that, Including: Obtain a first captured image of a target object; wherein, the first captured image is a two-dimensional image; Generate a first point cloud map according to the first captured image; Perform densification processing on the first point cloud map to obtain a second point cloud map; wherein, the point cloud density of the second point cloud map is greater than that of the first point cloud map; Construct a three-dimensional image of the target object according to the second point cloud map.

2. The three-dimensional mapping method according to claim 1, characterized in that, The performing densification processing on the first point cloud map to obtain a second point cloud map includes: Represent the point cloud in the first point cloud map as a three-dimensional Gaussian distribution; Perform densification processing on the three-dimensional Gaussian distribution to obtain a dense Gaussian distribution; Obtain the second point cloud map according to the dense Gaussian distribution.

3. The three-dimensional mapping method according to claim 2, wherein The performing densification processing on the three-dimensional Gaussian distribution to obtain a dense Gaussian distribution includes: Perform multi-scale depth estimation according to the three-dimensional Gaussian distribution to obtain depth maps corresponding to respective multiple scales; Perform detail correction on the depth maps corresponding to respective multiple scales to obtain corrected depth maps; Obtain the dense Gaussian distribution according to the corrected depth maps.

4. The three-dimensional mapping method according to claim 3, wherein, The obtaining the second point cloud map according to the dense Gaussian distribution includes: Project the dense Gaussian distribution into a two-dimensional image; Calculate the difference value between the two-dimensional image and the first captured image; Adjust the parameters of the dense Gaussian distribution according to the difference value to obtain an optimized Gaussian distribution; Obtain the second point cloud map according to the optimized Gaussian distribution.

5. The three-dimensional mapping method according to claim 4, wherein The adjusting the parameters of the dense Gaussian distribution according to the difference value to obtain an optimized Gaussian distribution includes: Adjust the attributes of the elements in the dense Gaussian distribution according to the difference value; Adjust the density of the elements in the dense Gaussian distribution according to the difference value; Obtain the optimized Gaussian distribution according to the attributes and density of the elements in the adjusted dense Gaussian distribution.

6. The three-dimensional mapping method according to any one of claims 1 to 5, characterized in that The generating a first point cloud map according to the first captured image includes: Estimate the camera pose according to the first captured image and a second captured image; wherein, the second captured image is the previous captured image of the first captured image, and the second captured image is a two-dimensional image; Generate the first point cloud map according to the camera pose.

7. The three-dimensional mapping method according to claim 6, wherein The estimating the camera pose according to the first captured image includes: Extract first feature points of the first captured image and second feature points of the second captured image; Perform feature matching on the first feature points and the second feature points to obtain multiple sets of matching points; Estimate the camera pose according to multiple sets of the matching points.

8. The three-dimensional mapping method according to claim 7, wherein The estimating the camera pose according to multiple sets of the matching points includes: Perform fitting of a data model according to multiple sets of the matching points to obtain a fitting model; Estimate the camera pose according to the matching points that satisfy the fitting model.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, the method described in any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, the method described in any one of claims 1 to 8 is implemented.

Images