An automatic modeling method, device, computer device, and storage medium

Abstract

The present invention discloses an automatic modeling method, device, computer device and storage medium. The method includes: extracting feature points from the photos in the image set through a first preset algorithm or a second preset algorithm to obtain two different sets of feature points, where the first preset algorithm and the second preset algorithm are two different feature point extraction algorithms, and the image set includes multiple photos taken of the same scene under different conditions; performing feature point matching on the two different sets of feature points to obtain a plurality of matching feature points; performing camera calibration based on the matching feature points and generating a point cloud model; transforming and optimizing the point cloud model to obtain a simplified model. Using this method, the modeling speed can be effectively improved.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of modeling technology, and in particular, to an automatic modeling method, apparatus, computer device, and storage medium. Background Art

[0002] Modeling technology is indispensable in fields such as reverse engineering, smart cities, digital twins, game scenes, and assisted medical care. Automatic construction of three-dimensional models is a key technology to support future digital twins. Rapidly reconstructing the real world, identifying and segmenting scene objects, and creating a scene topology structure not only saves modeling costs but also enables real-time dynamic changes, simulation and simulation calculations, and predictions of twin scenes.

[0003] Although the three-dimensional modeling methods in the prior art can construct high-precision three-dimensional scenes, the reconstruction speed is slow, and a large amount of time is required for scene reconstruction. Therefore, how to improve the modeling speed is an urgent technical problem to be solved currently. Summary of the Invention

[0004] Embodiments of the present invention provide an automatic modeling method, apparatus, computer device, and storage medium, which can effectively improve the modeling speed.

[0005] In a first aspect, an embodiment of the present invention provides an automatic modeling method, including:

[0006] Extracting feature points from the photos in the image set through a first preset algorithm or a second preset algorithm to obtain two different sets of feature points, where the first preset algorithm and the second preset algorithm are two different feature point extraction algorithms, and the image set includes multiple photos taken of the same scene under different conditions;

[0007] Matching the two different sets of feature points to obtain multiple matching feature points;

[0008] Performing camera calibration based on the matching feature points and generating a point cloud model;

[0009] Converting and optimizing the point cloud model to obtain a simplified model.

[0010] In a second aspect, an embodiment of the present invention further provides an automatic modeling apparatus, including:

[0011] An extraction module, configured to extract feature points from the photos in the image set through a first preset algorithm or a second preset algorithm to obtain two different sets of feature points, where the first preset algorithm and the second preset algorithm are two different feature point extraction algorithms, and the image set includes multiple photos taken of the same scene under different conditions;

[0012] A matching module, configured to perform feature point matching on the two different sets of feature points to obtain a plurality of matching feature points;

[0013] A generation module, configured to perform camera calibration based on the matching feature points and generate a point cloud model;

[0014] A transformation and optimization module, configured to transform and optimize the point cloud model to obtain a simplified model.

[0015] In a third aspect, an embodiment of the present invention further provides a computer device, including:

[0016] One or more processors;

[0017] A storage device, configured to store one or more programs;

[0018] The one or more programs are executed by the one or more processors, so that the one or more processors are configured to implement the automatic modeling method described in any embodiment of the present invention.

[0019] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, it implements the automatic modeling method provided in any embodiment of the present invention.

[0020] An embodiment of the present invention provides an automatic modeling method, device, computer device and storage medium. First, photos in an image set are subjected to feature point extraction through a first preset algorithm or a second preset algorithm to obtain two different sets of feature points; then the two different sets of feature points are subjected to feature point matching to obtain a plurality of matching feature points; then camera calibration is performed based on the matching feature points and a point cloud model is generated; finally, the point cloud model is transformed and optimized to obtain a simplified model. By using the above technical solution, the modeling speed can be effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a schematic flowchart of an automatic modeling method provided by Embodiment 1 of the present invention;

[0022] Figure 2 It is a schematic diagram of feature point matching in an automatic modeling method provided by Embodiment 1 of the present invention;

[0023] Figure 3 It is an exemplary flowchart of an automatic modeling method provided by Embodiment 2 of the present invention;

[0024] Figure 4 It is a schematic structural diagram of an automatic modeling device provided by Embodiment 3 of the present invention;

[0025] Figure 5Schematic diagram of a computer device provided in the fourth embodiment of the present invention. Detailed implementation manners

[0026] The embodiments of the present invention will be described in more detail with reference to the accompanying drawings. Although some embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. On the contrary, these embodiments are provided to more thoroughly and completely understand the present invention. It should be understood that the drawings and embodiments of the present invention are only for exemplary purposes and are not used to limit the protection scope of the present invention.

[0027] It should be understood that the steps recited in the method embodiments of the present invention can be executed in a different order and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this regard.

[0028] The term "including" and its variations used herein are open-ended, that is, "including but not limited to". The term "based on" is "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0029] It should be noted that the concepts such as "first" and "second" mentioned in the present invention are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0030] It should be noted that the modifications of "one" and "plural" mentioned in the present invention are illustrative rather than restrictive. Those skilled in the art should understand that unless otherwise clearly stated in the context, it should be understood as "one or more".

[0031] The names of the messages or information exchanged between multiple devices in the embodiments of the present invention are only for illustrative purposes and are not used to limit the scope of these messages or information.

[0032] Embodiment 1

[0033] Figure 1 Schematic flowchart of an automatic modeling method provided in Embodiment 1 of the present invention. This method is applicable to the situation of three-dimensional scene reconstruction. This method can be executed by an automatic modeling device, where the device can be implemented by software and / or hardware and is generally integrated on a computer device.

[0034] As Figure 1As shown in the figure, an automatic modeling method provided in Embodiment 1 of the present invention includes the following steps:

[0035] S110. Extract feature points from the photos in the image set through the first preset algorithm or the second preset algorithm to obtain two different sets of feature points.

[0036] Among them, the first preset algorithm and the second preset algorithm are two different feature point extraction algorithms, and the image set includes multiple photos taken of the same scene under different conditions.

[0037] Among them, the photos included in the image set are taken by the same camera of a target object under different conditions. The different conditions include different times, different angles, different resolutions, different illuminations, and different camera poses. The camera in this embodiment can be any ordinary camera, and the camera uses the same lens parameters for shooting. The photos in the image set need to have a certain relevance. Preferably, the overlap of two photos is about 30% at best.

[0038] Among them, the first preset algorithm and the second preset algorithm are pre-set algorithms capable of extracting feature points. Exemplarily, the second preset algorithm can be the Scale-invariant feature transform (SIFT) algorithm, and the first preset algorithm can be the Speeded Up Robust Features (SURF) algorithm.

[0039] In this embodiment, the SIFT algorithm extracts local features of the image, which is invariant to rotation, scale scaling, and brightness changes, and also maintains a certain degree of stability to view changes, affine transformations, and noises. The characteristics of the SIFT algorithm are good uniqueness, rich information content, and applicability to a large number of features. Quantity, even a few objects can generate a large number of SIFT feature vectors. Scalability, which can be easily combined with other forms of feature vectors.

[0040] Specifically, the SIFT algorithm may include the following steps:

[0041] 1. Extract key points

[0042] Key points are some very prominent points that will not disappear due to factors such as illumination, scale, and rotation, such as corner points, edge points, bright points in dark areas, and dark points in bright areas. This step is to search for image positions on all scale spaces. Identify potential interest points with scale and rotation invariance through the Gaussian differential function.

[0043] 2. Locate key points

[0044] At each candidate position, a well-fitted model is used to determine the position and scale. The selection of key points is based on their stability. Then, one or more directions are assigned to each key point position based on the local gradient direction of the image. All subsequent operations on the image data are transformed relative to the direction, scale, and position of the key points, thus providing invariance to these transformations.

[0045] 3. Determine the direction

[0046] Based on the local gradient direction of the image, one or more directions are assigned to each key point position. All subsequent operations on the image data are transformed relative to the direction, scale, and position of the key points, thus providing invariance to these transformations.

[0047] To find the extreme points through scale invariance, it is necessary to use the local features of the image to assign a reference direction to each key point so that the descriptor is invariant to image rotation. For the key points detected in the DOG pyramid, the gradient and direction distribution features of the pixels within the 3σ neighborhood window of the Gaussian pyramid image where they are located are collected. The modulus and direction of the gradient are as follows:

[0048]

[0049] θ(x, y) = tan -1 ((L(x, y + 1) - L(x, y - 1)) / (L(x + 1, y) - L(x - 1, y)))

[0050] Then, the gradient histogram statistics method is used to determine the key point direction by statistically analyzing the image pixel points within a certain area with the key point as the origin. After calculating the gradient of the key points, the histogram is used to statistically analyze the gradient and direction of the pixels in the neighborhood. The gradient histogram divides the direction range from 0 to 360 degrees into 36 bins, with each bin being 10 degrees. The peak direction of the histogram represents the main direction of the key point, and the peak of the direction histogram represents the direction of the neighborhood gradient at this feature point. The maximum value in the histogram is used as the main direction of this key point. To enhance the robustness of the matching, only the directions with a peak greater than 80% of the main direction peak are retained as the secondary directions of this key point.

[0051] 3. Key point description

[0052] In the neighborhood around each key point, the local gradient of the image is measured at the selected scale. These gradients are transformed into a representation that allows for relatively large local shape deformations and illumination changes.

[0053] For each key point, there are three pieces of information: position, scale, and orientation. A descriptor is established for each key point, and a group of vectors is used to describe this key point so that it remains unchanged under various changes, such as illumination changes, perspective changes, and so on. The descriptor not only includes the key point but also the pixel points around the key point that contribute to it, and the descriptor should have high distinctiveness to improve the probability of correct matching of feature points.

[0054] The feature points obtained by Difference of Gaussians (DoG) have scale invariance. So, even though the scale changes, the feature points can still be detected. Rotational invariance. That is, the definition of the principal direction as follows. Under the principal direction, there is rotational invariance. The principal direction is such that for this direction, the coordinates of other points in the image remain unchanged after operations such as rotation of the coordinates.

[0055] Specifically, the SURF algorithm can include the following steps:

[0056] 1. Construct the Hessian matrix

[0057] For each pixel point, a Hessian matrix can be calculated. The purpose of constructing the Hessian matrix is to generate stable edge points in the image for feature extraction. When the discriminant of the Hessian matrix reaches a local maximum, it is determined that the current point is a point that is brighter or darker than other points in the surrounding neighborhood, thereby locating the position of the key point.

[0058] 2. Construct the scale space

[0059] Similar to the SIFT algorithm, the scale space of the SUFT algorithm is also composed of O groups and L layers. The difference is that in Sift, the size of the next group of images is half of that of the previous group, and the image sizes within the same group are the same, but the Gaussian blur coefficients used gradually increase; while in Surf, the image sizes between different groups are the same, but the template sizes of the box filters used between different groups gradually increase, and the filters within the same group and different layers use the same size filter, but the blur coefficients of the filters gradually increase.

[0060] 3. Feature point localization

[0061] Each pixel point processed by the Hessian matrix is compared with 26 points in the two-dimensional image space and the scale space neighborhood to preliminarily locate the key points. Then, by filtering out key points with weak energy and mislocalized key points, the final stable feature points are selected.

[0062] 4. Feature point principal direction assignment

[0063] The harr wavelet features within the circular neighborhood of the statistical feature points are adopted. Within the circular neighborhood of the feature points, the sum of the horizontal and vertical harr wavelet features of all points within a 60-degree sector is statistically calculated. Then, the sector rotates at a certain interval and the harr wavelet feature values within this region are statistically calculated again. Finally, the direction of the sector with the largest value is taken as the main direction of the feature point.

[0064] 5. Generate feature point descriptors

[0065] A 4×4 rectangular region block is taken around the feature point, but the direction of the obtained rectangular region is along the main direction of the feature point. For each sub-region, the haar wavelet features in the horizontal and vertical directions of 25 pixels are statistically calculated. Here, the horizontal and vertical directions are relative to the main direction. The haar wavelet feature has 4 directions, namely the sum of the horizontal direction value, the vertical direction value, the absolute value of the horizontal direction, and the absolute value of the vertical direction. These 4 values are used as the feature vectors of each sub-block region. Therefore, there are a total of 64-dimensional vectors as the descriptors of the Surf feature.

[0066] Furthermore, the two different sets of feature points include a first set of feature points and a second set of feature points; wherein, the feature points extracted from the image set through a first preset algorithm are combined into the first set of feature points, and the feature points extracted from the image set through a second preset algorithm are combined into the second set of feature points.

[0067] In this embodiment, only one feature point extraction algorithm can be used to extract feature points from a single photo.

[0068] Among them, a single photo can obtain a 64-dimensional feature vector through feature point extraction by the SURF algorithm, and a single photo can obtain a 128-dimensional feature vector through feature point extraction by the SIFT algorithm.

[0069] Exemplarily, the feature points extracted from the photos in the image set through the SURF algorithm are combined into the first set of feature points, and the feature points extracted from the photos in the image set through the SIFT algorithm are combined into the second set of feature points.

[0070] Furthermore, the method of extracting feature points from the photos in the image set through the first preset algorithm or the second preset algorithm to obtain two different sets of feature points includes: for each photo in the image set, feature points are extracted through the first preset algorithm; if the number of feature points extracted through the first preset algorithm is less than the preset value, then feature points are extracted through the second preset algorithm.

[0071] Exemplarily, for each photo in the image set, first, feature points are extracted through the SURF algorithm. When it comes to the step of locating feature points, if it is judged when executing the feature point location step whether the feature points located in this step can generate sufficient key points to support subsequent feature point matching, and if not enough key points can be generated, then the SIFT algorithm is used to extract feature points for this photo.

[0072] Furthermore, if the number of feature points obtained by extracting feature points from the photos in the image set through the first preset algorithm is less than the preset value, and the feature points obtained after extracting feature points through the second preset algorithm cannot be used for feature point matching, then the photos are removed from the image set.

[0073] Exemplarily, if the number of feature points extracted from a photo through the SURF algorithm is not enough to generate sufficient key points, and the quality of the feature points extracted from this photo through the SURF algorithm does not meet the standard, then this photo is deleted from the image set.

[0074] S120. Perform feature point matching on the two different sets of feature points to obtain multiple matching feature points.

[0075] In this embodiment, the way of performing feature point matching in the first preset algorithm and the second preset algorithm is the same. Specifically, perform feature point matching on the feature points included in the first set of feature points, calculate the Euclidean distance of the key points. The smaller the Euclidean distance, the higher the similarity. When the Euclidean distance is less than the set threshold, it can be determined that the matching is successful. Figure 2 This is a schematic diagram of feature point matching in an automatic modeling method provided by Embodiment 1 of the present invention. Figure 2 It shows the matching situation of two photos with a certain degree of overlap.

[0076] Among them, the specific process of feature point matching can be: for a certain feature point p1 in Photo 1, find the key point p21 closest to p1 and the key point p22 next closest to p1 among the key points of Photo 2, obtain the distance d1 between the key point p21 and the feature point p1 and the distance d2 between the key point p22 and the feature point p1. If d1 / d2 < the threshold, then it is regarded as a correctly matched point pair, and the wrongly matched point pairs are deleted. It should be noted that Photo 1 and Photo 2 are photos in which feature points are extracted using the same preset algorithm. Exemplarily, both Photo 1 and Photo 2 use the SIFT algorithm to extract feature points. Herein, the feature points in Photo 1 and Photo 2 are matchable.

[0077] S130. Perform camera calibration based on the matching feature points and generate a point cloud model.

[0078] In this embodiment, camera calibration needs to be performed based on the matching feature points, and a point cloud model is generated based on the camera calibration.

[0079] Further, the camera calibration based on the matching feature points and generating a point cloud model includes: performing camera calibration based on the matching feature points to obtain a projection matrix; obtaining a unified coordinate system and unifying the matching feature points into the unified coordinate system; generating a point cloud model based on the projection matrix and the matching feature points in the unified coordinate system.

[0080] Among them, camera calibration is the mapping from world coordinates to pixel coordinates. Calibration is to solve this mapping relationship knowing the world coordinates and pixel coordinates of the calibration control points. By establishing this relationship, the world coordinates of a point can be deduced from its pixel coordinates. With the world coordinates, subsequent operations such as measurement can be carried out.

[0081] Specifically, the process of camera calibration is: determining the positional relationship between the matching feature points; calibrating the camera shooting positions corresponding to each photo in the image set according to the positional relationship; determining the position where the camera is located when shooting each photo according to the calibration result. The specific steps and principles of camera calibration will not be elaborated here, and relevant materials can be referred to.

[0082] In this embodiment, the unified coordinate system can be added to the computer manually after camera calibration so that the computer can obtain the unified coordinate system. Since two different algorithms are used for feature point extraction in the feature point extraction stage, the two sets of obtained feature point sets correspond to two coordinate systems respectively. Therefore, the two coordinate systems need to be unified into a unified coordinate system so that each feature point is in the unified coordinate system.

[0083] In this embodiment, a dense point cloud model is constructed based on MVS (Multi-view Stereo). The process of generating the point cloud model includes: generating a white object model based on the projection matrix and the matching feature points in the unified coordinate system; marking the white object model; evaluating the colors of the pixel points included in the marked white object model through photometric consistency constraint evaluation; constructing the internal structure of the concave polyhedron on the marked white object model through visibility constraint; coloring and texturing the marked white object model according to the colors of the pixel points to obtain the point cloud model.

[0084] Specifically, matching feature points are found on multiple photos respectively, and the spatial position of the point is calculated using the projection matrix. Similarly, many spatial positions can be obtained. Based on these spatial positions and depth information, a white film of the object can be obtained, where the object is the object in the photo. Among them, depth information can be obtained through epipolar search. The depth information is the distance between the spatial position of the object and the corresponding point on the photo. Epipolar search can effectively reduce the search range of feature points, improve the efficiency of point cloud generation, and increase the confidence of point positions.

[0085] The way to mark the white film of the object can be: mark the points inside the object as 1, mark the points outside the object as 0, and the points between 0 and 1 are the surface of the object.

[0086] After marking, photometric consistency constraints and visibility constraints can be used to calculate the points on the surface of the object to evaluate the calibration quality.

[0087] Among them, the color of the pixel points on the surface of the object can be obtained through photometric consistency constraint evaluation. Through visibility constraints, occlusion can be removed to construct the internal structure of the concave variant. After coloring and texturing the surface of the object white film according to the obtained color, a point cloud model can be obtained.

[0088] S140. Transform and optimize the point cloud model to obtain a simplified model.

[0089] In this embodiment, the transformation of the point cloud model into a polygon model can be achieved based on the Poisson reconstruction algorithm, and the simplification of the model can be achieved using mesh simplification technology to obtain a simplified model.

[0090] Further, the transformation and optimization of the point cloud model to obtain a simplified model includes: transforming the point cloud model into a colored object model through a preset transformation algorithm; optimizing the colored object model through a preset simplification algorithm to obtain a simplified model.

[0091] Among them, the preset algorithm can be any algorithm that can transform the point cloud model into a polygon model. Exemplarily, the preset algorithm can be the Poisson reconstruction algorithm.

[0092] Specifically, the transformation of the point cloud model into a colored object model through a preset transformation algorithm includes: constructing a colorless object model based on the point cloud model; generating a colored object model based on the colorless object model.

[0093] In this embodiment, when constructing a colorless object model, the point cloud model is divided into multiple parts, each part includes several point clouds, and an octree search index is set for the point cloud so that each sampling point falls on a leaf node with a depth of D, which can accelerate the subsequent solution of the Poisson equation.

[0094] Further, constructing a colorless object model based on the point cloud model includes: converting the point cloud in the point cloud model into a point cloud in a target format; dividing the point cloud in the target format into multiple groups of point clouds; setting an octree search index for each group of point clouds; and constructing a colorless object model according to the octree search index.

[0095] In this embodiment, converting the point cloud in the point cloud model into a point cloud in a target format includes: calculating the normal vector of the original point cloud included in the point cloud model to obtain a normal vector pointing to the inside of the object; obtaining normal information according to the normal vector pointing to the inside of the object; and superimposing the normal information on the point cloud to generate a point cloud in the target format.

[0096] Among them, the point cloud in the target format can be a point cloud in the format of position + color + normal.

[0097] Among them, calculating the normal vector of the original point cloud included in the point cloud model includes: using a normal estimation method to set a normal vector estimation object. The essence of this algorithm is principal component analysis, that is, first set the number of adjacent points and the search radius selected around each point, and use the adjacent points to establish a covariance matrix C.

[0098]

[0099] Among them, the eigenvector corresponding to the minimum eigenvalue of C is the normal vector of this point. Since Poisson reconstruction requires a normal vector pointing to the inside of the object, the vector needs to be reversed to obtain a normal vector pointing to the inside of the object.

[0100] In this embodiment, constructing a colorless object model according to the octree search index includes: setting a function space for multiple groups of point clouds according to the octree search index; creating a vector field based on the function space to obtain a Poisson equation; solving the Poisson equation to obtain an indicator function; and obtaining the object surface according to the indicator function.

[0101] The specific process is as follows:

[0102] Step 1: Set an octree search index for the point cloud.

[0103] Step 2: Set the function space

[0104] Given an object M with a boundary of δM, x M :R 3 →{0,1} is an indicator function that satisfies the value of 1 inside the object and 0 for the rest, and obtaining x M {q0} for each point q0 in the entire domain, and the entire object surface can be obtained.

[0105] Step 3: Create a vector field

[0106] Since x MDue to the discontinuity, the interpolated values between 0 and 1 are meaningless. Therefore, an indirect method is adopted. First, use the smoothing filter function F to smooth x M , which can be proved by the divergence theorem that the gradient field of the smoothed indicator function is equal to the smoothed surface normal vector field:

[0107]

[0108] where * represents the convolution symbol, which is the smoothing operation here, represents the normal vector at the object surface. The gradient direction of the indicator function coincides with the normal vector. The smoothing function F should neither be too large to cause over-smoothing errors nor too small to make the interpolation unreliable when far from the sample points. At the same time, to meet the sparsity in the final solution, Gaussian filtering and truncating the filtering range are required, that is, a third-order box filter, approximated by convolving the box filter three times continuously.

[0109] Due to the discreteness of the sample points, N is not known for every point q near the surface. It needs to be approximated piecewise, and the formula is as follows:

[0110]

[0111] where s is a point in the initial known sample point set S, which contains the s.p position and s.N normal vector information, and ρ s is the surface area near s divided by space. Due to the assumption of uniform distribution of sample points, the constant term |ρ s | can be omitted. Due to the range limitation of F, the smoothed result is a linear combination of sample information within a certain range. That is:

[0112]

[0113] Directly solving for X requires integration, but V is not necessarily an irrotational field and is often non-integrable. Therefore, its least-squares approximation should be solved:

[0114]

[0115] where △ is the Laplace operator and ▽ is the divergence operator. The above equation is the Poisson equation.

[0116] Step 4: Solve the Poisson equation.

[0117] Step 5: Extract the isosurface to obtain the reconstructed surface.

[0118] After solving the Poisson equation, the following formula is obtained:

[0119]

[0120] Among them, ||·|| and <·> represent integrating over all q in the continuous domain, and the indicator function χ can be obtained. Since the Laplacian coefficient matrix is sparse and symmetric, the conjugate gradient method is used for solving.

[0121] Specifically, generating a colored object model based on a colorless object model includes: using a kd tree to map the information of the original point cloud onto the surface of the colorless object and generating a colored object model. Specifically, a search index is established on the original RGB point cloud using a kd tree, and then for each point on the surface of the colorless object, the neighboring points on the original RGB point cloud are searched. Then, the mean values of the RGB channels of these corresponding neighboring points are obtained as the color information of that point on the colorless object.

[0122] Further, optimizing the colored object model through a preset simplification algorithm to obtain a simplified model includes: deleting the useless points on the colored object model through the preset simplification algorithm; constructing a simplified model based on the undeleted points.

[0123] Among them, the preset simplification algorithm can be any algorithm for deleting useless points in the model. Exemplarily, the preset simplification algorithm can include vertex clustering and incremental simplification.

[0124] Specifically, the principle of vertex clustering can be roughly divided into the following four steps:

[0125] 1. Generate clusters.

[0126] 2. Calculate the performance factor.

[0127] 3. Generate a mesh.

[0128] 4. Change the topological structure.

[0129] The specific principle and details of vertex clustering are not elaborated here, and relevant materials of this algorithm can be referred to.

[0130] In this embodiment, incremental simplification can use the method of triangle edge collapse to simplify the mesh, merging two vertices into one vertex. The process of deleting useless points using the method of triangle edge collapse can be: obtaining the vertex with the minimum collapse cost and the collapse target vertex from the colored object model; traversing the triangle face list of the vertex with the minimum collapse cost; if the triangle face list contains the collapse target vertex, deleting the triangle face corresponding to the collapse target vertex; if the triangle face list does not contain the collapse target vertex, replacing the vertex with the minimum collapse cost with the collapse target vertex; recalculating the collapse costs corresponding to all neighbor points of the vertex with the minimum collapse cost and the collapse target vertex; adding the collapse costs corresponding to all neighbor points and the collapse target vertex to the heap and updating the order of the heap until all vertices are processed.

[0131] Among them, the collapse calculation traverses all Vertex vertices in the grid. For each Vertex vertex, it calculates the edge collapse cost of all adjacent vertices, saves the value with the minimum edge collapse cost and the collapse target point, and adds the calculated Vertex vertex to the heap, sorting it in ascending order according to the edge collapse cost. It cyclically pops the vertex with the minimum edge collapse cost from the heap and performs edge collapse processing on the vertex.

[0132] Exemplarily, the specific process of the triangular edge collapse method may include: obtaining the vertex u with the minimum collapse cost and its collapse target vertex v. Traversing the triangular face list of u, if it contains v, delete the triangular face, otherwise replace u with v. Then recalculate the collapse cost and collapse target of all neighbor points of u and update the order of the heap until all vertices are processed.

[0133] An automatic modeling method provided in Embodiment 1 of the present invention first extracts feature points from the photos in the image set through a first preset algorithm or a second preset algorithm to obtain two different sets of feature points; secondly, performs feature point matching on the two different sets of feature points to obtain a plurality of matching feature points; then performs camera calibration based on the matching feature points and generates a point cloud model; finally, converts and optimizes the point cloud model to obtain a simplified model. Using the above method, the modeling speed can be effectively improved.

[0134] Embodiment 2

[0135] Figure 3 It is an example flowchart of an automatic modeling method provided in Embodiment 2 of the present invention. As Figure 3 shown, an automatic modeling method provided in Embodiment 2 of the present invention includes the following processes:

[0136] Extract feature points from the photos in the image set to obtain a feature set, that is, extract feature points from the photos in the image set through a first preset algorithm or a second preset algorithm to obtain two different sets of feature points; perform feature matching on the image features, that is, perform feature point matching on the two different sets of feature points to obtain a plurality of matching feature points; estimate camera parameters from the matching results, that is, perform camera calibration based on the matching feature points; then use the camera parameters to perform scene reconstruction and output a point cloud model, that is, generate a point cloud model based on the projection matrix and the matching feature points in the unified coordinate system; color the point cloud model to obtain a color model, that is, convert the point cloud model into a colored object model through a preset conversion algorithm; perform clipping and accuracy adjustment on the color model to obtain a vertex-index model, that is, optimize the colored object model through a preset simplification algorithm to obtain a simplified model.

[0137] An automatic modeling method provided in the second embodiment of the present invention has low requirements for the photo quality in the image set. Only photos taken by an ordinary camera are needed, and there are no excessive requirements for ambient light and shadows. By using an improved algorithm, the reconstruction speed is fast, the requirement for the overlap of image materials is small, and it only takes a few seconds to complete the indoor reconstruction of hundreds of square meters. The constructed model and texture have high precision, and different precision levels can be created according to user needs.

[0138] Embodiment 3

[0139] Figure 4 FIG. is a schematic structural diagram of an automatic modeling device provided in Embodiment 3 of the present invention. This device is applicable to the situation of three-dimensional scene reconstruction. Among them, this device can be implemented by software and / or hardware and is generally integrated on a computer device.

[0140] As Figure 4 shown, this device includes: an extraction module 110, a matching module 120, a generation module 130, and a transformation and optimization module.

[0141] The extraction module 110 is used to extract feature points from the photos in the image set through a first preset algorithm or a second preset algorithm, obtaining two different sets of feature points. The first preset algorithm and the second preset algorithm are two different feature point extraction algorithms. The image set includes multiple photos taken of the same scene under different conditions;

[0142] The matching module 120 is used to perform feature point matching on the two different sets of feature points to obtain multiple matching feature points;

[0143] The generation module 130 is used to perform camera calibration based on the matching feature points and generate a point cloud model;

[0144] The transformation and optimization module 140 is used to transform and optimize the point cloud model to obtain a simplified model.

[0145] In this embodiment, the automatic modeling device first extracts feature points from the photos in the image set through the extraction module 110 by using a first preset algorithm or a second preset algorithm, obtaining two different sets of feature points. The first preset algorithm and the second preset algorithm are two different feature point extraction algorithms. The image set includes multiple photos taken of the same scene under different conditions; secondly, the matching module 120 performs feature point matching on the two different sets of feature points to obtain multiple matching feature points; then, the generation module 130 performs camera calibration based on the matching feature points and generates a point cloud model; finally, the transformation and optimization module 140 transforms and optimizes the point cloud model to obtain a simplified model.

[0146] This embodiment provides an automatic modeling device, which can effectively improve the modeling speed.

[0147] Furthermore, the two sets of different feature point sets include a first set of feature point sets and a second set of feature point sets;

[0148] Among them, the feature points extracted from the image set through a first preset algorithm are combined into a first set of feature point sets, and the feature points extracted from the image set through a second preset algorithm are combined into a second set of feature point sets.

[0149] Based on the above optimization, the extraction module 110 is specifically configured to: for each photo in the image set, extract feature points through a first preset algorithm; if the number of feature points extracted through the first preset algorithm is less than a preset value, extract feature points through a second preset algorithm.

[0150] Furthermore, if the number of feature points extracted from the photo in the image set through the first preset algorithm is less than a preset value, and the feature points obtained after extracting feature points through the second preset algorithm cannot be used for feature point matching, then the photo is removed from the image set. Furthermore, the generation module 130 is specifically configured to: perform camera calibration based on the matching feature points to obtain a projection matrix; obtain a unified coordinate system, and unify the matching feature points into the unified coordinate system; generate a point cloud model based on the projection matrix and the matching feature points in the unified coordinate system.

[0151] Furthermore, the transformation optimization module 140 is specifically configured to: transform the point cloud model into a colored object model through a preset transformation algorithm; optimize the colored object model through a preset simplification algorithm to obtain a simplified model.

[0152] Furthermore, the transformation of the point cloud model into a colored object model through a preset transformation algorithm includes: constructing a colorless object model based on the point cloud model; generating a colored object model based on the colorless object model.

[0153] Furthermore, the construction of the colorless object model based on the point cloud model includes: transforming the point cloud in the point cloud model into a point cloud in a target format; dividing the point cloud in the target format into multiple groups of point clouds; setting an octree search index for each group of point clouds; constructing a colorless object model according to the octree search index.

[0154] Furthermore, the optimization of the colored object model through a preset simplification algorithm to obtain a simplified model includes: deleting the useless points on the colored object model through a preset simplification algorithm; constructing a simplified model based on the undeleted points.

[0155] The above automatic modeling device can execute the automatic modeling method provided by any embodiment of the present invention, and has corresponding functional modules and beneficial effects for executing the method.

[0156] Example 4

[0157] Figure 5 The following is a schematic structural diagram of a computer device provided by Example 4 of the present invention. As Figure 5 shown, the computer device provided by Example 4 of the present invention includes: one or more processors 41 and a storage device 42; the processor 41 in the computer device can be one or more, Figure 5 taking one processor 41 as an example; the storage device 42 is used to store one or more programs; the one or more programs are executed by the one or more processors 41, so that the one or more processors 41 implement the automatic modeling method described in any one of the embodiments of the present invention.

[0158] The computer device may further include: an input device 43 and an output device 44.

[0159] The processor 41, the storage device 42, the input device 43 and the output device 44 in the computer device can be connected by a bus or other means, Figure 5 taking the connection by bus as an example.

[0160] The storage device 42 in the computer device, as a computer-readable storage medium, can be used to store one or more programs, and the programs can be software programs, computer-executable programs and modules, such as the program instructions / modules corresponding to the automatic modeling method provided in Embodiment 1 or 2 of the present invention (for example, the modules in the automatic modeling device shown in the appendix Figure 4 , including: an extraction module 110, a matching module 120, a generation module 130, and a transformation and optimization module 140). The processor 41 executes various functional applications and data processing of the computer device by running the software programs, instructions and modules stored in the storage device 42, that is, implements the automatic modeling method in the above method embodiments.

[0161] The storage device 42 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system and application programs required for at least one function; the data storage area can store data created according to the use of the computer device, etc. In addition, the storage device 42 may include a high-speed random access memory, and may further include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state storage devices. In some instances, the storage device 42 may further include a memory remotely set relative to the processor 41, and these remote memories can be connected to the device through a network. Examples of the above network include but are not limited to the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0162] The input device 43 can be used to receive input digital or character information and generate key signal inputs related to the user settings and function controls of the computer device. The output device 44 can include display devices such as a display screen.

[0163] Moreover, when one or more programs included in the above computer device are executed by the one or more processors 41, the programs perform the following operations:

[0164] Extract feature points from the photos in the image set through a first preset algorithm or a second preset algorithm to obtain two different sets of feature points. The first preset algorithm and the second preset algorithm are two different feature point extraction algorithms. The image set includes multiple photos taken of the same scene under different conditions;

[0165] Match the two different sets of feature points to obtain multiple matching feature points;

[0166] Perform camera calibration based on the matching feature points and generate a point cloud model;

[0167] Transform and optimize the point cloud model to obtain a simplified model.

[0168] Embodiment Five

[0169] Embodiment Five of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, it is used to execute an automatic modeling method, and the method includes:

[0170] Extract feature points from the photos in the image set through a first preset algorithm or a second preset algorithm to obtain two different sets of feature points. The first preset algorithm and the second preset algorithm are two different feature point extraction algorithms. The image set includes multiple photos taken of the same scene under different conditions;

[0171] Match the two different sets of feature points to obtain multiple matching feature points;

[0172] Perform camera calibration based on the matching feature points and generate a point cloud model;

[0173] Transform and optimize the point cloud model to obtain a simplified model.

[0174] Optionally, when the program is executed by a processor, it can also be used to execute the automatic modeling method provided in any embodiment of the present invention.

[0175] The computer storage medium of the embodiment of the present invention may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (non-exhaustive list) of the computer-readable storage medium include: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable CD-ROM, an optical storage device, a magnetic storage device, or any suitable combination of the above. The computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0176] The computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries the computer-readable program code. Such a propagated data signal may take various forms, including but not limited to: an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer-readable signal medium may also be any computer-readable medium other than the computer-readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0177] The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, radio frequency (RF), etc., or any suitable combination of the above.

[0178] Computer program code for performing the operations of the present invention may be written in one or more programming languages or combinations thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and also including conventional procedural programming languages such as the "C" language or similar programming languages. The program code may execute entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0179] Note that the above is only the preferred embodiment of the present invention and the technical principles applied. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, re-adjustments, and substitutions can be made by those skilled in the art without departing from the protection scope of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments. Without departing from the concept of the present invention, more other equivalent embodiments may be included, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. An automatic modeling method, characterized in that, The method includes: Extracting feature points from the photos in the image set through a first preset algorithm or a second preset algorithm to obtain two different sets of feature points. The first preset algorithm and the second preset algorithm are two different feature point extraction algorithms. The image set includes multiple photos taken of the same scene under different conditions; Performing feature point matching on the two different sets of feature points to obtain multiple matching feature points; Performing camera calibration based on the matching feature points and generating a point cloud model; Converting and optimizing the point cloud model to obtain a simplified model; The converting and optimizing the point cloud model to obtain a simplified model includes: Converting the point cloud model into a colored object model through a preset conversion algorithm; Optimizing the colored object model through a preset simplification algorithm to obtain a simplified model; wherein, the preset simplification algorithm is an algorithm for deleting useless points in the model; The converting the point cloud model into a colored object model through a preset conversion algorithm includes: Constructing a colorless object model based on the point cloud model; Generating a colored object model based on the colorless object model; The constructing a colorless object model based on the point cloud model includes: Converting the point cloud in the point cloud model into a point cloud in a target format; Dividing the point cloud in the target format into multiple groups of point clouds; Setting an octree search index for each group of point clouds; Constructing a colorless object model according to the octree search index.

2. The method according to claim 1, wherein The two different sets of feature points include a first set of feature points and a second set of feature points; Among them, the feature points extracted from the image set through the first preset algorithm are combined into the first set of feature points, and the feature points extracted from the image set through the second preset algorithm are combined into the second set of feature points.

3. The method according to claim 2, characterized in that, The extracting feature points from the photos in the image set through a first preset algorithm or a second preset algorithm to obtain two different sets of feature points includes: for each photo in the image set, performing feature point extraction through the first preset algorithm; If the number of feature points extracted through the first preset algorithm is less than a preset value, then performing feature point extraction through the second preset algorithm.

4. The method according to claim 3, characterized in that If the number of feature points obtained by performing feature point extraction on the photos in the image set through the first preset algorithm is less than a preset value, and the feature points obtained after performing feature point extraction through the second preset algorithm cannot be used for feature point matching, then removing the photo from the image set.

5. The method according to claim 1, characterized in that The performing camera calibration based on the matching feature points and generating a point cloud model includes: Performing camera calibration according to the matching feature points to obtain a projection matrix; Obtaining a unified coordinate system and unifying the matching feature points into the unified coordinate system; Generating a point cloud model based on the projection matrix and the matching feature points in the unified coordinate system.

6. The method according to claim 1, wherein The optimizing the colored object model through a preset simplification algorithm to obtain a simplified model includes: Deleting the useless points on the colored object model through a preset simplification algorithm; Constructing a simplified model based on the points that are not deleted.

7. An automatic modeling device, characterized in that, The device includes: An extraction module, configured to extract feature points from the photos in the image set through a first preset algorithm or a second preset algorithm, to obtain two different sets of feature points. The first preset algorithm and the second preset algorithm are two different feature point extraction algorithms, and the image set includes multiple photos taken of the same scene under different conditions; A matching module, configured to perform feature point matching on the two different sets of feature points to obtain multiple matching feature points; A generation module, configured to perform camera calibration based on the matching feature points and generate a point cloud model; A transformation and optimization module, configured to transform and optimize the point cloud model to obtain a simplified model; The transformation and optimization module is specifically configured to: transform the point cloud model into a colored object model through a preset transformation algorithm; optimize the colored object model through a preset simplification algorithm to obtain a simplified model; wherein the preset simplification algorithm is an algorithm for deleting useless points in the model; The transforming the point cloud model into a colored object model through a preset transformation algorithm includes: constructing a colorless object model based on the point cloud model; generating a colored object model based on the colorless object model; The constructing a colorless object model based on the point cloud model includes: transforming the point cloud in the point cloud model into a point cloud in a target format; dividing the point cloud in the target format into multiple groups of point clouds; setting an octree search index for each group of point clouds; constructing a colorless object model according to the octree search index.

8. A computer device, characterized in that, including: One or more processors; A storage device, configured to store one or more programs; The one or more programs are executed by the one or more processors, so that the one or more processors are configured to execute the automatic modeling method according to any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the automatic modeling method according to any one of claims 1-6.

Images