Three-dimensional cloud reconstruction method and device
By integrating three-dimensional matching and two-dimensional Gaussian 3D cloud reconstruction method, the problem of low three-dimensional reconstruction accuracy in the existing technology is solved, and more efficient and accurate three-dimensional model output is achieved, suitable for dimensional measurement of buildings and decoration.
Patent Information
- Application Number
- CN202411137564.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-19
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2044-08-19
AI Technical Summary
The existing three-dimensional reconstruction methods have low accuracy in dimensional measurement. The traditional multi-viewpoint method relies on dominant features and has poor results in texture-lost areas. However, NeRF calculations are large and does not directly output mesh. 3DGS rendering speed is fast but not suitable for accurate measurement. .
A three-dimensional cloud reconstruction method that combines three-dimensional matching and two-dimensional Gaussian is used to construct a sparse point cloud collection by acquiring multiple images, learning the two-dimensional Gaussian representation of each pixel point, and projecting and optimizing under preset conditions to output a three-dimensional model.
Improves the accuracy and efficiency of the three-dimensional reconstruction model, reduces costs, and is suitable for dimensional measurements for buildings and decoration, providing closer to the real dimensional measurement results.
Smart Images

Figure CN119091043A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision, and in particular to a three-dimensional cloud reconstruction method and device. Background Art
[0002] In construction and decoration work, dimension measurement is a crucial link. However, existing measurement methods usually require manual visits, which are inefficient, costly, and dangerous to operate at heights. At the same time, the measurement results rely on the operation of the measurement personnel and the reading of the data, which is highly random and prone to errors. In addition, the process cannot process the original data. Once a large error occurs, it is necessary to visit the site again to re-measure in order to correct it.
[0003] With the continuous development of image processing and computer vision technology, image-based 3D reconstruction and dimension measurement methods have made great progress. They are mainly divided into traditional multi-viewpoint methods (MVS) and neural network radiation field-based methods that have emerged in recent years.
[0004] The traditional MVS method has a mature, stable and clearly expressed geometric mesh (polygonal mesh), but it relies heavily on explicit features and does not work well in areas with missing textures, occluded areas, and areas with severe noise.
[0005] NeRF (Neural Radiance Fields) models the light in the scene and implicitly learns the continuous volume field of the scene through a neural network. It is good at synthesizing new views of complex scenes from sparse views, but does not directly output meshes; even if the implicit field is converted into an explicit mesh through an additional step, its accuracy is still not as good as the explicit MVS. In addition, NeRF also faces the problem of large computational complexity.
[0006] 3DGS (3D Gaussian Splatting) uses Gaussian Splatting technology to approximate the contribution of a point in the scene to an image pixel, which makes rendering fast, but the explicit expression of the mesh is not its direct output. Although there are works such as GaMeS (a computer graphics and mixed reality online platform) that strengthen the editing of meshes, their work is still oriented towards rendering rather than precise 3D measurement. Summary of the invention
[0007] The present invention provides a three-dimensional cloud reconstruction method and device, which are used to solve the defect of low dimensional accuracy of three-dimensional reconstruction models in the prior art, and realize a three-dimensional cloud reconstruction method integrating stereo matching and two-dimensional Gaussian.
[0008] The present invention provides a three-dimensional cloud reconstruction method, comprising: Acquire multiple images of the scene to be measured, and construct a sparse point cloud set of the scene to be measured based on multi-view stereo matching; Taking each pixel point in the sparse point cloud set as the center, construct and learn a two-dimensional Gaussian representation of the neighborhood of each pixel point in the three-dimensional space; Projecting the two-dimensional Gaussian representation of the neighborhood of each pixel point to the original viewpoint, and calculating the residual between the projection result of each pixel point and the true value, wherein the original viewpoint represents the point in the sparse point cloud set corresponding to the two-dimensional Gaussian representation; When the residual is smaller than a preset residual threshold and the radius of the two-dimensional Gaussian representation of the pixel point neighborhood is smaller than a preset radius threshold, a three-dimensional model of the scene to be measured is output.
[0009] According to a three-dimensional cloud reconstruction method provided by the present invention, when the residual is not less than a preset residual threshold or the radius is not less than a preset radius threshold, multiple points are sampled as new center points in the two-dimensional Gaussian representation, and a two-dimensional Gaussian representation of the neighborhood of the new center point is constructed and learned; When the residual and radius corresponding to the two-dimensional Gaussian representation of the new center point neighborhood are respectively less than a preset residual threshold and a preset radius threshold, a three-dimensional model of the scene to be measured is output based on the two-dimensional Gaussian representation of the new center point neighborhood.
[0010] According to a three-dimensional cloud reconstruction method provided by the present invention, the step of constructing and learning a two-dimensional Gaussian representation of the neighborhood of each pixel point in the three-dimensional space with each pixel point in the sparse point cloud set as the center specifically includes: Construct a two-dimensional Gaussian circle with the coordinates of each pixel point in three-dimensional space as the center and a preset length as the radius; The color expression of each point in the two-dimensional Gaussian circle is determined according to the distance between each point and the center of the circle and the color of the center of the circle.
[0011] According to a three-dimensional cloud reconstruction method provided by the present invention, the step of constructing and learning a two-dimensional Gaussian representation of the neighborhood of each pixel point in the three-dimensional space with each pixel point in the sparse point cloud set as the center specifically includes: Projecting each point in the two-dimensional Gaussian representation to below the original viewpoint to obtain a projection image of the original viewpoint; When the color and space of the projection image satisfy a preset color constraint condition and a preset space constraint condition respectively, the parameters of the color expression are learned and obtained.
[0012] According to a three-dimensional cloud reconstruction method provided by the present invention, the step of projecting each point in the two-dimensional Gaussian representation to below the original viewpoint to obtain a projection image of the original viewpoint specifically includes: Each point in the two-dimensional Gaussian representation is projected below the original viewpoint, and the projection result is optimized using a comprehensive weight to obtain a projected image of the original viewpoint, wherein the comprehensive weight represents the influence of occlusion and transparency on the projected image.
[0013] The present invention also provides a three-dimensional cloud reconstruction device, comprising: An acquisition module, used to acquire multiple images of the scene to be measured, and construct a sparse point cloud set of the scene to be measured based on multi-view stereo matching; A learning module, used to construct and learn a two-dimensional Gaussian representation of the neighborhood of each pixel point in the three-dimensional space with each pixel point in the sparse point cloud set as the center; A calculation module, used to project the two-dimensional Gaussian representation of the neighborhood of each pixel point to an original viewpoint, and calculate the residual between the projection result of each pixel point and the true value, wherein the original viewpoint represents a point in the sparse point cloud set corresponding to the two-dimensional Gaussian representation; The output module is used to output the three-dimensional model of the scene to be measured when the residual is smaller than a preset residual threshold and the radius of the two-dimensional Gaussian representation of the pixel point neighborhood is smaller than a preset radius threshold.
[0014] A three-dimensional cloud reconstruction device provided by the present invention also includes: A collection module, used for collecting the multiple images of the scene to be measured and sending them to the acquisition module; A display module is used to receive and display the three-dimensional model of the scene to be measured output by the output module.
[0015] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any one of the above-mentioned three-dimensional cloud reconstruction methods is implemented.
[0016] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the three-dimensional cloud reconstruction method as described in any one of the above is implemented.
[0017] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned three-dimensional cloud reconstruction methods.
[0018] The 3D cloud-based reconstruction method and device provided by the present invention combine 2D Gaussian sputtering with the traditional MVS framework, utilize the advantages of the traditional MVS framework in linear 3D reconstruction accuracy, and the excellent rendering ability of 2D Gaussian sputtering in texture missing and repeated areas, to achieve 3D reconstruction based on multiple acquired images of the scene to be measured, so that the reconstructed 3D model has a size that is closer to the real size, and is used to measure various sizes in the scene to be measured based on the reconstructed model, thereby improving the efficiency of traditional building and decoration size measurement and reducing the cost of size measurement. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0020] Figure 1 This is one of the flow charts of the three-dimensional cloud reconstruction method provided by the present invention; Figure 2 This is the second flow chart of the three-dimensional cloud reconstruction method provided by the present invention; Figure 3 This is one of the structural schematic diagrams of the three-dimensional cloud reconstruction device provided by the present invention; Figure 4 It is a flowchart corresponding to the three-dimensional cloud reconstruction device provided by the present invention; Figure 5 It is a schematic diagram for displaying the acquisition module in the three-dimensional cloud reconstruction device provided by the present invention; Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION
[0021] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0022] Combine the following Figure 1 and Figure 2 The three-dimensional cloud reconstruction method of the present invention is introduced. Figure 1 As shown, including: Step 101, acquiring multiple images of a scene to be measured, and constructing a sparse point cloud set of the scene to be measured based on multi-view stereo matching; The scene to be measured is a scene whose dimensions need to be measured. The multiple images of the scene to be measured are images of the scene to be measured taken at different angles and different positions, and there needs to be some overlapping content in the multiple images of the scene to be measured.
[0023] Optionally, the images of the scene to be measured may be images taken by different mobile terminals such as mobile phones and cameras. The photographer holds the mobile terminal and takes multiple images of the scene to be measured at different nonlinear positions.
[0024] Optionally, the terminal transmits the acquired multiple images of the scene to be measured to the back end through the cloud, so that the back end realizes three-dimensional reconstruction of the scene to be measured based on the multiple images of the scene to be measured.
[0025] For multiple images of the scene to be measured I 1 ,I 2 ,……I N , a sparse point cloud set of the scene to be measured is constructed based on multi-viewstereo (MVS) technology.
[0026] Optionally, MVS performs sparse reconstruction on multiple acquired images of the scene to be measured based on an SFM algorithm (Structure from Motion) to obtain a sparse point cloud set of the scene to be measured.
[0027] Step 102, constructing and learning a two-dimensional Gaussian representation of the neighborhood of each pixel point in the three-dimensional space with each pixel point in the sparse point cloud set as the center; Each pixel in the sparse point cloud set is projected into three-dimensional space, where the i-th point in the set is represented as , the projection matrix of the i-th point is , where is the rotation matrix, is the translation vector, is the internal parameter matrix.
[0028] In three-dimensional space, for each real pixel , with each pixel as the center and a certain length as the radius, a 2D Gaussian Splatting (2DGS) representation of its neighborhood is constructed. It can be understood that That is, the projection point of each point in the sparse point cloud in three-dimensional space. The texture of the reconstructed model is enhanced by two-dimensional Gaussian to obtain a more realistic reconstructed model.
[0029] Among them, the two-dimensional Gaussian representation is mainly the color representation of each pixel in the neighborhood. The parameters in the two-dimensional Gaussian representation are determined based on the real value learning of multiple images of the scene to be measured, so as to obtain the two-dimensional Gaussian representation of the scene to be modeled.
[0030] In one possible implementation, for each point The two-dimensional Gaussian representation of the neighborhood is projected back to the original viewpoint, that is, it is projected back to its corresponding two-dimensional space, and based on the true value of the two-dimensional space, that is, the value corresponding to the image of the scene to be measured, the parameters in the two-dimensional Gaussian representation are learned based on the principles of color consistency and spatial consistency to obtain the two-dimensional Gaussian representation in the three-dimensional space of each pixel in the sparse point cloud set, and then the two-dimensional Gaussian representation of the scene to be measured in the three-dimensional space is learned.
[0031] Step 103, projecting the two-dimensional Gaussian representation of the neighborhood of each pixel point to the original viewpoint, and calculating the residual between the projection result of each pixel point and the true value, wherein the original viewpoint represents the point in the sparse point cloud set corresponding to the two-dimensional Gaussian representation; After determining the parameters of each two-dimensional Gaussian representation, the two-dimensional Gaussian representation of the neighborhood of each pixel is projected back to the original viewpoint, and the residual between the projection result and the true value of each pixel is calculated. The residual represents the gap between the projection result and the original result, which is used to measure the accuracy of three-dimensional reconstruction.
[0032] The true value is an actual value determined according to the image of the scene to be measured corresponding to the original viewpoint.
[0033] Step 104: when the residual is smaller than a preset residual threshold and the radius of the two-dimensional Gaussian representation of the pixel point neighborhood is smaller than a preset radius threshold, output a three-dimensional model of the scene to be measured.
[0034] If the calculated residual is less than the preset residual threshold, it means that the projection error is small and the reconstruction accuracy meets the requirements.
[0035] At the same time, if the radius of the generated two-dimensional Gaussian representation of each pixel neighborhood is smaller than a preset radius threshold, it means that the two-dimensional Gaussian is accurate.
[0036] Optionally, the preset residual threshold and the preset radius threshold are determined empirically.
[0037] Therefore, when the residual is less than the preset residual threshold and the radius of the two-dimensional Gaussian representation of the pixel neighborhood is less than the preset radius threshold, it is considered that the texture and size of the three-dimensional model reconstructed based on the two-dimensional Gaussian representation at this time are close to the true value. At this time, the views of the original i viewpoints can be projected onto the model to generate and output a complete three-dimensional model of the scene to be measured, thereby completing the measurement of the size of the scene to be measured based on the output three-dimensional model.
[0038] The present invention combines two-dimensional Gaussian sputtering with the traditional MVS framework, utilizes the advantages of the traditional MVS framework's linear three-dimensional reconstruction accuracy, and the excellent rendering ability of two-dimensional Gaussian sputtering in texture-missing and repeated areas, to achieve three-dimensional reconstruction based on multiple acquired images of the scene to be measured, so that the reconstructed three-dimensional model has a size that is closer to the real thing, and is used to measure various sizes in the scene to be measured based on the reconstructed model, thereby improving the efficiency of traditional building and decoration size measurement and reducing the cost of size measurement.
[0039] In the three-dimensional cloud reconstruction method of the present invention, the step of constructing and learning a two-dimensional Gaussian representation of the neighborhood of each pixel point in the three-dimensional space with each pixel point in the sparse point cloud set as the center specifically includes: Construct a two-dimensional Gaussian circle with the coordinates of each pixel point in three-dimensional space as the center and a preset length as the radius; The color expression of each point in the two-dimensional Gaussian circle is determined according to the distance between each point and the center of the circle and the color of the center of the circle.
[0040] Theoretically, the Gaussian plane has an infinite radius and an elliptical interface. However, considering its attenuation characteristics and the feasibility of actual calculation, the present invention simplifies it into a 2D Gaussian circle in a 3D space to construct a two-dimensional Gaussian representation.
[0041] Specifically, for each pixel point in three-dimensional space ,by As the center of the circle, As the radius, is the normal vector, and the color of its center point is .
[0042] in, The preset length.
[0043] It should be noted that the smaller the preset length, the more accurate the three-dimensional model obtained based on the two-dimensional Gaussian reconstruction, but the preset length needs to be satisfied so that the two-dimensional Gaussian finally constructed based on all sparsely reconstructed pixels can represent the entire scene to be measured. Therefore, the preset length should not be too small.
[0044] On this basis, any point in the 2DGS The color expression is: ; In the formula, for and the center point of the 2DGS The distance between .
[0045] In the above formula, G is a Gaussian function. and They are all determined according to the results of sparse reconstruction, and other parameters are those that need to be learned.
[0046] In the above manner, a two-dimensional Gaussian representation of the neighborhood in the three-dimensional space is constructed based on each pixel point of the sparse reconstruction.
[0047] In the three-dimensional cloud reconstruction method of the present invention, the step of constructing and learning a two-dimensional Gaussian representation of the neighborhood of each pixel point in the three-dimensional space with each pixel point in the sparse point cloud set as the center specifically includes: Projecting each point in the two-dimensional Gaussian representation to below the original viewpoint to obtain a projection image of the original viewpoint; When the color and space of the projection image satisfy a preset color constraint condition and a preset space constraint condition respectively, the parameters of the color expression are learned and obtained.
[0048] By projecting multiple two-dimensional Gaussian representations in three-dimensional space back to their corresponding original viewpoints, a projection of each original viewpoint is formed to learn the two-dimensional Gaussian representation in three-dimensional space.
[0049] The specific projection method is: ; In the formula, is the j-th two-dimensional Gaussian representation, For the grid points, is the projection matrix of the i-th viewpoint, and the obtained projection value is: .
[0050] It should be noted that in the actual calculation , which may be a floating point number and can be rounded to an integer.
[0051] Specifically, for the i-th original viewpoint, the image obtained by Gaussian projection is and the real image of the scene to be measured at the original viewpoint When the difference between the two is less than the preset color threshold, it is considered that the image obtained by Gaussian projection of the i-th original viewpoint is consistent with the real image, that is, the color of the projected image satisfies the preset color constraint.
[0052] The grid points of the projected image are compared with the grid points generated by sparse reconstruction. When the difference between the two is less than a preset length threshold, it is considered that the space of the projected image is consistent with the real image, that is, the space of the projected image satisfies the preset spatial constraints.
[0053] The preset color threshold and the preset length threshold are both determined based on experience.
[0054] The parameters in the constructed two-dimensional Gaussian representation are learned through the above two constraints, thereby obtaining a two-dimensional Gaussian representation of the three-dimensional space, that is, the parameters of color expression.
[0055] In the three-dimensional cloud reconstruction method of the present invention, the step of projecting each point in the two-dimensional Gaussian representation to the original viewpoint to obtain the projection image of the original viewpoint specifically includes: Each point in the two-dimensional Gaussian representation is projected below the original viewpoint, and the projection result is optimized using a comprehensive weight to obtain a projected image of the original viewpoint, wherein the comprehensive weight represents the influence of occlusion and transparency on the projected image.
[0056] Since the two-dimensional Gaussian is located in the three-dimensional space, in the process of projecting the two-dimensional Gaussian back to the original viewpoint, the projections of multiple two-dimensional Gaussians in the original viewpoint may overlap with each other. Therefore, the final image of the i-th original viewpoint is: ; In the formula, is the comprehensive weight formed by the projection of the j-th two-dimensional Gaussian to the i-th original viewpoint, and the comprehensive weight represents the influence of factors such as occlusion and transparency. are also parameters determined by learning.
[0057] In the three-dimensional cloud reconstruction method of the present invention, when the residual is not less than a preset residual threshold or the radius is not less than a preset radius threshold, multiple points are sampled as new center points in the two-dimensional Gaussian representation, and a two-dimensional Gaussian representation of the neighborhood of the new center point is constructed and learned; After learning and determining the relevant parameters of the two-dimensional Gaussian representation constructed in the above way, the two-dimensional Gaussian of the scene to be measured can be projected back to the original viewpoint to calculate the residual : ; In the formula, is the final projection image of the original viewpoint, is the real image of the scene to be measured corresponding to the original viewpoint.
[0058] On this basis, if the residual Greater than the preset residual threshold , it means that the projection error is large and the reconstruction accuracy is insufficient.
[0059] If the residual Less than the preset residual threshold , but its corresponding two-dimensional Gaussian Radius Greater than the preset radius threshold , it means that the corresponding two-dimensional Gaussian radius is too large to accurately express the size of the corresponding area.
[0060] Therefore, for the above two cases, it is necessary to calculate the two-dimensional Gaussian corresponding to the residual. Uniformly sample N points inside as the new center point The original two-dimensional Gaussian is divided into multiple two-dimensional Gaussians with smaller radius to obtain a more accurate two-dimensional Gaussian expression result, thereby reconstructing a three-dimensional model with higher dimensional accuracy. The method of constructing and learning a new two-dimensional Gaussian based on the new center point is the same as the method of constructing and learning the original two-dimensional Gaussian, so it will not be repeated.
[0061] When the residual and radius corresponding to the two-dimensional Gaussian representation of the new center point neighborhood are respectively less than a preset residual threshold and a preset radius threshold, a three-dimensional model of the scene to be measured is output based on the two-dimensional Gaussian representation of the new center point neighborhood.
[0062] For multiple two-dimensional Gaussians in their neighborhoods determined according to the new center point, re-project them back to the original viewpoint to obtain the projection map of the new original viewpoint and calculate the new residual. , and when the radius of the new two-dimensional Gaussian is less than a preset radius threshold, a three-dimensional model of the scene to be predicted is generated and output based on the new two-dimensional Gaussian.
[0063] If the residual corresponding to the new two-dimensional Gaussian is still not less than the preset residual threshold, or the radius of the new two-dimensional Gaussian is still not less than the preset radius threshold, the new two-dimensional Gaussian is divided in the same way until it is iterated to the corresponding residual and radius are respectively less than the preset residual threshold and the preset radius threshold, and the three-dimensional model of the scene to be measured is generated and output based on the current two-dimensional Gaussian. The complete process is as follows: Figure 2 shown.
[0064] The present invention uses an iterative method to continuously divide and obtain new two-dimensional Gaussians based on the two-dimensional Gaussians generated by sparsely reconstructed pixel points, thereby realizing a process of gradually transitioning from sparse rough matching to fine dense matching, so that the final output three-dimensional model has more realistic geometric features.
[0065] The three-dimensional cloud-based reconstruction device provided by the present invention is described below. The three-dimensional cloud-based reconstruction device described below and the three-dimensional cloud-based reconstruction method described above can be referenced to each other.
[0066] like Figure 3 As shown, the 3D cloud reconstruction device includes an acquisition module 301, a learning module 302, a calculation module 303 and an output module 304: An acquisition module 301 is used to acquire multiple images of the scene to be measured, and construct a sparse point cloud set of the scene to be measured based on multi-view stereo matching; The scene to be measured is a scene whose dimensions need to be measured. The multiple images of the scene to be measured are images of the scene to be measured taken at different angles and different positions, and there needs to be some overlapping content in the multiple images of the scene to be measured.
[0067] Optionally, the images of the scene to be measured may be images taken by different mobile terminals such as mobile phones and cameras. The photographer holds the mobile terminal and takes multiple images of the scene to be measured at different nonlinear positions.
[0068] The acquisition module, learning module, calculation module and output module in this embodiment are all deployed on the back end, thereby providing sufficient computing power support for three-dimensional reconstruction.
[0069] Optionally, the terminal transmits the acquired multiple images of the scene to be measured to the back end through the cloud, so that the back end realizes three-dimensional reconstruction of the scene to be measured based on the multiple images of the scene to be measured.
[0070] For multiple images of the scene to be measured I 1 ,I 2 ,……I N , a sparse point cloud set of the scene to be measured is constructed based on multi-viewstereo (MVS) technology.
[0071] Optionally, MVS performs sparse reconstruction on multiple acquired images of the scene to be measured based on an SFM (Structure from Motion) algorithm to obtain a sparse point cloud set of the scene to be measured.
[0072] A learning module 302 is used to construct and learn a two-dimensional Gaussian representation of a neighborhood of each pixel point in the three-dimensional space with each pixel point in the sparse point cloud set as the center; Each pixel in the sparse point cloud set is projected into three-dimensional space, where the i-th point in the set is represented as , the projection matrix of the i-th point is , where is the rotation matrix, is the translation vector, is the internal parameter matrix.
[0073] In three-dimensional space, for each real pixel , with each pixel as the center and a certain length as the radius, a two-dimensional Gaussian (2DGS) representation of its neighborhood is constructed. It can be understood that That is, the projection point of each point in the sparse point cloud in three-dimensional space. The texture of the reconstructed model is enhanced by two-dimensional Gaussian to obtain a more realistic reconstructed model.
[0074] Among them, the two-dimensional Gaussian representation mainly includes the color representation of each pixel in the neighborhood, and the parameters in the two-dimensional Gaussian representation are determined based on the real value learning of multiple images of the scene to be measured, so as to obtain the two-dimensional Gaussian representation of the scene to be modeled.
[0075] In one possible implementation, for each point The two-dimensional Gaussian representation of the neighborhood is projected back to the original viewpoint, that is, it is projected back to its corresponding two-dimensional space, and based on the true value of the two-dimensional space, that is, the value corresponding to the image of the scene to be measured, the parameters in the two-dimensional Gaussian representation are learned based on the principles of color consistency and spatial consistency to obtain the two-dimensional Gaussian representation in the three-dimensional space of each pixel in the sparse point cloud set, and then the two-dimensional Gaussian representation of the scene to be measured in the three-dimensional space is learned.
[0076] A calculation module 303 is used to project the two-dimensional Gaussian representation of the neighborhood of each pixel point to the original viewpoint, and calculate the residual between the projection result of each pixel point and the true value, wherein the original viewpoint represents the point in the sparse point cloud set corresponding to the two-dimensional Gaussian representation; After determining the parameters of each two-dimensional Gaussian representation, the two-dimensional Gaussian representation of the neighborhood of each pixel is projected back to the original viewpoint, and the residual between the projection result and the true value of each pixel is calculated. The residual represents the gap between the projection result and the original result, which is used to measure the accuracy of three-dimensional reconstruction.
[0077] The output module 304 is used to output the three-dimensional model of the scene to be measured when the residual is smaller than a preset residual threshold and the radius of the two-dimensional Gaussian representation of the pixel point neighborhood is smaller than a preset radius threshold.
[0078] If the calculated residual is less than the preset residual threshold, it means that the projection error is small and the reconstruction accuracy meets the requirements.
[0079] At the same time, if the radius of the generated two-dimensional Gaussian representation of each pixel neighborhood is smaller than a preset radius threshold, it means that the two-dimensional Gaussian is accurate.
[0080] Optionally, the preset residual threshold and the preset radius threshold are determined empirically.
[0081] Therefore, when the residual is less than the preset residual threshold and the radius of the two-dimensional Gaussian representation of the pixel neighborhood is less than the preset radius threshold, it is considered that the texture and size of the three-dimensional model reconstructed based on the two-dimensional Gaussian representation at this time are close to the true value. At this time, the views of the original i viewpoints can be projected onto the model to generate and output a complete three-dimensional model of the scene to be measured, thereby completing the measurement of the size of the scene to be measured based on the output three-dimensional model.
[0082] The present invention combines two-dimensional Gaussian sputtering with the traditional MVS framework, utilizes the advantages of the traditional MVS framework's linear three-dimensional reconstruction accuracy, and the excellent rendering ability of two-dimensional Gaussian sputtering in texture-missing and repeated areas, to achieve three-dimensional reconstruction based on multiple acquired images of the scene to be measured, so that the reconstructed three-dimensional model has a size that is closer to the real thing, and is used to measure various sizes in the scene to be measured based on the reconstructed model, thereby improving the efficiency of traditional building and decoration size measurement and reducing the cost of size measurement.
[0083] The three-dimensional cloud-based reconstruction device of the present application also includes a collection module and a display module.
[0084] A collection module, used for collecting the multiple images of the scene to be measured and sending them to the acquisition module; Specifically, the acquisition module is deployed at the front end.
[0085] In a possible implementation, Figure 4 As shown, the acquisition module is the front-end software, which collects images / videos of buildings to ensure the quality and clarity of the images / videos. The front-end software includes but is not limited to APP, WeChat applet, etc. Users can perform simple operations through the APP, such as selecting the shooting angle, adjusting the light, etc., to obtain the best image effect.
[0086] For the object or scene to be tested, the front-end software automatically generates shooting angle prompts to guide the user to shoot from different angles and positions with the object or scene to be tested as the center, so as to obtain multiple images of the scene to be tested at different angles and in a nonlinear manner.
[0087] It should be noted that in order to ensure the 3D reconstruction effect of the scene to be measured, the images collected by the front end need to be clear and the number should be more than 20. At the same time, the images obtained by two consecutive shots should try to ensure that the overlap area is more than 70%.
[0088] At the same time, in order to fully express the object or scene to be measured, it is necessary to expand the preset distance in all directions with the object or scene to be measured as the center, that is, the captured image needs to include the environment around the object or scene to be measured. In this embodiment, the preset distance is 1 meter.
[0089] Furthermore, for large-sized objects or scenes to be measured, it is necessary to take local shots of areas with more details of the object or scene to be measured after taking multiple overall images, and the overall image and the local images should be gradually transitioned.
[0090] Based on the above shooting principles, the front-end software gives shooting prompts to the user in the application page, thereby guiding the user to shoot and obtain multiple images of the scene to be measured. In a feasible implementation, Figure 5 As shown, for Figure 5 The application generates multiple camera icons at different positions and angles on the display page for the object to be measured, and prompts the user to take pictures of the object to be measured at the angle and position corresponding to the camera icon.
[0091] Furthermore, for the multiple images of the scenes to be measured collected by the front end, an efficient network transmission protocol is used to quickly and stably transmit the collected images / videos to the backend server through Ethernet / WIFI / cellular network, etc., to ensure the integrity of the data.
[0092] A display module is used to receive and display the three-dimensional model of the scene to be measured output by the output module.
[0093] The display module is deployed on the front end.
[0094] After the output module outputs the 3D model of the scene to be measured, professional engineers will mark and measure the distances of key parts based on the output high-precision 3D model, and send the results to the display module for display, providing accurate data support for construction and decoration.
[0095] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6As shown, the electronic device may include: a processor (processor) 610, a communication interface (Communications Interface) 620, a memory (memory) 630 and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 can call the logic instructions in the memory 630 to execute a three-dimensional cloud reconstruction method, which includes: acquiring multiple images of the scene to be measured, and constructing a sparse point cloud set of the scene to be measured based on multi-view stereo matching; constructing and learning a two-dimensional Gaussian representation of the neighborhood of each pixel point in the three-dimensional space with each pixel point in the sparse point cloud set as the center; projecting the two-dimensional Gaussian representation of the neighborhood of each pixel point to the original viewpoint, and calculating the residual between the projection result of each pixel point and the true value, wherein the original viewpoint represents the point in the sparse point cloud set corresponding to the two-dimensional Gaussian representation; when the residual is less than a preset residual threshold and the radius of the two-dimensional Gaussian representation of the pixel point neighborhood is less than a preset radius threshold, outputting a three-dimensional model of the scene to be measured.
[0096] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.
[0097] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the three-dimensional cloud reconstruction method provided by the above methods, which includes: acquiring multiple images of the scene to be measured, and constructing a sparse point cloud set of the scene to be measured based on multi-view stereo matching; constructing and learning a two-dimensional Gaussian representation of the neighborhood of each pixel point in the three-dimensional space with each pixel point in the sparse point cloud set as the center; projecting the two-dimensional Gaussian representation of the neighborhood of each pixel point to the original viewpoint, and calculating the residual between the projection result of each pixel point and the true value, wherein the original viewpoint represents the point in the sparse point cloud set corresponding to the two-dimensional Gaussian representation; when the residual is less than a preset residual threshold and the radius of the two-dimensional Gaussian representation of the pixel point neighborhood is less than a preset radius threshold, outputting a three-dimensional model of the scene to be measured.
[0098] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the three-dimensional cloud reconstruction method provided by the above-mentioned methods, the method comprising: acquiring multiple images of the scene to be measured, and constructing a sparse point cloud set of the scene to be measured based on multi-view stereo matching; constructing and learning a two-dimensional Gaussian representation of the neighborhood of each pixel point in the three-dimensional space with each pixel point in the sparse point cloud set as the center; projecting the two-dimensional Gaussian representation of the neighborhood of each pixel point to the original viewpoint, and calculating the residual between the projection result of each pixel point and the true value, wherein the original viewpoint represents the point in the sparse point cloud set corresponding to the two-dimensional Gaussian representation; when the residual is less than a preset residual threshold, and the radius of the two-dimensional Gaussian representation of the pixel point neighborhood is less than a preset radius threshold, outputting a three-dimensional model of the scene to be measured.
[0099] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0100] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0101] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A three-dimensional cloud reconstruction method, characterized in that: include: Acquire multiple images of the scene to be measured, and construct a sparse point cloud set of the scene to be measured based on multi-view stereo matching; Taking each pixel point in the sparse point cloud set as the center, construct and learn a two-dimensional Gaussian representation of the neighborhood of each pixel point in the three-dimensional space; Projecting the two-dimensional Gaussian representation of the neighborhood of each pixel point to the original viewpoint, and calculating the residual between the projection result of each pixel point and the true value, wherein the original viewpoint represents the point in the sparse point cloud set corresponding to the two-dimensional Gaussian representation; When the residual is smaller than a preset residual threshold and the radius of the two-dimensional Gaussian representation of the pixel point neighborhood is smaller than a preset radius threshold, a three-dimensional model of the scene to be measured is output.
2. The three-dimensional cloud reconstruction method according to claim 1, characterized in that: Also includes: When the residual is not less than a preset residual threshold or the radius is not less than a preset radius threshold, sampling a plurality of points in the two-dimensional Gaussian representation as new center points, and constructing and learning a two-dimensional Gaussian representation of the neighborhood of the new center point; When the residual and radius corresponding to the two-dimensional Gaussian representation of the new center point neighborhood are respectively less than a preset residual threshold and a preset radius threshold, a three-dimensional model of the scene to be measured is output based on the two-dimensional Gaussian representation of the new center point neighborhood.
3. The three-dimensional cloud reconstruction method according to claim 1, characterized in that: The step of constructing and learning a two-dimensional Gaussian representation of the neighborhood of each pixel point in the three-dimensional space with each pixel point in the sparse point cloud set as the center specifically includes: Construct a two-dimensional Gaussian circle with the coordinates of each pixel point in three-dimensional space as the center and a preset length as the radius; The color expression of each point in the two-dimensional Gaussian circle is determined according to the distance between each point and the center of the circle and the color of the center of the circle.
4. The three-dimensional cloud reconstruction method according to claim 3, characterized in that: The step of constructing and learning a two-dimensional Gaussian representation of the neighborhood of each pixel point in the three-dimensional space with each pixel point in the sparse point cloud set as the center specifically includes: Projecting each point in the two-dimensional Gaussian representation to below the original viewpoint to obtain a projection image of the original viewpoint; When the color and space of the projection image satisfy a preset color constraint condition and a preset space constraint condition respectively, the parameters of the color expression are learned and obtained.
5. The three-dimensional cloud reconstruction method according to claim 4, characterized in that: The step of projecting each point in the two-dimensional Gaussian representation to the original viewpoint to obtain a projection image of the original viewpoint specifically includes: Each point in the two-dimensional Gaussian representation is projected below the original viewpoint, and the projection result is optimized using a comprehensive weight to obtain a projected image of the original viewpoint, wherein the comprehensive weight represents the influence of occlusion and transparency on the projected image.
6. A three-dimensional cloud reconstruction device, characterized in that: include: An acquisition module, used to acquire multiple images of the scene to be measured, and construct a sparse point cloud set of the scene to be measured based on multi-view stereo matching; A learning module, used to construct and learn a two-dimensional Gaussian representation of the neighborhood of each pixel point in the three-dimensional space with each pixel point in the sparse point cloud set as the center; A calculation module, used to project the two-dimensional Gaussian representation of the neighborhood of each pixel point to an original viewpoint, and calculate the residual between the projection result of each pixel point and the true value, wherein the original viewpoint represents a point in the sparse point cloud set corresponding to the two-dimensional Gaussian representation; The output module is used to output the three-dimensional model of the scene to be measured when the residual is smaller than a preset residual threshold and the radius of the two-dimensional Gaussian representation of the pixel point neighborhood is smaller than a preset radius threshold.
7. The 3D cloud reconstruction device according to claim 6, characterized in that: Also includes: A collection module, used for collecting the multiple images of the scene to be measured and sending them to the acquisition module; A display module is used to receive and display the three-dimensional model of the scene to be measured output by the output module.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the three-dimensional cloud reconstruction method as described in any one of claims 1 to 5 is implemented.
9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the three-dimensional cloud reconstruction method as described in any one of claims 1 to 5 is implemented.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the three-dimensional cloud reconstruction method as described in any one of claims 1 to 5 is implemented.
Citation Information
Patent Citations
Color-information-based scale invariant feature point describing and matching method
CN102184411A
Automatic registration and fusion method of point cloud data and optical image based on point feature
CN106485690A
A target object size measurement method of sparse point cloud
CN109903327A
Radiation field model reconstruction method and device, computer equipment and storage medium
CN117593436A
3D modeling reconstruction system, method and device based on point cloud information and Gaussian cloud cluster
CN118196306A
Cited By
Rendering optimization method and system for updating cluster building in super-large scale city
CN119991903A
