Three-dimensional cloud reconstruction method and device

Abstract

This invention provides a method and apparatus for 3D cloud reconstruction. The method includes: acquiring multiple images of a scene to be measured, and constructing a sparse point cloud set of the scene to be measured based on multi-viewpoint stereo matching; constructing and learning a two-dimensional Gaussian representation of the neighborhood of each pixel in the sparse point cloud set as the center; projecting the two-dimensional Gaussian representation of the neighborhood of each pixel onto the original viewpoint, and calculating the residual between the projection result and the true value of each pixel; and outputting a 3D model of the scene to be measured when the residual is less than a preset residual threshold and the radius of the two-dimensional Gaussian representation of the neighborhood of the pixel is less than a preset radius threshold. This invention combines two-dimensional Gaussian sputtering with a traditional multi-viewpoint stereo matching framework to generate a more accurate 3D model.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and more particularly to a three-dimensional cloud reconstruction method and apparatus. Background Technology

[0002] In construction and renovation work, dimensional measurement is a crucial step. However, existing measurement methods typically require manual on-site visits, which are inefficient, costly, and dangerous due to the height of the work. Furthermore, the measurement results depend heavily on the surveyor's operation and data interpretation, introducing significant randomness and making errors prone to occur. Moreover, this process cannot verify the original data; once a significant error occurs, a second on-site measurement is necessary to correct it.

[0003] With the continuous development of image processing and computer vision technologies, image-based 3D reconstruction and dimensional measurement methods have made significant progress. These methods are mainly divided into traditional multi-view (MVS) methods and, in recent years, neural network-based radiation field methods.

[0004] Traditional MVS methods have mature and stable geometric meshes (polygonal meshes) with clear explicit expressions, but they rely heavily on explicit features and perform poorly in areas with missing textures, occluded areas, and areas with severe noise.

[0005] NeRF (Neural Radiance Fields) models light in a scene by implicitly learning the continuous volume field of the scene through a neural network. It excels at synthesizing new views of complex scenes from sparse views, but it does not directly output a mesh. Even with additional steps to convert the implicit field into an explicit mesh, its accuracy is still inferior to that of explicit MVS. In addition, NeRF also faces the problem of high computational cost.

[0006] 3DGS (3D Gaussian Splatting) approximates the contribution of points in a scene to image pixels using Gaussian splatting techniques, resulting in fast rendering. However, an explicit representation of the mesh is not its direct output. While works such as GaMeS (an online platform for computer graphics and mixed reality) have enhanced mesh editing, they remain primarily focused on rendering, not precise 3D measurement. Summary of the Invention

[0007] This invention provides a three-dimensional cloud reconstruction method and apparatus to address the shortcomings of low dimensional accuracy in existing three-dimensional reconstruction models, and to realize a three-dimensional cloud reconstruction method that integrates stereo matching and two-dimensional Gaussian methods.

[0008] This invention provides a three-dimensional cloud reconstruction method, comprising:

[0009] Acquire multiple images of the scene to be measured, and construct a sparse point cloud set of the scene to be measured based on multi-view stereo matching;

[0010] A two-dimensional Gaussian representation of the neighborhood of each pixel in the sparse point cloud set is constructed and learned in three-dimensional space;

[0011] The two-dimensional Gaussian representation of the neighborhood of each pixel is projected onto the original viewpoint, and the residual between the projection result and the true value of each pixel is calculated, wherein the original viewpoint represents the point in the sparse point cloud set corresponding to the two-dimensional Gaussian representation;

[0012] If the residual is less than a preset residual threshold and the radius of the two-dimensional Gaussian representation of the neighborhood of the pixel is less than a preset radius threshold, the three-dimensional model of the scene to be measured is output.

[0013] According to a three-dimensional cloud reconstruction method provided by the present invention, when the residual is not less than a preset residual threshold or the radius is not less than a preset radius threshold, multiple points are sampled in the two-dimensional Gaussian representation as new center points, and a two-dimensional Gaussian representation of the neighborhood of the new center points is constructed and learned.

[0014] If the residual and radius corresponding to the two-dimensional Gaussian representation of the new center point neighborhood are less than the preset residual threshold and preset radius threshold, respectively, a three-dimensional model of the scene to be measured is output based on the two-dimensional Gaussian representation of the new center point neighborhood.

[0015] According to a three-dimensional cloud reconstruction method provided by the present invention, the step of constructing and learning a two-dimensional Gaussian representation of the neighborhood of each pixel in the three-dimensional space centered on each pixel in the sparse point cloud set specifically includes:

[0016] A two-dimensional Gaussian circle is constructed with the coordinates of each pixel in three-dimensional space as the center and a preset length as the radius.

[0017] The color representation of each point is determined based on the distance between each point in the two-dimensional Gaussian circle and the center of the circle, as well as the color of the center of the circle.

[0018] According to a three-dimensional cloud reconstruction method provided by the present invention, the step of constructing and learning a two-dimensional Gaussian representation of the neighborhood of each pixel in the three-dimensional space centered on each pixel in the sparse point cloud set specifically includes:

[0019] Each point in the two-dimensional Gaussian representation is projected below the original viewpoint to obtain the projected image of the original viewpoint.

[0020] The parameters representing the color are learned when the color and space of the projected image satisfy preset color constraints and preset space constraints, respectively.

[0021] According to a three-dimensional cloud reconstruction method provided by the present invention, the step of projecting each point in the two-dimensional Gaussian representation onto the original viewpoint to obtain a projected image of the original viewpoint specifically includes:

[0022] Each point in the two-dimensional Gaussian representation is projected below the original viewpoint, and the projection result is optimized using a comprehensive weight to obtain the projected image of the original viewpoint. The comprehensive weight represents the influence of occlusion and transparency on the projected image.

[0023] The present invention also provides a three-dimensional cloud reconstruction device, comprising:

[0024] The acquisition module is used to acquire multiple images of the scene to be measured and construct a sparse point cloud set of the scene to be measured based on multi-view stereo matching;

[0025] The learning module is used to construct and learn a two-dimensional Gaussian representation of the neighborhood of each pixel in the three-dimensional space, centered on each pixel in the sparse point cloud set.

[0026] The calculation module is used to project the two-dimensional Gaussian representation of the neighborhood of each pixel point onto the original viewpoint, and calculate the residual between the projection result and the true value of each pixel point, wherein the original viewpoint represents the point in the sparse point cloud set corresponding to the two-dimensional Gaussian representation;

[0027] The output module is used to output a three-dimensional model of the scene to be measured when the residual is less than a preset residual threshold and the radius of the two-dimensional Gaussian representation of the neighborhood of the pixel is less than a preset radius threshold.

[0028] A three-dimensional cloud reconstruction device provided by the present invention further includes:

[0029] The acquisition module is used to acquire multiple images of the scene to be measured and send them to the acquisition module;

[0030] The display module is used to receive and display the 3D model of the scene to be measured output by the output module.

[0031] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the three-dimensional cloud reconstruction method as described above.

[0032] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the three-dimensional cloud reconstruction method as described above.

[0033] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements any of the three-dimensional cloud reconstruction methods described above.

[0034] The three-dimensional cloud reconstruction method and apparatus provided by this invention combine two-dimensional Gaussian sputtering with the traditional MVS framework. It utilizes the advantages of the linear three-dimensional reconstruction accuracy of the traditional MVS framework and the excellent rendering capabilities of two-dimensional Gaussian sputtering in areas with missing or repetitive textures to achieve three-dimensional reconstruction based on multiple images of the scene to be measured. This results in a three-dimensional model that is closer to the real size, which is then used to measure various dimensions in the scene to be measured based on the reconstructed model. This improves the efficiency of traditional building and decoration dimension measurement and reduces the cost of dimension measurement. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0036] Figure 1 This is one of the flowcharts illustrating the three-dimensional cloud reconstruction method provided by the present invention;

[0037] Figure 2 This is the second flowchart illustrating the three-dimensional cloud reconstruction method provided by the present invention;

[0038] Figure 3 This is one of the structural schematic diagrams of the three-dimensional cloud reconstruction device provided by the present invention;

[0039] Figure 4 This is a flowchart corresponding to the three-dimensional cloud reconstruction device provided by the present invention;

[0040] Figure 5 This is a schematic diagram illustrating the acquisition module in the three-dimensional cloud reconstruction device provided by the present invention;

[0041] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0043] The following is combined with Figure 1 and Figure 2 This invention introduces a three-dimensional cloud-based reconstruction method, such as... Figure 1 As shown, it includes:

[0044] Step 101: Acquire multiple images of the scene to be measured, and construct a sparse point cloud set of the scene to be measured based on multi-view stereo matching;

[0045] The scene to be measured is the scene whose size needs to be measured. Multiple images of the scene to be measured are taken from different angles and positions, and there needs to be some overlap between the multiple images of the scene to be measured.

[0046] Optionally, the images of the scene to be measured can be images taken by different mobile terminals such as mobile phones and cameras. The photographer holds the mobile terminal and takes multiple images of the scene to be measured at different non-linear positions.

[0047] Optionally, the terminal transmits multiple images of the scene to be measured to the backend via the cloud, thereby enabling the backend to perform 3D reconstruction of the scene to be measured based on the multiple images.

[0048] Multiple images of the scene to be measured were acquired. I 1 ，I 2 ，……I N A sparse point cloud set of the scene to be measured is constructed based on multi-viewstereo (MVS) technology.

[0049] Optionally, MVS uses the SFM algorithm (Structure from Motion) to sparsely reconstruct multiple images of the scene to be measured, thereby obtaining a sparse point cloud set of the scene to be measured.

[0050] Step 102: Construct and learn a two-dimensional Gaussian representation of the neighborhood of each pixel in the sparse point cloud set as the center;

[0051] Project each pixel in the sparse point cloud set into a 3D space, where the i-th point in the set is denoted as... The projection matrix of the i-th point is In the formula, Let be a rotation matrix. It is a translation vector. This is the intrinsic parameter matrix.

[0052] In three-dimensional space, for each real-world pixel A two-dimensional Gaussian splatting (2DGS) representation of the neighborhood of each pixel is constructed, centered on the pixel and with a radius of a certain length. This is understandable. In other words, it refers to the projection of each point in a sparse point cloud into a three-dimensional space. Two-dimensional Gaussian methods are used to enhance the texture of the reconstructed model, resulting in a more realistic reconstruction.

[0053] Among them, the two-dimensional Gaussian representation is mainly the color representation of each pixel in the neighborhood. The parameters in the two-dimensional Gaussian representation are determined by learning the true values ​​of multiple images of the scene to be measured, so as to obtain the two-dimensional Gaussian representation of the scene to be modeled.

[0054] In one feasible implementation, for each point The two-dimensional Gaussian representation of the neighborhood is projected back to the original viewpoint, that is, back to its corresponding two-dimensional space. Based on the true value of the two-dimensional space, that is, the value corresponding to the image of the scene to be measured, the parameters in the two-dimensional Gaussian representation are learned based on the principles of color consistency and spatial consistency to obtain the two-dimensional Gaussian representation of each pixel in the sparse point cloud set in the three-dimensional space, and then the two-dimensional Gaussian representation of the scene to be measured in the three-dimensional space is learned.

[0055] Step 103: Project the two-dimensional Gaussian representation of the neighborhood of each pixel point onto the original viewpoint, and calculate the residual between the projection result and the true value of each pixel point, wherein the original viewpoint represents the point in the sparse point cloud set corresponding to the two-dimensional Gaussian representation.

[0056] After determining the parameters of each 2D Gaussian representation, the 2D Gaussian representation of the neighborhood of each pixel is projected back to the original viewpoint, and the residual between the projection result and the true value of each pixel is calculated. The residual characterizes the difference between the projection result and the original result, and is used to measure the accuracy of the 3D reconstruction.

[0057] The true value is the actual value determined based on the image of the scene to be measured corresponding to the original viewpoint.

[0058] Step 104: If the residual is less than a preset residual threshold and the radius of the two-dimensional Gaussian representation of the neighborhood of the pixel is less than a preset radius threshold, output the three-dimensional model of the scene to be measured.

[0059] If the calculated residual is less than the preset residual threshold, it indicates that the projection error is small and the reconstruction accuracy meets the requirements.

[0060] Meanwhile, if the radius of the two-dimensional Gaussian representation of the neighborhood of each pixel is less than a preset radius threshold, it indicates that the two-dimensional Gaussian representation is accurate.

[0061] Optionally, the preset residual threshold and preset radius threshold are determined based on experience.

[0062] Therefore, when the residual is less than the preset residual threshold and the radius of the two-dimensional Gaussian representation of the pixel neighborhood is less than the preset radius threshold, it is considered that the texture and size of the three-dimensional model reconstructed based on the two-dimensional Gaussian representation at this time are close to the true value. At this time, the views of the original i viewpoints can be projected onto the model to generate and output the complete three-dimensional model of the scene to be measured, so as to complete the measurement of the size of the scene to be measured based on the output three-dimensional model.

[0063] This invention combines two-dimensional Gaussian sputtering with the traditional MVS framework. It leverages the advantages of the traditional MVS framework in linear 3D reconstruction accuracy and the excellent rendering capabilities of two-dimensional Gaussian sputtering in areas with missing or repetitive textures to achieve 3D reconstruction based on multiple images of the scene to be measured. This results in a 3D model that more closely approximates the actual dimensions. The reconstructed model is then used to measure various dimensions in the scene to be measured, thereby improving the efficiency of traditional architectural and interior design dimension measurement and reducing the cost of dimension measurement.

[0064] In the three-dimensional cloud reconstruction method of the present invention, the step of constructing and learning a two-dimensional Gaussian representation of the neighborhood of each pixel in the three-dimensional space, centered on each pixel in the sparse point cloud set, specifically includes:

[0065] A two-dimensional Gaussian circle is constructed with the coordinates of each pixel in three-dimensional space as the center and a preset length as the radius.

[0066] The color representation of each point is determined based on the distance between each point in the two-dimensional Gaussian circle and the center of the circle, as well as the color of the center of the circle.

[0067] Theoretically, the Gaussian plane has an infinite radius and an elliptical interface. However, considering its decay characteristics and the feasibility of practical calculations, this invention simplifies it to a two-dimensional Gaussian representation constructed from 2D Gaussian circles in 3D space.

[0068] Specifically, for each pixel in three-dimensional space ,by With the center as the center, With radius, The normal vector is and the color of its center point is . .

[0069] in, This is the preset length.

[0070] It should be noted that the smaller the preset length, the more accurate the 3D model reconstructed based on the 2D Gaussian. However, the preset length must be sufficient to ensure that the final 2D Gaussian constructed based on all sparsely reconstructed pixels can represent the entire scene to be measured. Therefore, the preset length should not be too small.

[0071] Based on this, any point in the 2DGS The color is expressed as: ;

[0072] In the formula, for and the center point of the 2DGS The distance between them; .

[0073] In the above formula, G is the Gaussian function. and All parameters are determined based on the results of sparse reconstruction, while other parameters are those that need to be learned.

[0074] The above method, which is based on the sparse reconstruction of each pixel, constructs a two-dimensional Gaussian representation of its neighborhood in three-dimensional space.

[0075] In the three-dimensional cloud reconstruction method of the present invention, the step of constructing and learning a two-dimensional Gaussian representation of the neighborhood of each pixel in the three-dimensional space, centered on each pixel in the sparse point cloud set, specifically includes:

[0076] Each point in the two-dimensional Gaussian representation is projected below the original viewpoint to obtain the projected image of the original viewpoint.

[0077] The parameters representing the color are learned when the color and space of the projected image satisfy preset color constraints and preset space constraints, respectively.

[0078] By projecting multiple two-dimensional Gaussian representations in three-dimensional space back to their corresponding original viewpoints, a projection of each original viewpoint is formed, which is used to learn the two-dimensional Gaussian representations in three-dimensional space.

[0079] Its projection method is as follows:

[0080] ;

[0081] In the formula, Let j be the j-th two-dimensional Gaussian representation. For the grid points, The projection matrix for the i-th viewpoint yields the following projection values: .

[0082] It should be noted that in actual calculations... It may be a floating-point number, which can be rounded to the nearest integer.

[0083] Specifically, for the i-th original viewpoint, the image obtained by Gaussian projection... and the actual image of the scene to be measured at this original viewpoint. The images are compared, and if the difference between the two is less than the preset color threshold, the image obtained by the Gaussian projection of the i-th original viewpoint is considered to be consistent with the real image, that is, the color of the projected image satisfies the preset color constraint condition.

[0084] The grid points of the projected image are compared with the grid points generated by sparse reconstruction. If the difference between the two is less than a preset length threshold, the space of the projected image is considered to be consistent with that of the real image, that is, the space of the projected image satisfies the preset spatial constraints.

[0085] The preset color threshold and preset length threshold are both determined based on experience.

[0086] By learning the parameters in the constructed two-dimensional Gaussian representation through the above two constraints, a two-dimensional Gaussian representation in three-dimensional space is obtained, which is the parameter for color representation.

[0087] In the three-dimensional cloud reconstruction method of the present invention, the step of projecting each point in the two-dimensional Gaussian representation onto the original viewpoint to obtain a projected image of the original viewpoint specifically includes:

[0088] Each point in the two-dimensional Gaussian representation is projected below the original viewpoint, and the projection result is optimized using a comprehensive weight to obtain the projected image of the original viewpoint. The comprehensive weight represents the influence of occlusion and transparency on the projected image.

[0089] Since the 2D Gaussian is located in 3D space, during the process of projecting the 2D Gaussian back to the original viewpoint, multiple 2D Gaussian projections in the original viewpoint may overlap. Therefore, the final image of the i-th original viewpoint is:

[0090] ;

[0091] In the formula, Let be the comprehensive weight formed by projecting the j-th 2D Gaussian vector onto the i-th original viewpoint. This comprehensive weight represents the influence of factors such as occlusion and transparency. These are also parameters determined through learning.

[0092] In the three-dimensional cloud reconstruction method of the present invention, when the residual is not less than a preset residual threshold or the radius is not less than a preset radius threshold, multiple points are sampled in the two-dimensional Gaussian representation as new center points, and a two-dimensional Gaussian representation of the neighborhood of the new center points is constructed and learned.

[0093] After learning and determining the relevant parameters of the constructed two-dimensional Gaussian representation using the above method, the two-dimensional Gaussian representation of the scene to be measured can be projected back to the original viewpoint, and the residual can be calculated. :

[0094] ;

[0095] In the formula, The final projected image of the original viewpoint. This is the real image of the scene to be measured corresponding to the original viewpoint.

[0096] Based on this, if the residual Greater than the preset residual threshold If the projection error is large, the reconstruction accuracy is insufficient.

[0097] If residuals Less than the preset residual threshold However, its corresponding two-dimensional Gaussian radius Greater than the preset radius threshold This indicates that the corresponding two-dimensional Gaussian radius is large, making it difficult to accurately represent the size of the corresponding region.

[0098] Therefore, for both of the above cases, it is necessary to use the two-dimensional Gaussian corresponding to the residual. N points are uniformly sampled within the area to serve as the new center point. The original two-dimensional Gaussian is divided into multiple two-dimensional Gaussian with smaller radii to obtain a more accurate two-dimensional Gaussian representation, thereby reconstructing a three-dimensional model with higher dimensional accuracy. The method of constructing and learning a new two-dimensional Gaussian based on a new center point is the same as the method of constructing and learning the original two-dimensional Gaussian, so it will not be described in detail.

[0099] If the residual and radius corresponding to the two-dimensional Gaussian representation of the new center point neighborhood are less than the preset residual threshold and preset radius threshold, respectively, a three-dimensional model of the scene to be measured is output based on the two-dimensional Gaussian representation of the new center point neighborhood.

[0100] For multiple 2D Gaussians whose neighborhoods are determined based on the new center point, they are reprojected back to the original viewpoint to obtain a new projection map of the original viewpoint, and a new residual is calculated. The new residual is considered to be less than a preset residual threshold. If the radius of the new two-dimensional Gaussian is less than the preset radius threshold, a three-dimensional model of the scene to be predicted is generated and output based on the new two-dimensional Gaussian.

[0101] If the residual corresponding to the new 2D Gaussian is still not less than the preset residual threshold, or the radius of the new 2D Gaussian is still not less than the preset radius threshold, the new 2D Gaussian continues to be divided in the same way until it iterates until the corresponding residual and radius are less than the preset residual threshold and preset radius threshold, respectively. Based on the current 2D Gaussian, a 3D model of the scene to be measured is generated and output. The complete process is as follows: Figure 2 As shown.

[0102] This invention uses an iterative approach to continuously divide a 2D Gaussian based on sparsely reconstructed pixels to obtain new 2D Gaussians, thus achieving a gradual transition from sparse, coarse matching to fine, dense matching, resulting in a final output 3D model with more realistic geometric features.

[0103] The three-dimensional cloud reconstruction device provided by the present invention is described below. The three-dimensional cloud reconstruction device described below and the three-dimensional cloud reconstruction method described above can be referred to in correspondence.

[0104] like Figure 3 As shown, the 3D cloud reconstruction device includes an acquisition module 301, a learning module 302, a calculation module 303, and an output module 304.

[0105] The acquisition module 301 is used to acquire multiple images of the scene to be measured and to construct a sparse point cloud set of the scene to be measured based on multi-view stereo matching.

[0106] The scene to be measured is the scene whose size needs to be measured. Multiple images of the scene to be measured are taken from different angles and positions, and there needs to be some overlap between the multiple images of the scene to be measured.

[0107] Optionally, the images of the scene to be measured can be images taken by different mobile terminals such as mobile phones and cameras. The photographer holds the mobile terminal and takes multiple images of the scene to be measured at different non-linear positions.

[0108] In this embodiment, the acquisition module, learning module, calculation module, and output module are all deployed on the backend, thereby providing sufficient computing power support for 3D reconstruction.

[0109] Optionally, the terminal transmits multiple images of the scene to be measured to the backend via the cloud, thereby enabling the backend to perform 3D reconstruction of the scene to be measured based on the multiple images.

[0110] Multiple images of the scene to be measured were acquired. I 1，I 2 ，……I N A sparse point cloud set of the scene to be measured is constructed based on multi-viewstereo (MVS) technology.

[0111] Optionally, MVS uses the SFM (Structure from Motion) algorithm to sparsely reconstruct multiple images of the scene to be measured, thereby obtaining a sparse point cloud set of the scene to be measured.

[0112] Learning module 302 is used to construct and learn a two-dimensional Gaussian representation of the neighborhood of each pixel in the three-dimensional space, centered on each pixel in the sparse point cloud set.

[0113] Project each pixel in the sparse point cloud set into a 3D space, where the i-th point in the set is denoted as... The projection matrix of the i-th point is In the formula, Let be a rotation matrix. It is a translation vector. This is the intrinsic parameter matrix.

[0114] In three-dimensional space, for each real-world pixel A two-dimensional Gaussian (2DGS) representation of the neighborhood of each pixel is constructed, centered on the pixel and with a radius of a certain length. This is understandable. In other words, it refers to the projection of each point in a sparse point cloud into a three-dimensional space. Two-dimensional Gaussian methods are used to enhance the texture of the reconstructed model, resulting in a more realistic reconstruction.

[0115] The two-dimensional Gaussian representation mainly includes the color representation of each pixel in the neighborhood. Based on the real values ​​of multiple images of the scene to be measured, the parameters in the two-dimensional Gaussian representation are determined, thereby obtaining the two-dimensional Gaussian representation of the scene to be modeled.

[0116] In one feasible implementation, for each point The two-dimensional Gaussian representation of the neighborhood is projected back to the original viewpoint, that is, back to its corresponding two-dimensional space. Based on the true value of the two-dimensional space, that is, the value corresponding to the image of the scene to be measured, the parameters in the two-dimensional Gaussian representation are learned based on the principles of color consistency and spatial consistency to obtain the two-dimensional Gaussian representation of each pixel in the sparse point cloud set in the three-dimensional space, and then the two-dimensional Gaussian representation of the scene to be measured in the three-dimensional space is learned.

[0117] The calculation module 303 is used to project the two-dimensional Gaussian representation of the neighborhood of each pixel point onto the original viewpoint, and calculate the residual between the projection result and the true value of each pixel point, wherein the original viewpoint represents the point in the sparse point cloud set corresponding to the two-dimensional Gaussian representation.

[0118] After determining the parameters of each 2D Gaussian representation, the 2D Gaussian representation of the neighborhood of each pixel is projected back to the original viewpoint, and the residual between the projection result and the true value of each pixel is calculated. The residual characterizes the difference between the projection result and the original result, and is used to measure the accuracy of the 3D reconstruction.

[0119] The output module 304 is used to output a three-dimensional model of the scene to be measured when the residual is less than a preset residual threshold and the radius of the two-dimensional Gaussian representation of the neighborhood of the pixel is less than a preset radius threshold.

[0120] If the calculated residual is less than the preset residual threshold, it indicates that the projection error is small and the reconstruction accuracy meets the requirements.

[0121] Meanwhile, if the radius of the two-dimensional Gaussian representation of the neighborhood of each pixel is less than a preset radius threshold, it indicates that the two-dimensional Gaussian representation is accurate.

[0122] Optionally, the preset residual threshold and preset radius threshold are determined based on experience.

[0123] Therefore, when the residual is less than the preset residual threshold and the radius of the two-dimensional Gaussian representation of the pixel neighborhood is less than the preset radius threshold, it is considered that the texture and size of the three-dimensional model reconstructed based on the two-dimensional Gaussian representation at this time are close to the true value. At this time, the views of the original i viewpoints can be projected onto the model to generate and output the complete three-dimensional model of the scene to be measured, so as to complete the measurement of the size of the scene to be measured based on the output three-dimensional model.

[0124] This invention combines two-dimensional Gaussian sputtering with the traditional MVS framework. It leverages the advantages of the traditional MVS framework in linear 3D reconstruction accuracy and the excellent rendering capabilities of two-dimensional Gaussian sputtering in areas with missing or repetitive textures to achieve 3D reconstruction based on multiple images of the scene to be measured. This results in a 3D model that more closely approximates the actual dimensions. The reconstructed model is then used to measure various dimensions in the scene to be measured, thereby improving the efficiency of traditional architectural and interior design dimension measurement and reducing the cost of dimension measurement.

[0125] The 3D cloud reconstruction device of this application also includes a data acquisition module and a display module.

[0126] The acquisition module is used to acquire multiple images of the scene to be measured and send them to the acquisition module;

[0127] Specifically, the data acquisition module is deployed at the front end.

[0128] In one feasible implementation, such as Figure 4 As shown, the acquisition module is front-end software that captures images / videos of the building, ensuring image / video quality and clarity. Front-end software can be implemented through various means, including but not limited to apps and WeChat mini-programs. Users can perform simple operations via the app, such as selecting shooting angles and adjusting lighting, to obtain the best image results.

[0129] For the object or scene to be tested, the front-end software automatically generates shooting perspective prompts, guiding the user to take pictures from different angles and positions with the object or scene to be tested as the center, so as to obtain multiple images of the scene to be tested from different angles and non-linear.

[0130] It should be noted that, in order to ensure the 3D reconstruction effect of the scene to be measured, the images acquired by the front end need to be clear and the number needs to be more than 20. At the same time, the overlap area between two adjacent images should be at least 70%.

[0131] Meanwhile, in order to fully represent the object or scene under test, it is necessary to extend a preset distance in all directions from the object or scene under test as the center. In other words, the captured image needs to include the environment surrounding the object or scene under test. In this embodiment, the preset distance is 1 meter.

[0132] Furthermore, for large objects or scenes to be tested, after taking multiple overall images, it is necessary to take local images of areas with more details of the object or scene to be tested, and the overall images and the local images should gradually transition.

[0133] Based on the aforementioned shooting principles, the front-end software provides shooting prompts to the user on the application's page, thereby guiding the user to capture multiple images of the scene to be measured. In one feasible implementation, such as... Figure 5 As shown, for Figure 5 When the object to be measured is displayed, the application generates multiple camera icons at different positions and angles on the display page, prompting the user to take a picture of the object at the angle and position corresponding to the camera icon.

[0134] Furthermore, for multiple images of the scene to be measured collected at the front end, an efficient network transmission protocol is used to quickly and stably transmit the collected images / videos to the back-end server via Ethernet / WIFI / cellular networks, ensuring data integrity.

[0135] The display module is used to receive and display the 3D model of the scene to be measured output by the output module.

[0136] The presentation module is deployed on the front end.

[0137] After the output module outputs a 3D model of the scene to be measured, professional engineers mark and measure the distances of key parts based on the high-precision 3D model, and send the results to the display module for display, providing accurate data support for construction and decoration.

[0138] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 can call logic instructions in the memory 630 to execute a 3D cloud reconstruction method. The method includes: acquiring multiple images of the scene to be measured, and constructing a sparse point cloud set of the scene to be measured based on multi-viewpoint stereo matching; constructing and learning a two-dimensional Gaussian representation of the neighborhood of each pixel in the sparse point cloud set as the center; projecting the two-dimensional Gaussian representation of the neighborhood of each pixel onto the original viewpoint, and calculating the residual between the projection result and the true value of each pixel, wherein the original viewpoint represents the point in the sparse point cloud set corresponding to the two-dimensional Gaussian representation; and outputting a 3D model of the scene to be measured when the residual is less than a preset residual threshold and the radius of the two-dimensional Gaussian representation of the neighborhood of the pixel is less than a preset radius threshold.

[0139] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0140] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the three-dimensional cloud reconstruction method provided by the above methods. The method includes: acquiring multiple images of a scene to be measured, and constructing a sparse point cloud set of the scene to be measured based on multi-viewpoint stereo matching; constructing and learning a two-dimensional Gaussian representation of the neighborhood of each pixel in the sparse point cloud set as the center; projecting the two-dimensional Gaussian representation of the neighborhood of each pixel onto the original viewpoint, and calculating the residual between the projection result of each pixel and the true value, wherein the original viewpoint represents a point in the sparse point cloud set corresponding to the two-dimensional Gaussian representation; and outputting a three-dimensional model of the scene to be measured when the residual is less than a preset residual threshold and the radius of the two-dimensional Gaussian representation of the neighborhood of the pixel is less than a preset radius threshold.

[0141] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the three-dimensional cloud reconstruction method provided by the above methods. The method includes: acquiring multiple images of a scene to be measured, and constructing a sparse point cloud set of the scene to be measured based on multi-viewpoint stereo matching; constructing and learning a two-dimensional Gaussian representation of the neighborhood of each pixel in the sparse point cloud set as the center; projecting the two-dimensional Gaussian representation of the neighborhood of each pixel onto the original viewpoint, and calculating the residual between the projection result of each pixel and the true value, wherein the original viewpoint represents a point in the sparse point cloud set corresponding to the two-dimensional Gaussian representation; and outputting a three-dimensional model of the scene to be measured when the residual is less than a preset residual threshold and the radius of the two-dimensional Gaussian representation of the neighborhood of the pixel is less than a preset radius threshold.

[0142] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0143] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0144] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A three-dimensional cloud reconstruction method, characterized in that, include: Acquire multiple images of the scene to be measured, and construct a sparse point cloud set of the scene to be measured based on multi-view stereo matching; A two-dimensional Gaussian representation of the neighborhood of each pixel in the sparse point cloud set is constructed and learned in three-dimensional space; The two-dimensional Gaussian representation of the neighborhood of each pixel is projected onto the original viewpoint, and the residual between the projection result and the true value of each pixel is calculated, wherein the original viewpoint represents the point in the sparse point cloud set corresponding to the two-dimensional Gaussian representation; If the residual is less than a preset residual threshold and the radius of the two-dimensional Gaussian representation of the neighborhood of the pixel is less than a preset radius threshold, output a three-dimensional model of the scene to be measured. The step of constructing and learning a two-dimensional Gaussian representation of the neighborhood of each pixel in the three-dimensional space, centered on each pixel in the sparse point cloud set, specifically includes: A two-dimensional Gaussian circle is constructed with the coordinates of each pixel in three-dimensional space as the center and a preset length as the radius. The color representation of each point is determined based on the distance between each point in the two-dimensional Gaussian circle and the center of the circle, as well as the color of the center of the circle.

2. The three-dimensional cloud reconstruction method according to claim 1, characterized in that, Also includes: If the residual is not less than a preset residual threshold or the radius is not less than a preset radius threshold, multiple points are sampled in the two-dimensional Gaussian representation as new center points, and a two-dimensional Gaussian representation of the neighborhood of the new center points is constructed and learned. If the residual and radius corresponding to the two-dimensional Gaussian representation of the new center point neighborhood are less than the preset residual threshold and preset radius threshold, respectively, a three-dimensional model of the scene to be measured is output based on the two-dimensional Gaussian representation of the new center point neighborhood.

3. The three-dimensional cloud reconstruction method according to claim 1, characterized in that, The step of constructing and learning a two-dimensional Gaussian representation of the neighborhood of each pixel in the three-dimensional space, centered on each pixel in the sparse point cloud set, specifically includes: Each point in the two-dimensional Gaussian representation is projected below the original viewpoint to obtain the projected image of the original viewpoint. The parameters representing the color are learned when the color and space of the projected image satisfy preset color constraints and preset space constraints, respectively.

4. The three-dimensional cloud reconstruction method according to claim 3, characterized in that, The step of projecting each point in the two-dimensional Gaussian representation onto the original viewpoint to obtain a projected image of the original viewpoint specifically includes: Each point in the two-dimensional Gaussian representation is projected below the original viewpoint, and the projection result is optimized using a comprehensive weight to obtain the projected image of the original viewpoint. The comprehensive weight represents the influence of occlusion and transparency on the projected image.

5. A three-dimensional cloud reconstruction device, characterized in that, include: The acquisition module is used to acquire multiple images of the scene to be measured and construct a sparse point cloud set of the scene to be measured based on multi-view stereo matching; The learning module is used to construct and learn a two-dimensional Gaussian representation of the neighborhood of each pixel in the three-dimensional space, centered on each pixel in the sparse point cloud set. The calculation module is used to project the two-dimensional Gaussian representation of the neighborhood of each pixel point onto the original viewpoint, and calculate the residual between the projection result and the true value of each pixel point, wherein the original viewpoint represents the point in the sparse point cloud set corresponding to the two-dimensional Gaussian representation; The output module is used to output a three-dimensional model of the scene to be measured when the residual is less than a preset residual threshold and the radius of the two-dimensional Gaussian representation of the neighborhood of the pixel is less than a preset radius threshold. The learning module is specifically used to construct a two-dimensional Gaussian circle with the coordinates of each pixel in three-dimensional space as the center and a preset length as the radius. The color representation of each point is determined based on the distance between each point in the two-dimensional Gaussian circle and the center of the circle, as well as the color of the center of the circle.

6. The three-dimensional cloud reconstruction device according to claim 5, characterized in that, Also includes: The acquisition module is used to acquire multiple images of the scene to be measured and send them to the acquisition module; The display module is used to receive and display the 3D model of the scene to be measured output by the output module.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the three-dimensional cloud reconstruction method as described in any one of claims 1 to 4.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the three-dimensional cloud reconstruction method as described in any one of claims 1 to 4.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the three-dimensional cloud reconstruction method as described in any one of claims 1 to 4.

Images