Close-view 3D reconstruction method, device and equipment based on 3DGS

Abstract

The present invention relates to the field of three-dimensional reconstruction technology, and discloses a close-up three-dimensional reconstruction method, device and equipment based on 3DGS. The method includes taking circular shots of a target close-up based on an image acquisition device to obtain a photo set; performing three-dimensional reconstruction based on the photo set using 3DGS technology to obtain a 3DGS model; reading the point cloud file information of the 3DGS model, and performing format conversion on the point cloud file information to obtain general three-dimensional grid model data; performing three-dimensional rendering based on the general three-dimensional grid model data to visualize the close-up of the target. In the present invention, with the help of the ability of 3DGS technology in three-dimensional reconstruction, a more refined three-dimensional model is formed in the scene of the close-up model. The ply point cloud file information generated by the 3DGS model is additionally converted into a format to form standardized general three-dimensional grid model data, which overcomes the problem that the traditional ply point cloud has obvious needle-like shape in the close-up mode and is not suitable for engineering applications.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional reconstruction, and in particular to a close-up three-dimensional reconstruction method, device and equipment based on 3DGS. Background Art

[0002] With the development of hardware equipment and technology, the management of data and business processes in various industries no longer remain at the stage of data tables and paper materials, but tend to be managed in a more intuitive way. Digital twins are gradually becoming popular in various industries under this background. However, digital twins require a large number of 3D models and 3D data support. For organizations or individuals without experience in 3D data collection and processing, it is difficult to realize the 3D empowerment of digital twins.

[0003] At present, the unbounded scene reconstruction and new perspective synthesis of 3DGS are still a theoretical method, which is still a certain distance away from real three-dimensional applications and cannot really solve the problems encountered in actual business and projects. The current 3DGS application does not consider the feasibility of data collection, automated data processing, and data presentation, and it is difficult to implement in actual engineering applications. The ply point cloud model generated by 3DGS itself is not a qualified digital twin scene data. The ply model generated by 3DGS will have obvious pricking phenomenon when viewed at close range in the 3DGS engine. In addition to improving rendering performance with mathematical expressions, the 3DGS engine itself does not have secondary development capabilities, making it difficult to overlay and engineer other business data. Summary of the invention

[0004] The main purpose of the present invention is to provide a close-up three-dimensional reconstruction method, device and equipment based on 3DGS, aiming to solve at least one of the above-mentioned technical problems.

[0005] To achieve the above object, the present invention provides a close-up 3D reconstruction method based on 3DGS, comprising:

[0006] Taking circular shots of the close-up of the target based on the image acquisition device to obtain a photo collection;

[0007] Using 3DGS technology to perform three-dimensional reconstruction based on the photo collection to obtain a 3DGS model;

[0008] Reading the point cloud file information of the 3DGS model, and performing format conversion on the point cloud file information to obtain general three-dimensional mesh model data;

[0009] Three-dimensional rendering is performed based on the general three-dimensional mesh model data to visualize the near view of the target.

[0010] In some embodiments, the image acquisition device is used to perform circular photography of the near view of the target to obtain a set of photos, including:

[0011] Taking circular photos of the close-up view of the target based on the image acquisition device to obtain multiple photos;

[0012] receiving an upload of a plurality of said photos;

[0013] The photos are subjected to photo verification to eliminate photos that do not meet preset rules and obtain a photo set; wherein the preset rules include that the overlap of the photos is greater than a preset overlap threshold.

[0014] In some embodiments, the three-dimensional reconstruction is performed based on the photo set using the 3DGS technology to obtain a 3DGS model, including:

[0015] Uploading the photo set to a 3D reconstruction server, and starting a 3D reconstruction data processing module of the 3D reconstruction server;

[0016] Generate point cloud data according to the photo set based on the three-dimensional reconstruction data processing module;

[0017] A 3DGS model is constructed according to the point cloud data using 3DGS technology.

[0018] In some embodiments, generating point cloud data according to the photo set based on the 3D reconstruction data processing module includes:

[0019] Running motion structure recovery software based on the three-dimensional reconstruction data processing module;

[0020] Selecting images taken at different angles and positions from the photo collection according to the motion structure recovery software;

[0021] Extracting and matching features of the image to obtain matching feature point pairs;

[0022] Camera pose estimation and triangulation are performed according to the matched feature point pairs to generate point cloud data.

[0023] In some embodiments, the step of constructing a 3DGS model based on the point cloud data using the 3DGS technology includes:

[0024] Modeling a three-dimensional Gaussian image according to the point cloud data, and performing rasterization processing and rendering on the three-dimensional Gaussian image to obtain an initial two-dimensional image;

[0025] Calculating the difference loss between the initial two-dimensional image and the captured real image;

[0026] Adjusting parameters of the initial two-dimensional image according to the difference loss, and performing point density control on the initial two-dimensional image;

[0027] Performing depth sorting and blending processing on the initial two-dimensional image to obtain a target two-dimensional image;

[0028] A 3DGS model is constructed according to the target two-dimensional image.

[0029] In some embodiments, performing depth sorting and blending on the initial two-dimensional image to obtain a target two-dimensional image includes:

[0030] Sorting the initial two-dimensional images based on the depth information to obtain a sorting result;

[0031] For each pixel point, based on the sorting result, the value of each of the initial two-dimensional images at the pixel point is calculated in a front-to-back order to obtain all values ​​of the pixel point;

[0032] Mix all the values ​​of the pixel points to obtain a target pixel value;

[0033] A target two-dimensional image is obtained according to the target pixel value.

[0034] In some embodiments, the step of reading the point cloud file information of the 3DGS model and converting the format of the point cloud file information to obtain the general three-dimensional mesh model data includes:

[0035] Reading point cloud file information of the 3DGS model based on a third-party library;

[0036] Obtaining the three-dimensional geometric data, object index data and mathematical expression attribute data generated by 3DGS in the point cloud file information;

[0037] Storing the three-dimensional geometric data, object index data, and mathematical expression attribute data generated by 3DGS in a memory;

[0038] The three-dimensional geometric data and the object index data are format converted to obtain general three-dimensional mesh model data.

[0039] In some embodiments, performing three-dimensional rendering based on the general three-dimensional mesh model data to visualize the near view of the target includes:

[0040] Performing a three-dimensional rendering operation based on the general three-dimensional mesh model data using WebGL technology to visualize a three-dimensional model of a near-view target; wherein the three-dimensional rendering operation includes dragging, rotating, scaling, and translating;

[0041] Business engineering functions are superimposed based on the generated three-dimensional model of the target close-up.

[0042] In addition, to achieve the above-mentioned purpose, the present invention also proposes a close-up three-dimensional reconstruction device based on 3DGS, comprising:

[0043] The acquisition module is used to take circular photos of the near view of the target based on the image acquisition device to obtain a photo collection;

[0044] A 3DGS reconstruction module, used for performing three-dimensional reconstruction based on the photo set using 3DGS technology to obtain a 3DGS model;

[0045] A format conversion module, used for reading the point cloud file information of the 3DGS model and performing format conversion on the point cloud file information to obtain general three-dimensional mesh model data;

[0046] The three-dimensional rendering module is used to perform three-dimensional rendering based on the general three-dimensional grid model data to visualize the near view of the target.

[0047] In addition, to achieve the above-mentioned purpose, the present invention also proposes an electronic device, which includes: a memory, a processor, and a 3DGS-based close-up three-dimensional reconstruction program stored in the memory and executable on the processor, wherein the 3DGS-based close-up three-dimensional reconstruction program is configured to implement the 3DGS-based close-up three-dimensional reconstruction method as described above.

[0048] The present invention provides a close-up three-dimensional reconstruction method based on 3DGS, including: taking circular shots of the close-up of the target based on an image acquisition device to obtain a photo set; performing three-dimensional reconstruction based on the photo set using 3DGS technology to obtain a 3DGS model; reading the point cloud file information of the 3DGS model, and performing format conversion on the point cloud file information to obtain general three-dimensional grid model data; performing three-dimensional rendering based on the general three-dimensional grid model data to visualize the close-up of the target. In the present invention, with the help of the ability of 3DGS technology in three-dimensional reconstruction, a three-dimensional model with more realistic details and a more three-dimensional model is formed in the scene of the close-up model. For the ply point cloud file information generated by the 3DGS model, an additional format conversion is performed to form standardized general three-dimensional grid model data, which is used for subsequent three-dimensional rendering and other links, thereby overcoming the problem that the traditional 3DGS model has obvious needle-like shape in the close-up mode and is not suitable for standardized application of engineering data, improving the secondary development capability and improving the feasibility of engineering implementation. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A schematic diagram of the structure of an electronic device in a hardware operating environment involved in an embodiment of the present invention;

[0050] Figure 2 It is a flow chart of an embodiment of a close-up 3D reconstruction method based on 3DGS of the present invention;

[0051] Figure 3It is a schematic diagram of a close-up three-dimensional reconstruction scheme based on 3DGS involved in an embodiment of the present invention;

[0052] Figure 4 A schematic diagram of a data collection process involved in an embodiment of the present invention;

[0053] Figure 5 A schematic diagram of a three-dimensional reconstruction process involved in an embodiment of the present invention;

[0054] Figure 6 It is a structural block diagram of an embodiment of a close-up 3D reconstruction device based on 3DGS of the present invention.

[0055] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0058] In addition, the descriptions of "first", "second", etc. in the present invention are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but it must be based on the ability of ordinary technicians in the field to implement. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by the present invention. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0059] Reference Figure 1 , Figure 1 The figure is a schematic diagram of the structure of an electronic device of the hardware operating environment involved in the embodiment of the present invention.

[0060] like Figure 1As shown, the electronic device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (Wireless-Fidelity, Wi-Fi) interface). The memory 1005 may be a high-speed random access memory (Random Access Memory, RAM memory) or a stable non-volatile memory (Non-Volatile Memory, NVM), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0061] Those skilled in the art will understand that Figure 1 The structure shown in the figure does not constitute a limitation on the electronic device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0062] like Figure 1 As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and a close-up 3D reconstruction program based on 3DGS.

[0063] exist Figure 1 In the electronic device shown, the network interface 1004 is mainly used for data communication with a network server; the user interface 1003 is mainly used for data interaction with a user; the processor 1001 and the memory 1005 in the electronic device of the present invention can be set in the electronic device, and the electronic device calls the 3DGS-based close-up three-dimensional reconstruction program stored in the memory 1005 through the processor 1001, and executes the 3DGS-based close-up three-dimensional reconstruction method provided by an embodiment of the present invention.

[0064] The present invention provides a close-up three-dimensional reconstruction method, device and equipment based on 3DGS.

[0065] The embodiment of the present invention provides a close-up 3D reconstruction method based on 3DGS, referring to Figure 2 , Figure 2 The figure is a flow chart of an embodiment of a close-up 3D reconstruction method based on 3DGS according to the present invention.

[0066] like Figure 2As shown, the close-up 3D reconstruction method based on 3DGS includes:

[0067] Step S100: taking circular photos of the near view of the target based on the image acquisition device to obtain a photo set;

[0068] Step S200: using 3DGS technology to perform three-dimensional reconstruction based on the photo set to obtain a 3DGS model;

[0069] Step S300: reading the point cloud file information of the 3DGS model, and performing format conversion on the point cloud file information to obtain general three-dimensional mesh model data;

[0070] Step S400: performing three-dimensional rendering based on the general three-dimensional mesh model data to visualize the near view of the target.

[0071] It should be noted that the execution subject in this embodiment may be an electronic device, which may be a computer device with data processing functions, or other devices that can achieve the same or similar functions. This embodiment does not limit this. In this embodiment, a computer device is used as an example for explanation.

[0072] It is understandable that if Figure 3 As shown, this embodiment adopts 3D Gaussian Splatting (abbreviated as 3DGS) as an explicit representation method for reconstructing scenes, and proposes a close-up 3D reconstruction solution based on 3DGS. The whole process of data collection, processing and rendering is completed by using a handheld device to collect data and combining a 3D reconstruction server, a data conversion module and a 3D rendering engine as an example. The method described in this embodiment is described below in combination with specific steps.

[0073] In one embodiment, circular shooting of a close-up of a target is performed based on an image acquisition device to obtain a photo collection, including: circular shooting of a close-up of a target is performed based on an image acquisition device to obtain multiple photos; receiving multiple uploaded photos; performing photo verification on the photos to eliminate photos that do not meet preset rules to obtain a photo collection; wherein the preset rules include that the overlap of the photos is greater than a preset overlap threshold.

[0074] Specifically, Figure 4 As shown in the figure, the first stage is data collection: use an image acquisition device (such as a mobile device) to take circular photos of the scene (such as a close-up of the target), with the number of photos not less than 18. Upload the photos taken in the mobile device APP, and the APP can automatically read the camera parameters of the mobile device. Perform photo verification, remove photos that do not meet the rules, and obtain a photo collection. If the photo collection can meet the requirements of 3D reconstruction, upload the photos in the photo collection to the 3D reconstruction server.

[0075] It should be noted that loop photography is usually used to capture 360-degree scenes or objects. In this shooting mode, the camera is rotated around a center point and a series of continuous shots are taken to ensure that the subject is covered from all angles. Loop photography has the following key features: all-round coverage, continuous shooting, and post-synthesis. Among them, loop photography keeps the camera at a fixed distance and height, and takes continuous shots around the center of the subject, with a certain overlap (coincidence) between each photo.

[0076] For example, this embodiment is described by taking the preset overlap threshold of 70% as an example. According to the preset rules, photo verification is performed, and two adjacent pictures need to have a overlap of more than 70%. Photos that do not meet the preset rules are eliminated to obtain a result set with erroneous data eliminated.

[0077] In one example, rule filtering can also be performed on the result set of excluding erroneous data. It can be understood that in the three-dimensional reconstruction process, rule filtering generally refers to removing images that cannot effectively contribute to the reconstruction of high-quality three-dimensional models. Non-compliant rules may include, but are not limited to: too narrow or repeated viewing angles; poor quality such as blurred, overexposed or underexposed photos or severe noise; motion blur; non-compliance with epipolar constraints; occlusion; inconsistent lighting conditions; incorrect exposure or white balance; lack of texture, etc. In this embodiment, photos that do not meet the rules are eliminated and rule filtering is performed to eliminate photos that do not meet the requirements to improve the accuracy of the reconstruction results. In actual applications, these rules may also be adjusted in combination with specific application scenarios and needs, and this embodiment does not limit this.

[0078] In one embodiment, 3DGS technology is used to perform three-dimensional reconstruction based on the photo set to obtain a 3DGS model, including: uploading the photo set to a three-dimensional reconstruction server, and starting a three-dimensional reconstruction data processing module of the three-dimensional reconstruction server; generating point cloud data based on the photo set based on the three-dimensional reconstruction data processing module; and using 3DGS technology to construct a 3DGS model based on the point cloud data.

[0079] Specifically, Figure 5As shown, the second stage is 3D reconstruction: the pictures (i.e., photo collections) uploaded by the APP are collected, and the 3D reconstruction data processing module of the 3D reconstruction server is started; the SfM technology is used to generate a point cloud in the image; each point is modeled as a 3D Gaussian map, and the position and color of the Gaussian map are inferred; according to the camera parameters, the obtained 3D Gaussian map is projected into 2D; the image is rendered for all current differentiable Gaussian functions, and the difference loss between the rendered image and the real image is calculated; the parameters of the Gaussian image are adjusted according to the difference loss, and the density of the points is controlled according to the current Gaussian image; the Gaussian maps are sorted by depth; for each pixel, the value of each Gaussian map at the pixel point is calculated from the sorting result from front to back, and all values ​​are mixed to obtain the final pixel value; the final 3DGS model is formed and stored in the 3D reconstruction server.

[0080] In one embodiment, point cloud data is generated according to the photo collection based on the three-dimensional reconstruction data processing module, including: running motion structure recovery software based on the three-dimensional reconstruction data processing module; selecting images taken at different angles and positions from the photo collection according to the motion structure recovery software; performing feature extraction and matching on the images to obtain matching feature point pairs; and performing camera posture estimation and triangulation based on the matching feature point pairs to generate point cloud data.

[0081] In this embodiment, the motion structure recovery software is run based on the 3D reconstruction data processing module, and the structure from motion (SfM) technology of the motion structure recovery software is used to generate a point cloud in the image. Specifically, the photo collection has sufficiently overlapping images, and these images are taken from different angles and positions, so that the 3D structure can be reconstructed through matching points between these images.

[0082] Exemplarily, point cloud generation may include the following steps: Image feature extraction for a photo collection: Use feature detection algorithms such as SIFT (Scale Invariant Feature Transform), SURF (Speeded Robust Features), and ORB (Oriented FAST and RotatedBRIEF) to extract key points in each image and calculate descriptors for these key points. Feature matching: Match the extracted feature points between images to establish a correspondence between the feature points. Here, the filtering strategy adopted may be to use the similarity of the feature point descriptors to filter out possible matches. Geometric verification: Verify the accuracy of the match through geometric constraints such as the essential matrix or the fundamental matrix, and remove incorrect matches. Camera pose estimation: Use matched feature point pairs to estimate the relative camera pose (rotation and translation) between images by solving the essential matrix or the fundamental matrix. Triangulation: For each matching point pair, use the triangulation method to calculate their positions in three-dimensional space. Here, it is necessary to calculate based on the camera parameters of the mobile device such as the intrinsic parameters (focal length, principal point, etc.) and the estimated camera pose. Global optimization: After obtaining the preliminary 3D point positions and camera poses, a global optimization process is performed to minimize the reprojection error, thereby improving the accuracy of the 3D point positions and camera poses. Generate point cloud: Through the above steps, a sparse 3D point cloud data can be generated, representing the structure of the reconstructed target close-up in the image.

[0083] In one example, a point cloud may be generated using SfM software such as OpenMVG, COLMAP, etc., which is not limited in this embodiment.

[0084] In one embodiment, a 3DGS model is constructed based on the point cloud data using 3DGS technology, including: modeling a three-dimensional Gaussian image based on the point cloud data, and rasterizing and rendering the three-dimensional Gaussian image to obtain an initial two-dimensional image; calculating the difference loss between the initial two-dimensional image and the captured real image; adjusting the parameters of the initial two-dimensional image based on the difference loss, and controlling the density of points in the initial two-dimensional image; performing depth sorting and mixing on the initial two-dimensional image to obtain a target two-dimensional image; and constructing a 3DGS model based on the target two-dimensional image.

[0085] In one embodiment, the initial two-dimensional image is depth sorted and mixed to obtain a target two-dimensional image, including: sorting the initial two-dimensional image based on depth information to obtain a sorting result; for each pixel point, calculating the value of each pixel point of the initial two-dimensional image in a front-to-back order based on the sorting result to obtain all the values ​​of the pixel point; mixing all the values ​​of the pixel points to obtain a target pixel value; and obtaining a target two-dimensional image according to the target pixel value.

[0086] Specifically, the points in the point cloud file are modeled as 3D Gaussian graphs. Map), which may include the following steps: First, it is necessary to read the point cloud file (such as .ply, .pcd and other formats) and extract the point coordinate information therein; select appropriate Gaussian function parameters for each point, such as the center (using the coordinates of the point as the center of the Gaussian function), the covariance matrix (calculated by the distribution of the neighborhood points around the point) and the standard deviation (determines the width of the Gaussian function and the influence range of the point); for each point in the point cloud, according to the parameters of its Gaussian function, calculate its contribution value at each position in the three-dimensional space; discretize the entire space into a voxel grid, and the voxel is the smallest unit in the three-dimensional space and can be regarded as a three-dimensional pixel; for each voxel in the space, calculate the sum of the contribution values ​​of the Gaussian functions of all points in the voxel, such as integrating or summing the Gaussian functions of each point; in order to ensure that the values ​​in the Gaussian map can be represented within a certain range, the contribution values ​​in the voxel need to be normalized; the values ​​in the voxel grid are used as the pixel values ​​of the 3D Gaussian map, so that the Gaussian function contribution value of each voxel constitutes a three-dimensional image, that is, a 3D Gaussian map is generated.

[0087] Specifically, the position (spatial information) and color (color information) of the Gaussian map are inferred. Position inference: determine the voxel center; calculate the maximum position of the Gaussian distribution. The position of the point in the voxel can be estimated by finding the maximum value of the Gaussian function in each voxel; three-dimensional spatial position interpolation: an interpolation algorithm (such as trilinear interpolation) can be used inside the voxel to estimate the exact position of the maximum value to further improve the accuracy. Color inference steps: color mapping, if the 3D Gaussian map is generated based on color point cloud data, the color of each voxel can be obtained by weighted averaging the colors of all points in the voxel; color interpolation: similar to the above position interpolation, color interpolation can be performed inside the voxel to obtain a more accurate color value. Among them, the original point cloud has no color information, and texture mapping or other techniques can be used to infer the color of the voxel. The inferred position and color information is output as a 3D model, which can be used for visualization or further processing.

[0088] Specifically, projecting a 3D Gaussian image into a 2D image may include the following steps: obtaining internal parameters and external parameters of camera parameters of a mobile device (such as a mobile phone) for taking pictures, wherein the internal parameters include focal length, principal point coordinates, etc., and the external parameters include a rotation matrix and a translation vector; using the internal parameters and external parameters, constructing a projection matrix; traversing the points in the 3D Gaussian image, using the external parameters, converting the 3D world coordinates into coordinates in the camera coordinate system; applying the camera intrinsic parameter matrix, converting the camera coordinates into normalized image plane coordinates; converting the normalized image coordinates into actual image coordinates through perspective transformation; for each 3D point, using a Gaussian function to calculate its contribution to each pixel on the 2D image plane; accumulating the contribution value of each 3D point to each pixel of the 2D image, thereby obtaining a final 2D Gaussian image; and normalizing the obtained 2D Gaussian image to ensure that all values ​​are within a reasonable range.

[0089] It should be noted that in practical applications, it may be necessary to handle edge cases, such as when the point is in front of or behind the camera, or when the point is projected outside the image. In addition, if the points in the 3D Gaussian image are very dense, sampling or filtering needs to be considered during the projection process to avoid excessive aliasing.

[0090] Specifically, the steps of rendering an image for all current differentiable Gaussian functions in the 3DGS algorithm may include: setting Gaussian points according to the center position, covariance matrix and weight of each Gaussian point; rasterization: for each pixel on the image plane, projecting the pixel to a point in the 3D space, and for each pixel, testing whether the pixel is located in the volume of the Gaussian point; rendering integration: for pixels located in the volume of the Gaussian point, calculating the Gaussian weight and synthesizing the color of each pixel; ensuring that the color values ​​of all pixels are within the valid range (usually 0 to 1); and outputting the rendered color values ​​as the final 2D image (i.e., the rendered image).

[0091] It can be understood that in the 3DGS algorithm, rasterization refers to the process of representing a three-dimensional scene or object as a two-dimensional image. Figure 5 As shown, in 3DGS, rasterization involves converting a three-dimensional Gaussian distribution into pixel values ​​in a two-dimensional image space.

[0092] Exemplarily, rasterization in 3DGS may include the following steps: converting the center and covariance matrix of a 3D Gaussian distribution from a three-dimensional world space or camera space to a two-dimensional screen space; dividing the screen space into smaller tiles to improve rendering efficiency; for each tile, determining which 3D Gaussian distributions intersect with the tile, if a Gaussian distribution intersects with the tile, then it will be instantiated in the tile, that is, its contribution will be calculated and accumulated in the tile; sorting all instantiated Gaussian distributions, usually based on their depth information, this sorting ensures that in the subsequent blending process, the Gaussian distributions can be processed in order from far to near, thereby reducing the number of required blending operations; for each pixel, mixing the contribution values ​​of all Gaussian distributions covering the pixel, for example, calculating the value of each Gaussian distribution at the pixel position, and adding these values ​​to obtain the final pixel value. Outputting the mixed pixel value as the final 2D image is the target two-dimensional image.

[0093] It should be noted that, in this embodiment, the goal of rasterization is to efficiently render a three-dimensional scene or object so that it can be displayed in a two-dimensional image. This process is usually performed on a graphics processing unit (GPU), and the GPU is used to implement parallel processing of pixel-level operations.

[0094] Specifically, Figure 5 As shown, difference comparison: calculate the difference loss between the rendered image and the real image, adjust the parameters of the Gaussian image according to the difference loss, and control the density of points according to the current Gaussian image to realize the training of the 3DGS model. In the 3DGS algorithm, calculating the difference loss between the rendered image and the real image is an important step in evaluating the rendering quality. The difference loss is used to guide the optimization process (model training process), such as minimizing the difference between the rendered image and the real image. Exemplarily, the following loss functions can be used: mean square error MSE, root mean square error RMSE, cross-entropy loss (Cross-Entropy Loss), structural similarity index SSIM and mean absolute error MAE.

[0095] In practical applications, the loss function is selected according to the specific application scenario and requirements. For example, if the difference between the rendered image and the real image is small, it is more appropriate to choose the mean square error MSE and the root mean square error RMSE; if the structural information of the image needs to be considered, it is more appropriate to choose the structural similarity index SSIM. In the 3DGS algorithm, the loss function can be used to optimize the position, shape and weight of the Gaussian points. This is achieved through gradient descent or other optimization algorithms to minimize the loss function and improve the quality of the rendered image.

[0096] Specifically, in the 3DGS algorithm, the point density control based on the current Gaussian image mainly includes the following steps: restore the initial sparse point cloud through the input image sequence, and the initial sparse point cloud is used as the center point of the subsequent Gaussian distribution; for each pixel at the current perspective, calculate its gradient information at multiple perspectives, and these gradient information reflects the degree of detail of the surface around the pixel; use the calculated gradient information to evaluate the density around the pixel and determine which areas need more points to represent richer details; assign dynamic weights to each pixel according to the coverage of each pixel point by different perspectives, and these weights will affect the contribution of the pixel point to the point density control; according to the gradient density evaluation results, adaptively increase or decrease the points in the point cloud, for example, add new points to areas with large gradients or merge adjacent points to increase local density; scale the gradient according to the distance from the pixel point to the camera to prevent too many points from being generated in the area near the camera (inhibiting floater growth).

[0097] In practical applications, the above density adjustment step is combined with an optimization loop (model training based on difference loss) to continuously iterate and update the density and position of the point cloud. After each iteration, the updated point cloud is used for rendering, and the rendering result is compared with the target image. Based on the comparison result, the point cloud density is further adjusted. When the preset number of iterations, density threshold or rendering quality meets certain conditions, the algorithm terminates. In this embodiment, the density of the point cloud is adaptively adjusted based on the Gaussian image and gradient information to achieve high-quality 3D scene reconstruction.

[0098] In one embodiment, the point cloud file information of the 3DGS model is read, and the format of the point cloud file information is converted to obtain general three-dimensional mesh model data, including: reading the point cloud file information of the 3DGS model based on a third-party library; obtaining the three-dimensional geometry data, object index data and mathematical expression attribute data generated by 3DGS in the point cloud file information; storing the three-dimensional geometry data, object index data and mathematical expression attribute data generated by 3DGS in a memory; and converting the format of the three-dimensional geometry data and the object index data to obtain general three-dimensional mesh model data.

[0099] Specifically, point cloud files are usually in .ply or .bin format. Based on third-party libraries such as Apose.3d, the point cloud file information of the 3DGS model (such as ply file information) can be read, and the 3D geometry data, object index data, and mathematical expression attribute data generated by the 3DGS model can be stored in memory. The 3D geometry data and object index data are converted into general 3D model formats such as obj or glTF to obtain general 3D mesh model data, and then exported as files.

[0100] In one embodiment, three-dimensional rendering is performed based on the general three-dimensional mesh model data to visualize the target close view, including: performing a three-dimensional rendering operation based on the general three-dimensional mesh model data using WebGL technology to visualize the three-dimensional model of the target close view; wherein the three-dimensional rendering operation includes dragging, rotating, scaling and translating; and superimposing a business engineering function based on the generated three-dimensional model of the target close view.

[0101] Specifically, the file finally exported by this embodiment is a universal three-dimensional mesh model data based on a universal format. Three-dimensional rendering can be performed using WebGL technology based on the universal three-dimensional mesh model data, supporting operations such as dragging and rotating. Other functions can also be superimposed based on the generated three-dimensional model to solve business problems.

[0102] Exemplarily, this embodiment uses WebGL technology for three-dimensional rendering: load three-dimensional model files, such as obj, fbx or glTF formats; perform necessary preprocessing on the model, such as culling, merging meshes, and calculating normals; configure the scene's lighting model, such as point light, directional light, ambient light, etc.; set material properties, such as color, reflectivity, and transparency; if user interaction is required, add event listeners to respond to user input, such as mouse clicks and drags, etc., and update the camera perspective or model state according to user input; if animation is required, update the model's state according to keyframes or skeletal animation, and apply post-processing effects, such as shadows, blurs, reflections, etc.; perform performance analysis and optimization on the rendering process to ensure smooth rendering effects. It is understandable that the specific implementation steps will vary according to project engineering requirements and scene complexity, and this embodiment does not limit this.

[0103] It should be noted that the method described in this embodiment improves the data collection, automated data processing, and data presentation links on a theoretical basis. For the ply point cloud model generated by 3DGS, an additional data conversion is performed to form standardized three-dimensional mesh data (i.e., general three-dimensional mesh model data), and the general three-dimensional mesh model data is imported into common platforms such as cesium.js and three.js, thereby improving the secondary development capabilities and the feasibility of engineering implementation. With the help of the AI ​​capabilities of 3DGS in three-dimensional reconstruction, base data with more realistic details and a more three-dimensional model is formed in the scenario of the close-up model.

[0104] In this embodiment, a handheld device is used as a data acquisition terminal. After taking photos, a large model is trained based on the 3DGS algorithm to form three-dimensional point cloud data for completing the three-dimensional reconstruction process. The reconstructed data is displayed based on the three-dimensional rendering engine to meet the data collection, processing, and rendering of the three-dimensional close-up. The 3DGS algorithm technology, handheld device acquisition capabilities, and three-dimensional rendering capabilities are combined to form a complete full-life cycle solution, which can perform three-dimensional reconstruction of the three-dimensional close-up at the lowest cost, empower the industry with digital twin three-dimensional, and effectively solve the problem of the implementation of 3DGS theoretical knowledge in actual engineering applications.

[0105] It is understandable that since the ply point cloud model generated by 3DGS itself is not a qualified digital twin scene data, the ply model generated by 3DGS will have obvious pricking phenomenon when viewed at close range in the 3DGS engine, and the engine itself does not have secondary development capabilities except for improving rendering performance with mathematical expressions, and it is difficult to superimpose and engineer the remaining business data. The method described in this embodiment uses the 3DGS algorithm model to process data in the three-dimensional reconstruction module to produce a more refined close-up ply model, and then converts the ply point cloud model into a general three-dimensional mesh model data, and finally displays the reconstructed model in the general three-dimensional rendering engine, and supports superimposing the remaining business data and secondary development to form a complete engineering solution.

[0106] This embodiment provides a close-up three-dimensional reconstruction method based on 3DGS, including: taking circular shots of the close-up of the target based on an image acquisition device to obtain a photo set; using 3DGS technology to perform three-dimensional reconstruction based on the photo set to obtain a 3DGS model; reading the point cloud file information of the 3DGS model, and converting the format of the point cloud file information to obtain general three-dimensional mesh model data; performing three-dimensional rendering based on the general three-dimensional mesh model data to visualize the close-up of the target. In this embodiment, with the help of the ability of 3DGS technology in three-dimensional reconstruction, a three-dimensional model with more realistic details and a more three-dimensional model is formed in the scene of the close-up model. For the ply point cloud file information generated by the 3DGS model, an additional format conversion is performed to form standardized general three-dimensional mesh model data for subsequent three-dimensional rendering and other links, which overcomes the problem that the traditional 3DGS model has obvious needle-like shape in the close-up mode and is not suitable for standardized application of engineering data, improves the secondary development capability and improves the feasibility of engineering implementation.

[0107] In addition, an embodiment of the present invention further proposes a storage medium, on which a close-up three-dimensional reconstruction program based on 3DGS is stored. When the close-up three-dimensional reconstruction program based on 3DGS is executed by a processor, the steps of the close-up three-dimensional reconstruction method based on 3DGS as described above are implemented.

[0108] Reference Figure 6 , Figure 6 It is a structural block diagram of an embodiment of a close-up 3D reconstruction device based on 3DGS of the present invention.

[0109] like Figure 6 As shown, the close-up 3D reconstruction device based on 3DGS includes:

[0110] The acquisition module 10 is used to perform circular photography of the near view of the target based on the image acquisition device to obtain a photo set;

[0111] A 3DGS reconstruction module 20, configured to perform three-dimensional reconstruction based on the photo set using 3DGS technology to obtain a 3DGS model;

[0112] The format conversion module 30 is used to read the point cloud file information of the 3DGS model and perform format conversion on the point cloud file information to obtain general three-dimensional mesh model data;

[0113] The three-dimensional rendering module 40 is used to perform three-dimensional rendering based on the general three-dimensional grid model data to visualize the near view of the target.

[0114] It is understandable that since the ply point cloud model generated by 3DGS itself is not a qualified digital twin scene data, the ply model generated by 3DGS will have obvious pricking phenomenon when viewed at close range in the 3DGS engine, and the engine itself does not have secondary development capabilities except for improving rendering performance with mathematical expressions, and it is difficult to superimpose and engineer the remaining business data. The method described in this embodiment uses the 3DGS algorithm model to process data in the three-dimensional reconstruction module to produce a more refined close-up ply model, and then converts the ply point cloud model into a general three-dimensional mesh model data, and finally displays the reconstructed model in the general three-dimensional rendering engine, and supports superimposing the remaining business data and secondary development to form a complete engineering solution.

[0115] This embodiment provides a close-up 3D reconstruction device based on 3DGS. In this embodiment, by leveraging the capabilities of 3DGS technology in 3D reconstruction, a 3D model with more realistic details and a more three-dimensional model is formed in the scene of the close-up model. For the ply point cloud file information generated by the 3DGS model, an additional format conversion is performed to form standardized general 3D mesh model data for subsequent 3D rendering and other links, overcoming the problem that the traditional 3DGS model has obvious needle-like shapes in the close-up mode and is not suitable for standardized application of engineering data, improving the secondary development capabilities and improving the feasibility of engineering implementation.

[0116] It should be noted that the technical details that are not described in detail in the embodiment of the 3DGS-based close-up 3D reconstruction device can be found in the 3DGS-based close-up 3D reconstruction method described above provided in any embodiment of the present invention, and will not be repeated here.

[0117] It should be understood that the above is only an example and does not constitute any limitation on the technical solution of the present invention. In specific applications, technicians in this field can make settings as needed, and the present invention does not limit this.

[0118] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of the present invention. In practical applications, technicians in this field can select part or all of them according to actual needs to achieve the purpose of the present embodiment, and no limitation is made here.

[0119] In addition, it should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0120] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0121] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory (ROM) / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0122] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A close-up 3D reconstruction method based on 3DGS, characterized in that: For engineering applications of digital twin scenarios, the method includes: Taking circular shots of the close-up of the target based on the image acquisition device to obtain a photo collection; Using 3DGS technology to perform three-dimensional reconstruction based on the photo collection to obtain a 3DGS model; Reading the point cloud file information of the 3DGS model, and performing format conversion on the point cloud file information to obtain general three-dimensional mesh model data; Performing three-dimensional rendering based on the general three-dimensional grid model data to visualize the near view of the target; The point cloud file information of the 3DGS model is read, and the format of the point cloud file information is converted to obtain general three-dimensional mesh model data, including: reading the point cloud file information of the 3DGS model based on a third-party library; obtaining the three-dimensional geometry data, object index data and mathematical expression attribute data generated by 3DGS in the point cloud file information; storing the three-dimensional geometry data, object index data and mathematical expression attribute data generated by 3DGS in a memory; and converting the format of the three-dimensional geometry data and the object index data to obtain general three-dimensional mesh model data.

2. The method according to claim 1, characterized in that The method of performing circular shooting of the near view of the target based on the image acquisition device to obtain a photo set includes: Taking circular photos of the close-up view of the target based on the image acquisition device to obtain multiple photos; receiving an upload of a plurality of said photos; The photos are subjected to photo verification to eliminate photos that do not meet preset rules and obtain a photo set; wherein the preset rules include that the overlap of the photos is greater than a preset overlap threshold.

3. The method according to claim 1, characterized in that The 3DGS technology is used to perform three-dimensional reconstruction according to the photo set to obtain a 3DGS model, including: Uploading the photo set to a 3D reconstruction server, and starting a 3D reconstruction data processing module of the 3D reconstruction server; Generate point cloud data according to the photo set based on the three-dimensional reconstruction data processing module; A 3DGS model is constructed according to the point cloud data using 3DGS technology.

4. The method according to claim 3, characterized in that The generating point cloud data according to the photo set based on the three-dimensional reconstruction data processing module includes: Running motion structure recovery software based on the three-dimensional reconstruction data processing module; Selecting images taken at different angles and positions from the photo collection according to the motion structure recovery software; Extracting and matching features of the image to obtain matching feature point pairs; Camera pose estimation and triangulation are performed according to the matched feature point pairs to generate point cloud data.

5. The method according to claim 3, characterized in that The method of using the 3DGS technology to construct a 3DGS model according to the point cloud data includes: Modeling a three-dimensional Gaussian image according to the point cloud data, and performing rasterization processing and rendering on the three-dimensional Gaussian image to obtain an initial two-dimensional image; Calculating the difference loss between the initial two-dimensional image and the captured real image; Adjusting parameters of the initial two-dimensional image according to the difference loss, and performing point density control on the initial two-dimensional image; Performing depth sorting and blending processing on the initial two-dimensional image to obtain a target two-dimensional image; A 3DGS model is constructed according to the target two-dimensional image.

6. The method according to claim 5, characterized in that The depth sorting and mixing of the initial two-dimensional image to obtain a target two-dimensional image includes: Sorting the initial two-dimensional images based on the depth information to obtain a sorting result; For each pixel point, based on the sorting result, the value of each of the initial two-dimensional images at the pixel point is calculated in a front-to-back order to obtain all values ​​of the pixel point; Mix all the values ​​of the pixel points to obtain a target pixel value; A target two-dimensional image is obtained according to the target pixel value.

7. The method according to claim 1, characterized in that The performing three-dimensional rendering based on the general three-dimensional grid model data to visualize the near view of the target includes: Performing a three-dimensional rendering operation based on the general three-dimensional mesh model data using WebGL technology to visualize a three-dimensional model of a near-view target; wherein the three-dimensional rendering operation includes dragging, rotating, scaling, and translating; Business engineering functions are superimposed based on the generated three-dimensional model of the target close-up.

8. A close-up 3D reconstruction device based on 3DGS, characterized in that: For engineering applications of digital twin scenarios, the device comprises: The acquisition module is used to take circular photos of the near view of the target based on the image acquisition device to obtain a photo collection; A 3DGS reconstruction module, used for performing three-dimensional reconstruction based on the photo set using 3DGS technology to obtain a 3DGS model; A format conversion module, used for reading the point cloud file information of the 3DGS model and performing format conversion on the point cloud file information to obtain general three-dimensional mesh model data; A three-dimensional rendering module, used for performing three-dimensional rendering based on the general three-dimensional grid model data to visualize the near view of the target; The point cloud file information of the 3DGS model is read, and the format of the point cloud file information is converted to obtain general three-dimensional mesh model data, including: reading the point cloud file information of the 3DGS model based on a third-party library; obtaining the three-dimensional geometry data, object index data and mathematical expression attribute data generated by 3DGS in the point cloud file information; storing the three-dimensional geometry data, object index data and mathematical expression attribute data generated by 3DGS in a memory; and converting the format of the three-dimensional geometry data and the object index data to obtain general three-dimensional mesh model data.

9. An electronic device, characterized in that: The electronic device includes: a memory, a processor, and a 3DGS-based close-up 3D reconstruction program stored in the memory and executable on the processor, wherein the 3DGS-based close-up 3D reconstruction program is configured to implement the 3DGS-based close-up 3D reconstruction method as described in any one of claims 1 to 7.

Images